Re: Horrible PostgreSQL performance with NFS

[Slawek Zak <slawek.zak@xxxxxxxxx>]

On 1/13/06, Arne Woerner <arne_woerner@xxxxxxxxx> wrote:
> Hiho!
>
> --- Slawek Zak <slawek.zak@xxxxxxxxx> wrote:
> > A couple of days ago I've moved our
> > production database from local disks
> > to NetAPP filer serving NFS. Performance
> > for this server dropped by factor of 10
> > if not more.
> >
> I would like to suggest some tests (I do not have a clear idea,
> why your system becomes slower; a hypothesis will follow below):
>
> 1. benchmark the local disc (watch cpu usage via e. g. vmstat 1)
> % dd if=/dev/zero of=/some/local/filesystem/a bs=1m count=1000

40MB/s. CPU load negligible. I don't have an exact number, as this
machine has other processes running. But overall, the system load
didn't exceed 5%.

> 2. benchmark the NFS filesystem (watch cpu usage via e. g. vmstat
> 1)
> % dd if=/dev/zero of=/some/NFS/filesystem/a bs=1m count=1000

I saturated fast ethernet on the host with this test. Filer is
connected with Gb and can spew around 70MB/s easily. CPU load on the
host didn't exceed 4%.

> 3. test the NIC performance with (again watching the CPU usage
> might help)
> % ping -s 50000 <nfs-server>
> I get 17Mbit/sec which is the actual rate...
> neo# ping -s 50000 vaako
> PING vaako.riddick.homeunix.org (10.1.1.3): 50000 data bytes
> 50008 bytes from 10.1.1.3: icmp_seq=0 ttl=64 time=46.584 ms

Filer doesn't respond to large icmp packets. 20000 was the biggest
that worked. CPU load negligible.

> My theory would be, that your NICs need a lot of CPU time, while
> your local discs dont need so much CPU time. :-)

I don't think so. Drivers accout for system time. It doesn't exceed
20% of overall load. The postgres processes are very busy doing almost
nothing. Semops is most of the work they seem to do.

Thanks, /S
--
Sławek Żak / UNIX Systems Administrator