Re: Horrible PostgreSQL performance with NFS

[Brooks Davis <brooks@xxxxxxxxxxxxxxxxxx>]

On Fri, Jan 13, 2006 at 01:10:48PM -0800, Arne Woerner wrote:
> Hiho!
> 
> --- Slawek Zak <slawek.zak@xxxxxxxxx> wrote:
> > A couple of days ago I've moved our
> > production database from local disks
> > to NetAPP filer serving NFS. Performance
> > for this server dropped by factor of 10
> > if not more.
> >
> I would like to suggest some tests (I do not have a clear idea,
> why your system becomes slower; a hypothesis will follow below):
> 
> 1. benchmark the local disc (watch cpu usage via e. g. vmstat 1)
> % dd if=/dev/zero of=/some/local/filesystem/a bs=1m count=1000
> 
> 2. benchmark the NFS filesystem (watch cpu usage via e. g. vmstat
> 1)
> % dd if=/dev/zero of=/some/NFS/filesystem/a bs=1m count=1000

I'd suggest also testing with small blocks as well.  I suspect that a
large portion of many database accesses quite small and thus transaction
overhead, not streaming performance is likely to be the issue.  If
streaming performance is an issue, increasing your network's MTU should
help somewhat.

> 3. test the NIC performance with (again watching the CPU usage
> might help)
> % ping -s 50000 <nfs-server>
> I get 17Mbit/sec which is the actual rate...
> neo# ping -s 50000 vaako
> PING vaako.riddick.homeunix.org (10.1.1.3): 50000 data bytes
> 50008 bytes from 10.1.1.3: icmp_seq=0 ttl=64 time=46.584 ms

I don't think this tells you much if anything.  You need to know what
your protocol overhead is, not how many ICMP packets you can spew.

> My theory would be, that your NICs need a lot of CPU time, while
> your local discs dont need so much CPU time. :-)

That's certainly part of it, but I doubt it's the whole story.

-- Brooks

-- 
Any statement of the form "X is the one, true Y" is FALSE.
PGP fingerprint 655D 519C 26A7 82E7 2529  9BF0 5D8E 8BE9 F238 1AD4

Attachment: pgp54rE40llCc.pgp
Description: PGP signature