Hi Brian, All,
This email has one theme: GEOM! :)
On Sep 24, 2005, at 10:10 AM, Brian Candler wrote:
Hello,
I was wondering if anyone would care to share their experiences in
synchronising filesystems across a number of nodes in a cluster. I
can think
of a number of options, but before changing what I'm doing at the
moment I'd
like to see if anyone has good experiences with any of the others.
The application: a clustered webserver. The users' CGIs run in a chroot
environment, and these clearly need to be identical (otherwise a CGI
running
on one box would behave differently when running on a different box).
Ultimately I'd like to synchronise the host OS on each server too.
Note that this is a single-master, multiple-slave type of filesystem
synchronisation I'm interested in.
I just wanted to throw out some quick thoughts on a totally different
approach which nobody has really explored in this thread, solutions
which are production level software. (Sorry if I'm repeating things or
giving out info yall' already know:)
--
Geom:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom- intro.html
The core Disk IO framework for FreeBSD, as of 5.x, led by PHK:
http://www.bsdcan.org/2004/papers/geom.pdf
This framework itself is not as useful to you as the utilities which
make use of it,
--
Geom Gate:
http://kerneltrap.org/news/freebsd?from=20
Network device-level client/server disk mapping tool.
(VERY IMPORTANT COMPONENT, it's reportedly faster, and more stable than
NFS has ever been- so people have immediately and happily deployed it
in production systems!)
--
Gvinum and Gmirror:
Gmirror
http://people.freebsd.org/~rse/mirror/
http://www.ie.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom.html
(Sidenote: even Greg Lehey (original author of Vinum), has stated that
it's better to use Geom-based tools than Vinum for the forseeable future.)
--
In a nutshell, to address your needs, let me toss out the following
example setup:
I know of one web-shop in Canada, which is running 2 machines for every
virtual cluster, in the following configuration:
2 servers,
4 SATA drives per box,
quad copper/ethernet gigabit nic on each box
each drive is mirrored using gmirror, over each of the gigabit ethernet
nics
each box is running Vinum Raid5 across the 4 mirrored drives
The drives are then sliced appropriately, and server resources are
distributed across the boxes- with various slices mounted on each box.
The folks I speak of simply have a suite of failover shell scripts
prepared, in the event of a machine experiencing total hardware failure.
Pretty tough stuff, very high-performance, and CHEAP.
--
With that, I'm working towards similar setups, oriented around
redundant jailed systems, with an eventual end to tie CARP (from pf)
into the mix to make for nearly-instantaneous jailed failover
redundancy- (but it's going to be some time before I have what I want
worked out for production on my own).
Regardless, it's worth tapping into the GEOM dialogues, as there are
many new ways of working with disks coming into existence- and the GEOM
framework itself provides an EXTREMELY solid base to bring 'exotic'
disk configurations up to production level quickly.
(Also noteworthy, there's a couple of encrypted disk systems based on
GEOM emerging now too...)
