MCFARM File-Server Node Preparation Outline

MCFARM File-Server Node Preparation Outline

4/21/02

Once you have a job server established (which is also a file-server), you can convert any regular production node into an additional file-server (node FFF here) using these steps. Additional file-servers provide a way to place the shared minbias data onto more nodes that that no single node becomes a file-serving bottleneck (and also to simply provide more cache space).

·  Prepare the node as you would for regular production, except that you probably want to set max=0 in the distribute.conf file so that this node never gets any production jobs. In a moderate to large size farm, the d0sim binaries will be hitting this file-server fairly heavily, and it must have CPU time available to respond to their needs. You can try the file server with and without production of its own (or, if the node has 2 CPUs, you could allocate only one of them to production). Examine the prodsumm output for other nodes for a period of several days to see what the impact is on having production on this file server.

·  The file server must export its scratch directory(s) to every other node. As root on node FFF, modify the /etc/exports file to include a line for every other node, such as

/scratch hepfmNNN(rw,no_root_squash)

/scratch2 hepfmNNN(rw,no_root_squash) # if there are additional disk partitions…

Then as root issue the command /usr/sbin/exportfs -ar to place the new exports file in play.

·  As root on the file server, make these directories and links (may already exist):

mkdir /scratch/cache_A

chown mcfarm.mcfarm /scratch/cache_A

ln -s /scratch/cache_A /cacheFFF_A

chown mcfarm.mcfarm /cacheFFF_A

(repeat for all other cache partitions, using _B and /scratch2, etc.).

·  Every other production node (not the job server) must auto-mount this new file-server partition. On every such node now on the farm, as root, add these directories and links:

mkdir /mnt/hepfmFFF

mkdir /mnt/hepfmFFF/cache_A

ln -s /mnt/hepfmFFF/cache_A /cacheFFF_A

chown mcfarm.mcfarm /mnt/hepfmFFF

chown mcfarm.mcfarm /mnt/hepfmFFF/cache_A

chown mcfarm.mcfarm /cacheFFF_A

(repeat the above for any additional partitions B, C, etc).

then add this line to the /etc/fstab file of every production node:

hepfmFFF.uta.edu:/cacheFFF_A /mnt/hepfmFFF/cache_A nfs

rw,intr,rsize=16384,wsize=16384,actimeo=5

hepfmFFF.uta.edu:/cacheFFF_B /mnt/hepfmFFF/cache_B nfs (this line for 2nd partition)

rw,intr,rsize=16384,wsize=16384,actimeo=5

and then as root on each of those nodes, issue a mount –a command. As mcfarm on each

production node, you should now be able to see the /cacheFFF_A contents (and B, C, etc.). Test this

by copying test files in and out of the directory.

NOTE: You may find the root daemon useful for the above mass changes to all the production nodes. The commands for the directory and link creations may be put into a script, made executable, and then run on each production node as in

root_command hepfmNNN --script=/home/mcfarm/your_script

(issued for each production node NNN). You can also make this script do a “cat” of the necessary lines to be added to the /etc/fstab file, and you can issue the mount command also. Remember that when the script runs on the target node, it runs as root, and it has access to the /home/mcfarm directory.

·  Since this node is offering file-serving to the farm, place these two lines in the ~/bin/setup_farm script so that farm software knows to use it (FFF is this node number):

export FARM_FILESERVER_CACHE_FFF_A=$FARM_CACHE'FFF_A'

export FARM_FILESERVER_CACHE_FFF_A_DEV=hda7

Repeat those two lines for B, C, etc. Specify the correct partition. . You will have to stop all gather daemons, issue the new setup_farm command, and restart them before this will take effect.

·  IF this node is ALSO going to be used to hold a portion of the archive queue (which it probably should, because the archives can get large and thus needs to be segmented over multiple nodes), then as root on the new file server FFF create these directories and links:

mkdir /scratch/cache_A/archive

mkdir /scratch/cache_A/archive/jobs

ln -s /scratch/cache_A /archiveFFF_A

chown mcfarm.mcfarm /scratch/cache_A/archive

chown mcfarm.mcfarm /scratch/cache_A/archive/jobs

chown mcfarm.mcfarm /archiveFFF_A

Then modify the ~/bin/setup_farm script to identify this new archive segment, and specify how much room in MB to keep on the disk (e.g., don’t fill up the disk with archive info):

export FARM_ARCHIVE_JOBS_QUEUE_NN=$FARM_ARCHIVE'FFF_A/archive/jobs'

export FARM_ARCHIVE_MINIMUM_NN=2048 # leave 2GB

where NN is a number from 00 to 99 and specifies the order in which this segment is to be filled during archiving, and FFF is the new node. You will have to stop all gather daemons, issue the new setup_farm command, and restart them before this will take effect.

Then on the job server, and on each other gather-server, as root, make these directories and links so they can see this segment of the archive queue:

mkdir /mnt/hepfmFFF/archive_A

ln -s /mnt/hepfmFFF/archive_A /archiveFFF_A

chown mcfarm.mcfarm /mnt/hepfmFFF/archive_A

chown mcfarm.mcfarm /archiveFFF_A

·  Modify ~/bin/attach_node to specifically mention this node as a file-server (in two places). Do the same change to ~/bin/detatch_node. Propagate both script to the /scratch/localbin directory of all nodes (you can use the root daemon for this).

·  The node FFF should now be able to accept new files in its cache directory(s), and they should be visible to every other node. Since the file server was built out of a production node, it should auto-mount the job server. File-serving per se does not require NFS access to any other node but the job server.