I've been asked by several people to be specific about what I did
to use the NFS-mounted /usr on OSF/1 V3.0. In the original summary,
I mentioned Jon Forrest's method, but I wasn't sure if it's proper
to include his article. As so many people are interested in this
question, and as I'm not sure if it's proper for me to just quote
part of his article, I'm including the whole article as follows.
Basically, I did exactly as what Jon said in the article. If anyone
has more questions, please let me know and I'll try to help.
Regards,
Yizhong Zhou, University of Southern Maine
zhou_at_usmcs.maine.edu
==========================Start of Jon Forrest's article=================
Setting Up A "Dataless" Environment under OSF/1 (2.X & 3.X)
and
A Comment on DEC's Dataless Management Services
Jon Forrest
forrest_at_CS.Berkeley.EDU
(Draft of 11/10/94)
The following is a description of how to create an environment in which
a DEC Alpha OSF/1 machine serves its local /usr file system to any
number of Alpha clients. Such an environment on a client machine is
called a "dataless" environment. This is actually an inaccurate
description, but I'll stick with it anyway. The key point is that the
clients in question mount their /usr file system from a remote server.
Note that this definition is different than the one used by DEC in
their "dataless" environment in OSF 3.0. Later in this document I'll
discuss these differences.
Although this description is worded as if an OSF/1 server is required,
in reality the /usr file system can be served from any kind machine on
which a complete copy of /usr from an OSF/1 machine exists.
This document is written assuming that it is being read by an experienced
OSF/1 system manager. Not every detail behind every explanation is
included.
This is based on information from Norman Gaywood, Marco Luchini, Arrigo
Triulzi, and Castor Fu as well as my own research. Nobody from DEC
contributed to this.
In writing this description I'm making the following assumptions:
1) Mounting the remote /usr on top a small local /usr file system isn't a
good idea because it wastes space on the local disk and because it
violates one of the reasons for doing the remote mount in the first
place - that there should be just one copy of everything under /usr.
Indeed, once you've set up the remote /usr, you can reinitialize
/usr on every local disk and let it be used for storing private data.
When you install OSF/1 on the server, follow the advice in the
installation manual to make /usr large enough to hold all the optional
subsets and install them all when you are asked.
2) The remote /usr file system will be mounted read-only on the client
machines. One thing the OSF/1 people (and Ultrix people) did right was
to identify everything on a traditional /usr file system that is
machine specific or needs to be written during normal system operation,
and to put this collection under the /var file system if you choose to
do so during system setup. This means that every OSF/1 system, both
client and server, must be configured with a separate /var file system.
In comments to the first draft of this some people have said that rather
than having a separate /var file system they prefer to put /var under
the root. My feeling about this is that although this will work, I like
to treat the root as a static file system that doesn't change in size
(except for /tmp). Given the various error message files, spool files,
and other temporary files that OSF puts in /var this doesn't seem like
a good idea but it would certainly work.
3) When upgrade time comes around you accept the fact that you'll
have to do a complete installation. You won't be able to do an
incremental upgrade. These days, with CDROMs as a common distribution
medium, the length of time it takes to do a complete installation is
much less than it used to take when tapes were ubiquitous. Plus, my
unprovable feeling from reading various newsgroups and mail lists
is that bizarre problems arise when doing upgrades because the upgrade
scripts can't anticipate all the equally bizarre ways people have setup
their environments.
There are two sets of modifications you have to make - one to the NFS
server machine that will be exporting /usr, and the other to each client
machine doing a remote mount of /usr. I'll show these separately. You
must do the server first.
On The Server:
1) Make sure the server is configured to have /var as a separate file
system. If you didn't do this when the server was first setup you're out
of luck and you'll have to reconfigure it from scratch.
2) Add the following entry to the /etc/exports file.
/usr -root=0 -ro [hosts]
where [hosts] is a list of clients that are permitted to mount /usr
from this server. Ordinarily the '-root=0' would be dangerous but since
you're exporting /usr read-only, there is no danger. The reason why
'-root=0' is necessary is because /usr/sbin/siainit needs it.
3) (OSF 2.X only) If you run xdm on the client machines, create the directory
/var/X11/xdm and modify /usr/lib/X11/xdm/xdm-config so that the
first four lines are as follows:
DisplayManager.errorLogFile: /var/X11/xdm/xdm-errors
DisplayManager.authDir: /var/X11/xdm
DisplayManager.pidFile: /var/X11/xdm/xdm-pid
DisplayManager.keyFile: /var/X11/xdm/xdm-keys
In /usr/lib/X11/xdm/Xkeymaps, modify the line near the top of the
file that says
/usr/lib/X11/keymap_default
to
/var/X11/xdm/keymap_default
The idea here is that any file that is created by a client when
it starts xdm must be placed somewhere under /var.
Ironically, DEC changed the contents of xdm-config in OSF 3.X so
that almost all the DisplayManager resources point to locations
under /usr/var/X11/xdm, making the above step unnecessary.
While this is a nice change, it also means that any cluster-wide
xdm configuration changes must be made on each system in the cluster
instead of in just one place, as in OSF 2.X. Had I been in charge I would
have made xdm-config look like my modified OSF 2.X version.
My approach here would be to put all the files that the client needs
to write to under /var and all the rest, which can be considered
read-only, under /usr/lib/X11/xdm where they would all be shared.
Clients:
1) When installing OSF/1 on a client make sure you select the System
Management option and configure your disk(s) to have partitions for
root, swap, /var, and /usr. Of course, you can have more partitions
than these. On a system with a single RZ26 and OSF 2.X I use the
following partition table:
size offset fstype [fsize bsize cpg]
a: 95760 0 4.2BSD 1024 8192 16 # (Cyl. 0 - 119)
b: 282492 95760 unused 1024 8192 # (Cyl. 120 - 473)
c: 2050860 0 unused 1024 8192 # (Cyl. 0 - 2569)
d: 105336 378252 4.2BSD 1024 8192 16 # (Cyl. 474 - 605)
e: 0 0 unused 1024 8192 # (Cyl. 0 - -1)
f: 0 0 unused 1024 8192 # (Cyl. 0 - -1)
g: 0 0 unused 1024 8192 # (Cyl. 0 - -1)
h: 1567272 483588 4.2BSD 1024 8192 16 # (Cyl. 606 - 2569)
When I installed OSF 3.X for the first time I found that this layout
didn't leave a big enough root partition. So, I'm using the following
on my OSF/1 3.X systems:
# size offset fstype [fsize bsize cpg]
a: 150024 0 4.2BSD 1024 8192 16 # (Cyl. 0 - 187)
b: 282492 150024 4.2BSD 1024 8192 16 # (Cyl. 188 - 541)
c: 2050860 0 unused 1024 8192 # (Cyl. 0 - 2569)
d: 105336 432516 4.2BSD 1024 8192 16 # (Cyl. 542 - 673)
e: 0 0 unused 1024 8192 # (Cyl. 0 - -1)
f: 0 0 unused 1024 8192 # (Cyl. 0 - -1)
g: 0 0 4.2BSD 1024 8192 16 # (Cyl. 0 - -1)
h: 1513008 537852 4.2BSD 1024 8192 16 # (Cyl. 674 - 2569)
When you are asked which of the layered products you want to install,
choose MANDATORY SUBSETS ONLY. There is no need for anything else since
the server machine will have a fully populated /usr.
Make sure the client is configured to have /var as a separate file
system. This is critical.
2) Make the following changes to /etc/fstab:
Add
/usr_at_server /usr nfs ro,soft,intr,bg 0 0
to the point in /etc/fstab after all the local file systems have been
mounted.
Modify
/dev/rz3h /usr ufs rw 1 2
to
/dev/rz3h /usr ufs xx 1 2
3) Edit /sbin/rc2 and /sbin/rc3 and comment out the following lines:
# Just exit if /usr not mounted.
if [ ! -d "/usr/sbin" ]
then
exit
fi
4) Move /sbin/rc3.d/S20nfsmount to /sbin/rc3.d/S02nfsmount.
This will cause /usr to be mounted just after the network is initialized
but before any other important commands from /usr are executed.
5) Modify /sbin/rc3.d/S12route to change all references of /usr/sbin/routed
to /sbin/routed so that your client can run a version of routed that
doesn't rely on any of the shared libraries from /usr which isn't
mounted yet. You can do this with any editor. Once you've done this you
need to rename S12route so that it is executed before S02nfsmount. What
I did, since I don't use quotas, was to rename S12route to S01route.
Although this works, I get error messages from S01route complaining
about commands like 'head', 'ps', and 'sed' not being found since /usr
hadn't been mounted yet. You can ignore them since they aren't really
needed yet.
This step is actually not necessary if the machine serving /usr is on
the same subnet as the client you're configuring.
If you're running gated then you're out of luck because I don't know of
a version of gated that comes from DEC that has been statically linked.
6) Move /sbin/rc2.d/S00savecore to /sbin/rc3.d/S06savecore,
and /sbin/rc2.d/S35streams to /sbin/rc3.d/S07streams.
This is necessary so that the executables from /usr run in these scripts
are present.
7) Reboot. You should see that /usr is mounted from the server machine
and that everything that ran before during startup time still runs.
Once you're up in this environment you shouldn't see any difference
between having a local /usr and having the remote /usr except possibly
for slightly slower access times for files on /usr. We use FDDI a lot
here and accessing /usr over FDDI feels as fast as having a local /usr.
Once you feel confident that this arrangement is working then you can
run 'newfs' on the original /usr file system and use all the space for
whatever you want. Don't forget to change the 'xx' in your /etc/fstab
file to 'rw' so that this file system will be mounted automatically.
Dataless Support in OSF/1 3.X
Even though I had had some informal off the record conversations with
some OSF/1 engineers about "dataless" environments on OSF/1 3.0 I was
very surprised when I finally saw what they did. In a word, they merely
changed the 'D' in the formally "Diskless Management Services" to "Dataless".
In this section I'll contrast DEC's approach towards a dataless
environment ("DEC dataless") with mine ("Forrest dataless").
The primary difference between the two approaches is clearly shown in
the first page of Chapter 8 in the "Sharing Software on a Local Area
Network" manual that comes with OSF/1 3.0. DEC clearly states that in
DEC dataless the server system maintains the root, /usr, and /var file
systems for all clients. This means that a client's local disk is only
used for swapping. (Ironically, this isn't far from how a local disk is
used under VMS in a Local Area Vaxcluster.) In this day and age, I'm not
convinced that this is such a good use of a local disk, especially when
you can buy a 340Mb SCSI drive for $250 with no trouble at all (except
from DEC).
I think DEC dataless can be a waste of disk space and network resources.
DEC dataless requires that every file access not satisfied by the local
memory cache will take place over the network. The big unknown here is
how much the new client caching code in NFS V3 will help. Perhaps a
better approach would have been to allow system managers to specify
which file systems will be local and which will be served remotely.
Then, each site could select the combination that best fits their
network capacity. As it is, I can see how DEC dataless could actually
make things worse.
Another unexpected feature of DEC dataless is that the /usr partition
used by all the clients is different than the /usr used by the server
itself. I see absolutely no point in this. Since DEC dataless correctly
has all the clients share the same /usr, why not go all the way and
have the server use it too?
Perhaps the worst feature of DEC dataless is that the DMS server and all
DMS clients must be within the same subnet. Today's routers are fast
enough so that there is no inherent reason for this requirement.
Indeed, system managers often have no control over which subnets their
systems land on so this requirement seems especially harsh.
Another drawback to DEC dataless is its complexity. In fact, my first
reaction on reading the DMS chapters is how DEC managed to make
something simple into something complex, and to take 50 pages to
describe it.
(There appears to be a major editing error on page 8-4 in the
Sharing Software on a Local Area Network where the text in the
first paragraph describing Figure 8-2 doesn't correspond to what
is actually in Figure 8-2. There are many other editing errors
and sections that could be clairified in the DMS sections of this
manual.)
I do recognize that DEC's approach can result in a more mechanized
setup procedure, although I think that using a disk cloning technique
to set up a new system can also reduce the effort in Forrest dataless.
If I ever get some time, I plan on researching this.
==============================End of Article=================================
Received on Wed Feb 22 1995 - 22:02:05 NZDT