USENIX 2005: Unix in Southern California (day two)

This is a report on the second day of USENIX 2005. You might want to start with my report on day one. Day two started off a bit slowly. Oh wait, actually I started off a bit slowly. That could have been from the margaritas last night at La Casa Garcia, a Mexican restaurant a few blocks down the street from the Anaheim Marriott. Good Mexican food is one of the things I miss the most since I moved from California to New York two years ago. Luckily, the food at La Casa Garcia was excellent.

I started off day two with an invited talk about NFSv4. This is an
example of how USENIX excels at bringing together the best of industry
and academia. The presentation was by Spencer Shepler who works on
NFSv4 at Sun and is the co-chair of the IETF NFSv4 working group. You
can’t get much closer to the source when it comes to NFS expertise.

I was a bit drowsy for the first part of the talk, so my notes were
pretty sketchy. Blame it on a lack of coffee. I did manage to pick
up a few interesting tidbits later on in the speech, though.

The fourth version of NFS provides many advantages for distributed
filesharing. Some people tend to dismiss NFS as an obsolete mechanism
or something “only used by Unix geeks”. This is far from the case.
The NFS developers have analyzed the shortcomings of NFS and provided
some excellent improvements. One I am very excited about is the
folding of all the legacy NFS protocols into one protocol running on
TCP. Historically, NFS was extremely difficult to pass over firewalls
because it depended on several protocols and some of those did not
have well-defined port numbers. That has been completely fixed. As
Spencer responded to one of the questions at the end of his talk, all
you need to do to support NFSv4 over firewalls is allow TCP on port
2049. Certainly this is a welcome addition for any network
administrator.

Another new NFS improvement which caught my attention is file
delegations. Traditionally, every file operation by the client
requires a call to the NFS server. Delegations can greatly
reduce this network overhead. The NFS developers recognized that in
many case, files on the server are accessed by only one user. A
classic example of this is the home directory – it’s quite likely that
only the owner of that directory will be accessing those files.

Delegations allow the client to take control of a file. After reading
a file, the client can write, close, reopen, or perform other actions
on the file. The client only needs to communicate with the server
when it is done with the file. Obviously this can provide greatly
increased performance when files are not shared between multiple
clients. One caveat is this will currently not work well over
firewalls because the use of delegations requires a callback.

Other highlights: Spencer’s opinion on traditional NFS file locking
is, “it sucks”. Thus there are a number of improvements to locking in
NFSv4. One of the biggest of course is that file locking is now part
of the one unified protocol and doesn’t require a separate lock
manager. Another is that clients (not servers) now do lock release
when a server restarts. Owner attributes in NFSv4 are now of the form
user@domain instead of traditional 32-bit UIDs/GIDs. This will
provide more flexibility, but will cause headaches for some existing
installations.

One small change that will have a big impact in how NFS evolves in the
future is the new ‘minor version’ support. Previously, each release
of NFS (v2, v3, v4) was a completely new protocol. This model did not
allow for incremental changes. Now going forward new operations and
attributes can be added to NFS without forcing a move to a whole new
major version of NFS.

Basically the talk was a wealth of information about NFS. One of
those ‘USENIX moments’ occurred when the presenter had a question
about Linux NFS support. He just asked the audience because he knew
the Linux NFS developers were there. I’m going to end my discussion
of NFS now otherwise it will take over my whole update for today. I
will note that you can check out Spencer
Shepler’s weblog for the current status of the NFSv4 work.

A few notes on the morning coffee break: first of all, does anyone
really think ‘strawberry-flavored’ cream cheese really tastes like
strawberries?

I overheard a few interesting comments from one of the USENIX
attendees while enjoying my bagel (I wasn’t eavesdropping, I swear!).
He asserted that there has been a fundamental shift in USENIX since
the mid-nineties. During that time, a decision was made to make
USENIX a more academic conference. The result of this has been a
substantial reduction in papers from people in the industry. This has
been a negative result because many of the best papers in USENIX
history came from industry (one example: the original mid-eighties
paper on NFS). The academic thrust has therefore reduced the general
quality of USENIX by shutting out people “in the trenches” making
important contributions in our field.

I can’t really comment on that since I wasn’t attending USENIX back
then, but I think it is food for thought. There is a balance between
academia and industry, and it is possible that USENIX may be too far
over on the academic side. Still, there’s no denying that there are
some excellent papers being presented at this year’s conference.

For the 11-12:30 timeframe I chose the Freenix track and three
presentations on network security and monitoring. These were all
high-quality, but I’m going to concentrate on the one that I found
most intriguing.

The Ourmon network
monitor illustrates how open-source tools can be combined to produce
very sophisticated analysis network analysis. In this case, the
administrators at Portland State University use Berkley Packet Filter
to filter the massive amounts of data going through their DMZ and then
store the data with RRDtool. They wrap this all up in a web interface
so you can see at a glance:

• Do the number of TCP SYNs match the number of FINs? A large
  number of SYNs and few FINs indicates an attack (lots of connection
  attempts that are being aborted).

• Is the number of TCP RSTs
  higher that usual? This could indicate lots of port scanning activity
  (which just makes connections and immediately aborts them).

I was quite impressed with the statistics they are gathering and I
encourage you to check out the realtime data available at the Ourmon
link above. I’ve only scratched the surface.

The big presentation of the day (for me anyway) was on the upcoming
Mac OS X 10.4 release, Tiger. This was presented by Dave Zarzycki,
Senior Engineer with the BSD group at Apple (there was an additional
Apple presenter there but I neglected to get his name). Dave knew his
audience: the whole presentation was about the changes in Mac OS X
from a Unix perspective. It’s clear that Apple is listening to the
Unix world and is generally interested in improving the Unix
experience.

Beyond the usual software updates and bug fixes, Tiger changes some of
the Unix conventions in a very fundamental way. Dave was very
emphatic that Apple doesn’t implement Unix features just because,
“that’s the way it’s always been done”. Instead, they look at the
problem and try to figure out the best way to solve it. An example of
this is launchd, their complete reimplementation of system daemon
management. As Dave says, it’s “one daemon to rule them all”. They
looked at the problems inherent in the current daemon implementation
in Unix. These problems include:

• Daemons are started in various places (init, inetd, rc.d, etc.)
• Daemon writers have to go through a lot of work programming work
  to start a daemon properly (disassociating with the terminal, closing file descriptors, etc.)

• The current model doesn’t give regular users a way to run their
  own daemons easily.

• launch on demand and automatic restarting are not handled very
  well (if at all).

launchd attempts to address all of these issues. It handles all the
‘daemonizing’, so you can simply throw all this code out of existing
daemons. It’s one uniform place to start your daemons, and it
includes automatic restart and launch on demand features. Plus, each
user gets their own launchd so system and user daemons all work in the
same way, and the security issues of one launchd for everyone are
avoided.

The examples really illustrated the power of launchd, particularly for
launch on demand. One example was the handling of repeating jobs via
cron. Traditionally the cron daemon is always running, whether or not
there are any jobs to run. Now, launchd watches
/var/cron/tabs. If there aren’t any files in that
directory (i.e. there are no user cronjobs configured) then cron isn’t
running. When a file does appear in that directory, cron
automatically starts running. This feature helps reduce the OS memory
footprint as daemons with nothing to do just don’t run.

Another big change is in system logging. Apple has introduced ASL,
the Apple System Logger. Similar to what they are attempting with
launchd, ASL regularizes system logging. All logs are now stored in a
database and there are powerful tools to filter and prune those logs.
For example, you can configure the system to prune away messages from
‘chatty’ daemons once a week, while for other quieter daemons you
might want to keep the log entries for a month. The ASL filtering
also occurs in the program generating the log messages, not in some
log filtering program. This reduces the overhead as messages are
turned off before they are even generated.

One of the historical annoyances with Mac OS has always been the
dreaded ‘resource fork’. While other OSes have stored all the data in
a file in one stream, Mac OS has always split a file into two stream,
data and resource. The resource fork stores things such as program
icons and i18n text. This works great when you just use the files on
one Mac or trade files between Macs. However, the moment you start
trading files with other OSes, problems happen. Usually the resource
fork will be silently stripped when the file is transported. The
workaround has always been to encapsulate the file in question, for
example in a disk image or zipfile.

Now there is finally an ‘official’ mechanism to deal with this: POSIX
Extended Attributes. This is a standardized way to add arbitrary
name/value pairs to a file. This is used to support Access Control
Lists (ACLs), which will also be in Tiger (based on the Windows ACL
semantics). It can also store all the resource fork information. One
problem is that tools must be modified to support EAs. Apple believes
they have modified all the command line tools appropriately, including
cp, rm, and tar. Rsync works with the ‘-E’ switch. CVS is one
exception, basically because its just too broken to begin with. This
work is a welcome development and will go a long way towards smoothing
the interoperability issues between Mac OS and other systems.

Any presentation which deviates from “pure unix” is guaranteed to get
a big response in the Q&A portion, and this presentation was no
exception. One big demand was for dump/restore on Mac OS X. This is
where the Apple guys really emphasized they won’t implement a unix
solution without carefully considering the best way to do it. In this
case their response was that the existing Apple System Restore tool is
the replacement. Some of the (somewhat cranky) attendees responded
there were various problems with that, including the lack of
incremental backups with ASR. The main problem, however, seemed to be
that no one even knew that ASR existed.

I discovered at the afternoon coffee break that there was again no
coffee! The signs on the lobby said, “Coffee Break” so I was quite
confused. Memo to Marriott: if you’re giving me a coffee break, there
better be coffee. Luckily, again, the Starbucks saved the day.

After the break I attended several short paper presentations. These
are limited to 15 minutes and provide a good quick overview of the
‘state of the art’. However, I found them to be so theoretical that I
didn’t get much from them. I then retired to finish this report.

In conclusion, today was a productive day at the conference. The
NFSv4 and Mac OS X presentations stood out as excellent examples of
the best of USENIX. Tonight I will be attending the 30th Anniversary
USENIX reception, where the alcohol and reminiscing will be flowing in
equal portions. Stay tuned for the next and final update tomorrow.

About the Author:
Phil Hollenback is a System Administrator at a financial company in Manhattan. When he’s not taking care of computers, he
spends his time skateboarding and working on his website.