FreeBSD Week: Interview with Robert Watson

Robert Watson is a member of the FreeBSD Project’s core and security-officer teams, and founder of the TrustedBSD Project. For his day job, he is a Research Scientist in the network security research group at NAI Labs, studying operating and network security issues. His primary contributions on the FreeBSD Project come in the form of security enhancements to the system; the TrustedBSD feature set arriving in FreeBSD 5.0 will include file system access control lists, mandatory access control, and support for fine-grained privileges. DARPA is now funding a FreeBSD security research and development project at NAI Labs, and they also sub-contract to a number of independent developers in the FreeBSD community to complete that work. Read more for our exclusive interview with Robert.1. What is new in FreeBSD 4.5?

Robert Watson: 4.5-RELEASE is a minor version increment, meaning that while there are a large number of new features, bug fixes, and other types of improvements, the focus has been on refining the 4.x product line. Users can peruse the release notes at their leisure, but the highlights include:

– Improved auto-tuning of system resources

FreeBSD 4.5 does a far better job at self-tuning than previous versions, tuning default system limits better to system memory, and doing a better job at keeping to those limits.

– Network performance improvements and tuning

Tuning changes have been made to the system to improve performance in high bandwidth environments; several bugs relating to TCP performance have been fixed. Performance tweaks have been introduced to better support environments with many IP addresses on a single host. There are also observable performance improvements in a number of network drivers.

– Denial of service resistance

Early results are now in from the DARPA and NAI Labs-sponsored security work. FreeBSD 4.5-RELEASE includes a syncache and SYN cookies, greatly increasing resistance to common denial of service attacks. Much more to come in this space over the next 12 months.

– General hardware support improvements: drivers for more devices, drivers for larger devices (48 bit ATA, etc).

– File system performance enhancements

A wide variety of file system performance improvements are present in FreeBSD 4.5. This includes enhanced directory performance through an improved layout algorithm, more effective caching of directory data, as well as performance optimizations for large directories. Also, soft updates, which improves both performance and safety of the file
system, is now enabled by default during the install for most file systems.

– Stability improvements

There have been a large number of VM, UFS, and NFS stability fixes that should result in improved behavior under high load.

2. Is this release the last before version 5, or there is an additional one to be expected?

Robert Watson: There will be at least one more minor release, 4.6, before 5.0-RELEASE, which will consists of a similar class of changes: performance and stability enhancements, and a conservative set of feature improvements. As part of the road towards 5.0-RELEASE, we’ll also make at least one developer snapshot available, to allow early adopters to experiment with the new technologies in 5.0, as well as broaden the testing base.

Once 5.0 is out the door, we’ll continue to provide a high level of support for the 4.x branch, including at least one minor release on 4.x. 5.0-RELEASE will include a number of broad architectural enhancements that may take some time to settle, so some consumers may wish to stick with 4.x for the time being.

3. Please tell us about the performance of the Java VM featured in 4.5.

Robert Watson: The feature here is the nativeness rather than the performance. The FreeBSD Foundation has negotiated with Sun to release a natively compiled version of the JDK for FreeBSD. Right now, it’s in final conformance testing, and will probably be available shortly after 4.5-RELEASE. This will be a dramatic improvement for those doing Java development on FreeBSD, bringing greater stability, better support for the native feature set (such as JNI with native libraries), and probably some amount of performance also. It will also permit the JDK to grow stronger native support for the up-coming 5.x features, including fine-grained threading through Scheduler Activations, a work in progress for the 5.x branch.

4. What is the best new feature to be found in version 5, in your opinion?

Robert Watson: Personally, I’m looking forward to two of the major new features that will be production-ready by 5.0: fine-grained SMP support in the kernel, and support for a variety of new security services.

Support for fine-grained SMP in FreeBSD 5.0-CURRENT has involved a fundamental redesign of the FreeBSD kernel. For those interested in the details, it has involved a change in the threading and locking model of the kernel, from an older co-thread model to a fully preemptive kernel based on modern locking primitives. Previously, SMP was supported with a single “Giant” lock, which limited the kernel to executing on a single CPU at a time; this realized some of the performance gain of the SMP computing architecture, but presented a substantial bottleneck for kernel-intensive loads. In 5.0, a much larger set of relatively fine-grained locks will be present, allowing concurrent execution of the kernel on many CPUs at once. To do this, much of the older code of the system is being carefully analyzed and modified to clean up previously under-specified behavior, and a number of subtle bugs have been identified and fixed. The end result of “SMPng” is a much cleaner and more efficient kernel architecture, with
careful documented assumptions. This work has been a serious undertaking, but I think it will be well worth the wait.

On the security side, I’m very excited about the TrustedBSD features that will be present in FreeBSD 5.0-RELEASE. This includes basic new security functionality, such as file system ACLs, but also a number of advanced features such as support for Mandatory Access Control. We’ll be shipping support for Multi-Level Security, a fixed-label Biba integrity policy, Low Watermark (LOMAC) floating label integrity policy, and an experimental Type Enforcement implementation. To do this, we’ve developed a new security extension framework for FreeBSD 5.0, allowing the development and incorporation of security features at a much lower cost than the previous “hack and slash” security extension approach. A great deal of effort has also been put into improving the assurance of the FreeBSD system:
abstractions have been improved, security functionality has been isolated and carefully documented, security regression tests have been introduced, and more. Users of existing security features in FreeBSD, such as jail, will notice that the degree of integration is tighter, and level of functionality higher. Making this functionality available as part of the base FreeBSD system will provide this broad suite of security features to FreeBSD users in a manner not previously possible using widely available open source systems.

5. Does the FreeBSD project have any plans to target “the desktop”, or the server market is the only focus?

Robert Watson: As with many other operating system cores, you could consider FreeBSD to be a building block for constructing larger or more complex systems. FreeBSD has been successfully used to support large server clusters (over 4000 FreeBSD servers are used in the Yahoo server environment), as well as workstation environments. An example of success for FreeBSD as the core of a workstation environment is Mac OS X: Darwin successfully combines elements of FreeBSD, Mach, and NeXTStep, along with the Aqua display environment to produce a strong desktop product. As with Linux, you can combine FreeBSD with the XFree86 window system, and a windowing environment of your choice to build quite a reasonable desktop environment.

Users of recent versions of FreeBSD will have noticed a dramatic improvement in sound support quality, and this improvement has continued in 4.5-RELEASE. Likewise, integration of the KDE and GNOME desktop environments has improved. Obviously, there’s more work to do, but I think it’s easy to demonstrate that the FreeBSD Project has a continued commitment to the desktop. This will only improve as we move forwards.

6. Are there plans to re-write the installation process in a more graphical or easy to use way?

Robert Watson: Yes. There’s on-going work on a new installation and management framework called libh. This framework provides a configuration and installation back end that may be plugged into a variety of UI front ends, including a traditional text based user interface, and graphical front ends based on X11. While this is a work in progress, we have high hopes that this framework will allow us to improve the user experience not only at install-time, but also for day-to-day management activities. Look for more news in this space.

7. What do you think about the Linux advancements recently? New VM, scheduler, preemptive patches…

Robert Watson: I think these are all exciting improvements to the Linux system, and many mirror changes that are already present in FreeBSD, destined for FreeBSD 5.0, or things that we should definitely be doing :-).

As you know, the FreeBSD Project has long prided itself in having a scalable and tightly tuned high-performance virtual memory system. Part of the recent contention in the Linux community over virtual memory has been whether or not to adopt more aspects of a Mach/FreeBSD-like VM system. I think it’s fair to say that the VM story isn’t finished yet: we’ll be eagerly watching for new VM techniques in the Linux space that we can learn from.

The scalability improvements in the queue for the Linux scheduler are ones that will probably be required to improve FreeBSD scheduler behavior also. As support for SMP improves, these problems will become more visible, and reducing contention in the presence of scheduler activations (a more scalable threading technology that Linux has not yet adopted) will help to some extent, but not entirely. I encourage the Linux community to publish the results of their research and experimentation in forums such as FREENIX and USENIX so that the broader open source and operating system communities can learn from their experiences.

Support for a preemptive kernel is actually part of our on-going SMP architecture. There are two important parts to this work: (1) redesigning the kernel to support fine-grained locking so as to improve the concurrency and reentrance of the kernel, and (2) increasing the opportunities for switching the active task to reduce latency on
latency-sensitive transactions. For (2) to be really effective, you need to have first done (1). We actually have a number of FreeBSD 5.0 systems running with fully or partially preemptive kernels, and this should be a bullet feature for 5.0 when it ships later this year. We’ve certainly kept an eye on the work on the Linux side, and likewise hope to see that work described in detail in publications. I personally follow the Linux-kernel mailing list, but know that many FreeBSD developers don’t due to the volume of messages involved (no doubt this is true of Linux kernel developers regarding FreeBSD mailing lists :-).

Another feature that you don’t mention is the on-going work on improved support for access control in Linux. We have been watching this work closely, as we have related work going on in FreeBSD and TrustedBSD also. While the FreeBSD and Linux kernel implementations differ substantially, I think it’s important that we continue to provide application developers with a consistent and standards-compliant development environment. As this new functionality becomes available, it will be important for FreeBSD and Linux security developers to work hard towards this goal, and collaboration may be one of the most important ways to make that happen. No one expects the same kernel modules to run on both FreeBSD and Linux, but it would be nice if security-extended versions of applications could run on both easily.

There should be an exciting assortment of Linux and BSD-related papers at FREENIX this summer. Likewise, for those interested in FreeBSD and other BSD-related work, I recommend the BSD Conference hosted by USENIX in San Francisco next month. Both will provide insight into the innovative and fast-paced work going on in the open source operating system communities. I hope to see you there.