Today we feature a very interesting interview with well known *BSD hacker Matthew Dillon over his latest project, DragonFly BSD (also known for his Linux kernel contributions, Amiga C compiler hacking back in the day and the Backplane distributed database). Matthew discusses DragonFly’s status, goals, the overall BSD platform, innovation, and more. Update: Added one more question at the end of the Q&A.
1. Please tell us about the general status of DragonFly BSD.
Matthew Dillon: The project has been going very well. We’ve primarily been focused on the ‘guts’ of the system during this initial period, and you can get a fair idea of the work that has been accomplished so far by looking at the Diary page on our site.
Most of the work so far has been to operating systems internals. The
work has been a combination of new work, like our light weight kernel
threading core, plus selective backports from FreeBSD-5 to keep the
system’s device drivers up to date (e.g. such as the USB subsystem).
From a userland perspective we have maintained a FreeBSD style
environment, so DragonFly basically runs everything that FreeBSD-4.x
can run. The packaging system probably won’t be done until the second
release so we are at the moment leveraging off of FreeBSD’s ports system
for user apps. Everything you’d expect of a BSD system (X, mozilla,
etc) is available to DragonFly users.
The first release is slated for some time in mid-June, hopefully
before the USENIX Technical Conference. That will be the 1.0 release.
We’ve been fairly careful to maintain as high a level of reliability
as possible during development and I think we’ve done a pretty good job
meeting that goal. The first release is intended to be more of a
technology showpiece then an integrated end-user platform.
2. Are you using any bits and pieces from FreeBSD-5, or you only strictly importing/exporting to FreeBSD-4 codebase?
Matthew Dillon: DragonFly began as a fork off of FreeBSD-4, because that was the most
reliable starting point and because we wanted to do major core pieces
of the system quite differently from the direction FreeBSD-5 took. For
example, we are focused on more of a compartmentalized threading model
to scale to SMP rather then the mutex model that FreeBSD-5 has chosen
to use. But the FreeBSD-4 codebase is of strictly limited utility as
a source of new code and maintainance updates. FreeBSD developers
are doing nearly all new coding in the FreeBSD-5 branch.
So, basically, we are doing the major core pieces of the OS differently,
such as our significantly evolved threading and messaging subsystems,
but we are also maintaining a FreeBSD-5 compatible (or mostly compatible)
device driver API in order to be able to bring in all the excellent
device driver work that has gone into FreeBSD-5. It’s simple logic,
really… we don’t have the manpower to be able to accomplish both
our infrastructure goals *AND* be able to maintain pace with new PC
hardware at the same time. This methodology allows us to proceed on
both fronts by focusing our own new work on the infrastructure and
bringing in FreeBSD-5’s device driver work. This isn’t to say that we
don’t do some of our own DD work, but the vast majority of it is
take from FreeBSD-5 by design.
3. What is the primary goal of dragonfly, servers or desktops?
Matthew Dillon: Both. When it comes right down to it the idea of targeting a system
to the ‘server’ is simply another word for ‘reliability and scaleability’,
and the idea of targeting a system to the ‘desktop’ is simply another
word for ‘out of the box GUI’. It’s not really an either-or deal.
All UNIX systems, including Linux, the BSD’s, DragonFly… basically
use the same desktop components so supporting a desktop environment
comes down to being able to provide integrated solutions that work out
of the box.
It is extraordinarily difficult to make GUIs work out of the box on
PCs due to the wide variability in hardware and peripherals, but at the
same time technology has continued to progress over the years towards
standards that actually make this easier to accomplish. At some point
the standards going in one direction will meet the software going in the
other and systems such as Linux and the BSDs (including DragonFly) will
be able to approach the out-of-the-box compatibility that took Microsoft
billions of dollars of development to accomplish. It isn’t a matter of
if, it’s a matter of when.
4. Do you eye the embedded systems market at all?
Matthew Dillon: It is not a focus. We fully intend to make DragonFly operate in
64 bit environments such as the AMD64 (which is also compatible with
the Intel’s latest 64 bit announcements even though Intel doesn’t like to
admit it), but embedded systems are already very well served by
Linux and NetBSD and we would prefer to focus on technology rather
then platform ports. FreeBSD-5 has had a difficult time supporting
the half dozen or so platforms they are trying to work with and I doubt
we would fare any better, so I am not even going to try. Eventually, if
DragonFly becomes popular enough, embedded systems ports will probably
happen, but it’s probably several years down the line.
5. How big is the DragonFly team currently? Are you happy with the development pace?
Matthew Dillon: Very happy. The main kernel@ mailing list has 241 people subscribed
to it as of now and I have handed out 9 commit bits, with another dozen
or so people submitting patch sets outside of that. We have a good
spread of developers focused on different subsystems. For example,
Jeffrey Hsu is focused on threading the network stack while Joerg
Sonnenberg has been focused on the PCI bus and ATA disk driver subsystems,
which frees me up to be able to focus on technological infrastructure such
as the threading subsystem. It’s about at the limit which I can sustain
and still have time to do significant programming myself, in fact!
6. Are you planning to import some of OpenBSD’s “tricks” for extra security?
Matthew Dillon: We’ve been looking at both OpenBSD’s security features, such as their
system call filters, and FreeBSD’s extended attribute and MAC security
infrastructure. Security is important but it’s no where near the top
of the list. There is a huge amount of other infrastructure that really
needs to be done first before we have the resources to address security
beyond the standard UNIX security model. In fact, a good chunk of the
infrastructure we intend to put in, such as syscall messaging and VFS
environments will be needed to serve as a basis for future security work
so it would be a bit premature to start on the security work at this
early date.
7. What do you think of the bsd terrain today? Are the BSDs doing/achieving enough as a platform to keep up with other Unix/Linux/Windows competition? Do you think that BSDs should re-innovate themselves at all levels in order to “go with the times”?
Matthew Dillon: I believe that the BSD’s are doing quite well as a platform, even though
Linux obviously has the eye of the masses. All the BSD’s have steadily
increased their development resources over time and the Linux phenomenon
hasn’t really dampened it. Keep in mind that developers work on the BSDs
for very different reasons then developers who work on Linux. The BSDs
are all about advancing software technologies into new arenas, while
Linux is all about leverage (which is why you see a very well integrated
security subsystem in FreeBSD-5 and OpenBSD and ten different types of
filesystems in Linux). Even better, nearly 100% of the user application
base developed under Linux compiles and runs natively on the BSD’s (and
people often forget that major pieces of software such as the X windowing
system existed long before Linux came on the scene, running on BSD and
commercial UNIX systems). People often forget that Open-Source means
precisely that… open source code, which means portability across
platforms and operating systems.
In anycase, you have to keep in mind that programmers always have their
own reasons for working on a project, and since programmers are not
consumers the reasoning is always very different from the reasoning
a typical consumer might have in choosing a system. This pretty much
guarentees that the behind-the-scenes interest, economics, and even
politics driving the people who actually work on a project like
DragonFly or *BSD, or Linux, has nothing whatsoever to do with what
you might read in the popular press (which is consumer-centric).
8. Currently the installation of DragonFly is very manual. What plans do you have for the installation procedure?
Matthew Dillon: At the moment the Live CD ISO is targeted towards developers rather then
end-users. We have been discussing installer ideas for months. We
probably will not have anything ‘good’ in the first release, but we
will certainly have something reasonable in the second release.
This is both good and not so good. It’s good in that it means we don’t
have the pressure of having to deal with bug reports from non-technical
end users yet. It’s not so good because, obviously, a good installer
will greatly improve our user base and interest in the project.
9. Do you have any plans to port to PPC and maybe merge some code with Darwin?
Matthew Dillon: The PPC is a good target to port to and will probably wind up being
second on my list. The first port is going to be to AMD64 (which also
means Intel’s latest 64 bit announcement, since it’s compatible with
the AMD64). There is no time frame for that but it will likely be
started after the first release. A PPC port has a likely time frame of
a year or two.
10. What feature on your OS has you very excited? How is Dragonfly going to differentiate from other similar solutions? Is innovation among your goals? If yes, what kind of innovations are you looking into doing?
Matthew Dillon: Well I strongly believe that any project needs to have an unattainable
goal, and our unattainable goal (which I hope actually winds up being
attainable) is to develop DragonFly into a transparently cluster-capable
system implementing native SSI (Single System Image). It is something
that no non-commercial system today can do (the type of clustering Linux
supports isn’t even close to the type of clustering that we have as our
goal, and clustering has never been one of the other BSD’s goals as far
as I can tell).
In the short term, I have become very exciting about our light weight
kernel threading technology, in particular the methodlogy we are
adopting to serialize data access by partitioning major subsystems
into threads instead of serializing data access with mutexes (FreeBSD-5
and Linux use a mutex-centric model, DragonFly uses a thread-centric
model). The reason for this excitement is that it is becoming clear
to us that we can develop very clean-looking, elegant, debuggable,
SMP scaleable software using this model whereas using the mutex model
generally results in much less elegant (even ugly),
difficult-to-debug code. Code complexity and code quality is a very
important issue in any large piece of software and we believe we have
hit on a model that directly addresses the issue in an SMP environment
without compromising performance.
11. What is the status of the Backplane DB’s Apache PHP module? Do you still work on the db full time?
Matthew Dillon: The Backplane DB is virtually a whole project unto itself. Due to a number of business issues with the software we were unable to give it a completely open-source license, and because of that encumberance I felt uncomfortable making an OSS project out of it. Backplane is on the backburner for now, I have been working on DragonFly full time for most of the last year.
Backplane Inc is shopping the database around. It’s a very technically competent multi-master database and replication technology but it really needs the resources of a larger company to advance further.
Raised some very good points about scalability. Is there anyone here who can point me to a Multex vs threadcentric paper comparing the two techniques?
There is some stuff about it on the dragonfly website http://www.dragonflybsd.org. I can’t see how they’re really different to any other sort of mutual exclusion lock. Slightly different sementics perhaps. Result is you can’t have more than one thread in a critical region at a time.
There is some stuff about it on the dragonfly website http://www.dragonflybsd.org. I can’t see how they’re really different to any other sort of mutual exclusion lock. Slightly different sementics perhaps. Result is you can’t have more than one thread in a critical region at a time.
Hmm maybe I’m wrong. Further digging found this:
http://www.dragonflybsd.org/docs/BAFUG1_SLIDES/img8.html
I wonder if he’s going to try to use this model for as much high level stuff as possible?
>Raised some very good points about scalability. Is there >anyone here who can point me to a Multex vs >threadcentric paper comparing the two techniques?
before anything you should know what is a thread : it is a computing unity inside a processus. A processus may have more than one thread running in parallel.
Threads may share some data. What would happened if two thread use the same data at the same time ?? Kaboom !
So you can specify this data as mutex.
Each time a thread need to use the data, it check if the data is locked, if yes it wait, else it locks it for its own use and proceed it. Finaly, it unlocks the mutex so i’ll usable by other thread.
If I understand right, the thread centric approch is to have no data in common, so no locking, but to allow the threads to communicate via some messenging system, so they can share some information.
Hope I am not to far from the truth!
Alex
Thank you for the links, from what it appears, the dragonflybsd team are trying to make BSD simplier to make it possible to add more features without adding more complexity to the equation. Sounds like a great idea and hopefully some other projects pick up on this idea.
Maybe he should contact Genesi, and get his OS to work on Pegasos? 🙂
3. What is the primary goal of dragonfly, servers or desktops?
Matthew Dillon: Both.
dfBSD is heading for the user base since now has nvidia drivers support.
What better than a daemon with sneakers?
A daemon with sneakers and wings
if they are going for desktop use i hope they implemenent some sort of user-level package installation. If they are going to innovate they might as well do it here too.
There is some discussion of a new package management system on the dfBSD home page. From my understanding that there will be some extensive innovation. There home page is definately worth browsing.
dfBSD is heading for the user base since now has nvidia drivers support.
Err, how so? Where?
Matt has been a prolific coder for many years and is an unsung hero in free software.
As a long time FreeBSD user at work I welcome a new option at the precipitous juncture from FreeBSD 4 to 5. One more resource for novel, peer reviewed ideas and implementations.
It will be interesting to see how DragonFly will fare with the growing dominance of linux in the free OS sphere. The linux talent roster is very full and it is difficult to get attention for ideas and implementation. That one value of the BSD projects – a project you can get in on the ground floor.
>>Linux obviously has the eye of the masses
I guess that would be qualified as “geek masses”, cause windows has the eye of the unwashed masses…
😉
informative interview.
Well, it’s using FreeBSD native one with the patches.
http://www.dragonflybsd.org/cgi-bin/cvsweb.cgi/dfports/x11/nvidia-d…
I have been using DragonFly on and off for a while now, and for the most part it’s been very nice for day to day use. It’s fast, and it can handle whatever loads I can throw at it. The scheduler needs work however. Some things that have never caused jerkyness in things I was doing in the foreground (like rebuilding the system in the background etc.) can sometimes be painfull in DragonFly. It’s not consistent, nor am I expecting such from a pre-beta OS, but I thought I’d mention it anyway.
DragonFly also builds slower than does FreeBSD 4 or 5, largely due to the inclusion of two versions of GCC. I’m sure that they’ll drop GCC 2.95.x soon, but until then, it takes just barely more than twice as long as it used to.
They include directions to manually install DragonFly on the Live CD, and I’ve wondered what has stopped them from reworking it into a small handful of shell scripts so that relative newbies could install it more easilly, without having to spend ages waiting for whatever the final implementation will evolve into. It should take someone like Matt maybe a couple of hours (if that!) to do so, and the benefits would be great enough IMO to put the real work on hold for that few hour period.
I am very much looking forward to DragonFly’s first release, and if they have the desire to sell copies of it on CDs, I will most definately buy one.
As far as I knew DragonFly was still using the standard (mostly) FreeBSD 4.x scheduler.
Having followed DragonFly pretty closely for around 6 months, I can’t say I’ve noticed any of these scheduler issues. My p2-450 can buildworld in the background, and if not for limited disk bandwidth I’d never know it was there. It does not skip my music, or anything like that. You might want to make sure you have DMA enabled on your hard drive.
That said, there have been a number of problems that have gotten in my way — the most significant I think being the package situation. Many FreeBSD ports/packages no longer build or install properly on DragonFly. This is an issue that will get worked on soon I hope.
Overall DragonFly feels very snappy to me.
I’m very sure that it could have been the particular snapshots I was using at the time, but that in itself does not mean that the scheduler doesn’t need work. I’d considder myself about the biggest fan of DragonFly there is right now, but that’s not going to stop me from pointing out flaws I find, sporadic as they may be (aren’t pre-release OSs great!
.
As to the ports collection, I’ve had mostly successes, except a few core X related ones. FreeBSD packages so far have offered me no headaches. I would very much like to see a greatly expanded dfports tree however. It will come.
It’s always great to hear about DragonFly, I’m really looking forward to a stable release… Perhaps soon it will replace my FreeBSD 4.9 boxes
Getting the GUI to run isn’t a problem, and has little
to do with the kernel. Oh, DragonFly isn’t just a kernel?
So what, it’s not like they are going to develop their
own entire GUI.
The difference is throughput vs. responsiveness (interactivity). On the “server” it costs you (in throughput)
if you are too heavily weighted towards “interactivity”.
On the “desktop” it’s the reverse.
No, you cannot have the best of both worlds. When I log into
my “server” it SHOULD keep high throughput and my interactive
(shell) process should not get the kind of response it needs
on the desktop.
IMHO!!
There are of course many many areas where better desktop
performance IS the same as better server performance, eg
threads scalability. But my point is that the answer to
the question leaves something to be desired.
No, you cannot have the best of both worlds. When I log into my “server” it SHOULD keep high throughput and my interactive (shell) process should not get the kind of response it needs on the desktop.
Don’t be a dumbass. There are two things that would allow one to have the best performace for both types of systems. You could have more than one scheduler and use whatever one best suits your needs (Linux does this), or you could have (better yet) an adaptive scheduler that modifies it’s behavior as needed.
It is your thinking that leaves something to be desired. There are few things in this world that we cannot eventually do. Having an OS that can work out of the box as either a server or as a desktop isn’t one of those things that we will be denied.
Learn to think.
Its a new project and all new projects deal with those issues, nothing new.
He keeps saying AMD cmpatiible with AMD, when in reality it is the other way around. Get th egacts straight please!
He did get it right; Intel’s new 64 bit line (not Itanium) will be made to be compatible with AMD64, not the other way around.
our unattainable goal (which I hope actually winds up being attainable) is to develop DragonFly into a transparently cluster-capable system implementing native SSI (Single System Image). It is something that no non-commercial system today can do (the type of clustering Linux supports isn’t even close to the type of clustering that we have as our goal, and clustering has never been one of the other BSD’s goals as far as I can tell).
I would suggest a look at http://www.openssi.org/ , which is SSI clustering for Linux. Soon DRBD will be integrated and then you can have a high availability shared root without shared disk hardware. (At the moment, shared disk hardware is still required for HA — but not if you only want a non-HA shared root)
That is off topic here. Also, unlike what is being done with DragonFly BSD, OpenSSI is now, and always will be a tacked on grotesque hack to make Linux do things that it was never intended to do, and will likely never be modified to do correctly.
Please stop trolling here, this is a DragonFly article.
I know that you are just trolling, but hey, you might also be a retard, so I’ll try to enlighten you a little.
The SMP approach being undertaken in DragonFly is essentially lockless, meaning that it works without using mutexes. As FreeBSD 5.x is very heavilly into the fine grained mutex thing, it would have taken much more time and effort for Matt to have implemented the lockless SMP in DragonFly if he had of started with 5.x. Simple logic really, start off with a system that has fewer mutexes to begin with, and you’ll have fewer mutexes that will eventually need to be removed.
YeeHaw!
The jerkyness I spoke of ages ago when this story first came up was fixed today. Matt found some issues in kern_fork.h and a couple other files that was causing newly forked processes to have realtime priority until they were rescheduled. I’ve just rebuilt (for the second time today) and the laggyness that I expereinced before is now gone.
All is well with the world. 😉