As someone who is always looking to maximize performance and efficiency from my computers, I was delighted to have stumbled upon a program named Distcc a while back. According to their website distcc is a “fast, free distributed c/c++ compiler.”Distcc is brought to you by the same people who created and maintain Samba, the open-source program that allows Unix-like operating systems to act as Windows file and print servers. Distcc is small, simple to use, and works on a variety of Operating Systems. Distcc can speed up compilations in a near linear fashion. For example, tripling the number of computers in the build will nearly triple the speed of the build. It even includes a Gnome-based graphical monitor that will show you which file is being compiled on which machine. I will take the remainder of this article to cover installation and some basic uses of this amazing free software.
Installation and Configuration
Distcc provides binaries as well as source code for many available versions of Unix and Linux. Binaries are provide in RPM and DEB format for easy installation for both Redhat and Debian based systems. Installation from source code is simple as well with the traditional ./configure, make, and make install commands. Each server machine will need to run the distcc daemon. Typing the command distccd is all that needs to be done. Each client machine will need to know the names of the servers. This is done by exporting the environment variable DISTCC_HOSTS. This can be done by typing export DISTCC_HOSTS=”localhost server1 server2 server3′ at a terminal prompt. This can be made permanent by placing the same line in a file that is read at startup (/etc/profile or .bashrc for example).
Performance
To test the performance of distcc, I decided to compile two of the most popular open-source projects. The first and most important is the Linux kernel itself. The second is Wine, the open-source project that allows some windows applications to be run on Linux. All compilations were done in run level 3 (network services running without a graphical user interface). I timed all the results by preceding the compilation commands with the time command. The number reported is the real time between invocation and termination of the compile. The test machines were both running RedHat Linux 9.0. One machine was a 900MHz AMD Duron with a 5400RPM hard drive and 256Mb RAM. The other was a 1.2GHz Duron with a 7200RPM hard drive and 256Mb of RAM.
To test the compilation of the Linux kernel, I first downloaded kernel 2.4.22 from kernel.org. I then unpacked the archived and ran through a standard configuration for my system. I then ran make BzImage without using distcc and timed the results:
9 minutes, 29.110 seconds
Next, I removed the kernel directory and rebooted the system. I then unpacked the archive once again and configured the kernel with the same options. This time I compiled the kernel with make -j8 bzImage CC=distcc. The results were impressive:
3 minutes 44.11 seconds
Using distcc gave a 61% performance increase. Let’s see what it can do for Wine.
I downloaded Wine-20031016.tar.gz from the download section of winehq.org. I unpacked the archive, ran the configure script, and then the make command. The results without distcc:
32 minutes 15.580 seconds
I once again removed the archive and rebooted the system. I unpacked the archive and issued the command make -j8 CC=distcc. Once again the results were impressive:
11 minutes 57.344 seconds
This time we see 63% improvement in performance.
Conclusion
Distcc is one of those little useful programs that you may have never heard about or explored. As you can see, it is a breeze to set up and can save a tremendous amount of time if you have a LAN and need to compile a program from source. With the emergence of source based distributions such as Gentoo, there has never been a better time to have a tool like Distcc at your disposal. More information can be found at distcc.samba.org.
About the Author
Paul Virijevich has been studying and learning open-source technologies including GNU/Linux since 1996.
Well that is if they have more than one Linux box. Even then you don’t need another Linux box all you do need is a Knoppix CD and a computer.
now, if tehy can build that into say kDevelop or what ever Gnomoe IDE they are coming up with, you will have something that competes will with Xcode.
how does it do the distributed communication? zeroconf? that would be sweet.
I use it constantly, it shaves literally hours off of KDE builds — and when you’re packaging (ie, rebuilding it over and over again), that’s an amazing amount of time saved.
Also, for those who didn’t know, the much-touted automatic distributed compile feature of XCode (Apple’s new developer tools for Mac OS X) are actually just distcc under the covers, with a patch to discover nodes through rendezvous.
The performance numbers given are not what I think an everyday user would see unless he has seven computers. <–note -j8, which equals number of computers + 1. But, undoubtedly distcc does increase compile time dramatically.
That is cool ability to do this. I went to the site and saw this “distcc is not supported on any SCO operating system” which I think is very funny. I am going to try this when I get my x86 computer.
Anybody know of a good distcc monitoring program? Maybe one that works with xCode’s version?
I’d like to suggest you using ccache, it caches then reuses the .o files in a safe way. Try it, it’s very usefull when packaging KDE, Mozilla, etc.
I’ve been using distcc at work now for a couple of months across 4 dual processor machines (733p3, 866p3, 2x1ghz p3). Definitely makes the g++ compiles tolerable again.
Thanks for the suggestion on ccache, I’ll definitely try it out at work…how would you run this in conjunction with distcc?
Something like:
$ ccache distcc g++
???
And, as it is a front end to GCC, it might be able to work with GCJ, and cut my ever growing Java compile times down.
I like.
A couple tips for those with large heterogeneous clusters available to them.
Run discc on the fastest node in the cluster as scalability drops off when the head node becomes saturated pre-compiling (cc0) source files. It farms out the precompiled code to the host nodes which finish the build to object files (cc1).
Here are some build comparisons when farming a build out to eight dual athlon 2.1Ghz boxes.
Linux Kernel (make bzImage)
gmake, 1 thread : 244 seconds
gmake -j3 : 126 seconds
gmake -j18 distcc : 40 seconds
gmake, 100% ccache : 31 seconds
Science codes I’m working on
Rational clearmake : 607 seconds
gmake -j3 (parallel build) : 242 seconds
gmake -j18 distcc : 140 seconds
gmake, 100% ccache hits : 136 seconds
the comparison made is unfair. in fact it is almost unreasonable to assume that 2 computers will give a 60+ % performance increase. to compare it faily, compare the results between
make -j8
make -j8 CC=distcc
there are several reasons why the make -j8 is faster than the plain make, just starting at the fact that all source files need to be loaded into memory from disk, and disk is VERY slow. so if you can read in several files at once, you have performance gain
furthermore i would like to see a comparion with ccache, and with distcc + ccache . what kind of cumulative improvements do you have in that case.
is there a way to use distcc/ccache while rebuilding rpms
from source?
To compare it fairly, he should have used 2 similar machines instead of comparing a 900MHz Duron with the same Duron PLUS a 1.2 GHz Duron.
“One machine was a 900MHz AMD Duron with a 5400RPM hard drive and 256Mb RAM. The other was a 1.2GHz Duron with a 7200RPM hard drive and 256Mb of RAM.”
two different machines isn’t unfair since they are still relatively close together in perforance
Used to Delphi or FreePascal compilers, C/C++ compilers result very slow. Delphi users simply wouldn’t understand the need for a distributed compiler. Even huge projects (millions of lines) get ready in very little time. If you are just modifying a single source file, then it takes two or three seconds to make and execute from the IDE.
I think that Visual Studio and some other C/C++ compilers have “precompiled headers” and that accelerates the process a bit. But it seems that there’s something at the language design level that slow down compilation. Otherwise I couldn’t understand why nobody got it right.
gcc 3.4 will have precompiled headers, largely contributed by Apple, should speed up compiling, especially for C++.
Incremental linking would also help alot, though I havn’t seen any plans for that for binutils.
Yup, using Delphi right now and the compile times are almost non-existant. Great language too. Not as clumsy or random as C & co., an elegant language from a more elegant time.
I find distcc to be very usefull. But when compiling KDE and KDE-Apps it isn´t used when you configured them with –enable-final. This configure switch uses a lot of memory but speeds compilation more than distcc.
C++ is a much more complicated language than pascal.
@Daniel. c/c++ are systems language. Not a little gui front-end language like Delphi. I guess c and c++ is just a little to complicated for you. It’s alright, you can stick with Delphi and VB.
(disclaimer: I use C++ mostly)
Do you really think that people criticize C++ design only because it is too complicated for them ? You are just the next person in loong line of people that consider themselves cool just because they were *so smart* to learn C(++) and scorn those who use some saner language …. LOL
Is the binary produced using distcc exactly the same as a binary produced using gcc on a single machine?
Apple’s XCode uses distcc for it’s distributed compile feature. XCode finds other machines running XCode through RedezVous and compiles it through distcc.
the only complaint i have with c++ is the compile times.
i can compile a c programm of the same size in less then a 1/4
of the time it would take to compile the c app
They save time compiling, but lose a lot writing code…
Dear Roy,
i don’t know what made you say what you said but it just goes to show how little you know.
c++ has very few ( significant ) features over delphi, like operator overloading, templates.
besides those, delphi has everything and can build very complex systems. the complexity of learning templates and op. overloading is really nothing once you know the rest.
the fact of the matter is, i compile constantly over 1,000,000 lines of code with delphi in less than 10 seconds. try to show me that with c++.
and again, dont dismiss other people because you think that knowing what templates are makes you smarter
This guys really deserve some pizza.
How to send:
http://de.samba.org/samba/ftp/docs/faq/Samba-meta-FAQ-2.html#ss2.11
🙂
I use distcc on my desktop (AMD 1600+) and my laptop (500MHz Celeron) and monitoring on my desktop using the nifty gnome tool I noticed that distcc tends to send jobs to the laptop leaving the desktop nearly idle (a job every once in a while gets shipped to it) – which is just insane since it slower than just compiling on the desktop in the first place.
I thought the idea was to have the faster machines do the bulk of the work, not leave the actual machine that needs the compile job done as idle as possible…
Is there some way to tell distcc to stop this foolish behavior ?
>>I thought the idea was to have the faster machines do the >>bulk of the work, not leave the actual machine that needs >>the compile job done as idle as possible…
>>Is there some way to tell distcc to stop this foolish >>behavior ?
I would say that this is by design. If you’ve got your workstation and a bunch of PCs, you’d want to offload the compile so that you can continue to do other things on your workstation. Your setup (2 pcs) isn’t ideal to realize the benefits.
> Is the binary produced using distcc exactly the same as a binary produced using gcc on a single machine?
The only requirement is that the machines in the cluster have compatible c/c++ compilers. Ideally they should have exactly the same compiler. I basically have gcc-3.3.2 installed in /usr/local on the systems and links set up for gcc-3.3.2 and g++-3.3.2 in the path.
All the preprocessing and linking is done on the master machine so no libraries need to be on members of the cluster.
If you have a read at http://distcc.samba.org/faq.html#gets-slower it’ll tell you to put the fastest host at the beginning of $DISTCC_HOSTS. That should resolve your problems.
Has anyone had good luck getting make release to work with distcc? Mine always fails during the build.
tia
I had the error “distcc not found” when trying to build a world. I think the buildworld process just sets another PATH. I just did: ln -s /usr/local/bin/distcc /bin/distcc and it worked. I installed distcc from ports.
I started with QBasic, went to boyscout camp and saw someone using C. i still used QBasic for a while then finaly got my own C compiler. used that for a long time (didnt know what was happening until i ran into quake2 gamecode source). i took a class for pascal and i felt more comfortable in C, pascal had many cases where it didnt make any sense (like some End(or whatever it was) without a Begin), also why have those long names when ‘{‘ ‘}’ work just fine? so now im converting my engine into C++ and im looking into a LISP like language for scripting.
anyway my point is, program whatever you want in whatever you want and i wont care.
Being a source distribution, Gentoo’s portage have been taking advantage of distcc and ccache for quite a while. They even have this neat utility called distcc-config to configure it with a breeze.
I must warn people that it sometimes can slow things down if you add a slow volounteer. I tried once to throw an old 486 in the pack, but the box is just to slow, which eventualy leads the main box waiting for the job. Even after my P3 had compiled severals objects, it would stall waiting for the single object compiled by the 486.
I have similar issues with my P1. So you can’t be wrong if you have several boxes that runs at the same clock speed, but care and judgements is needed when mixing boxes that show a big speed differences. For myself, I keep the P1 out of compilations, so even the P1’s compilations are always delegated to faster remote machines, globally it often ends up with faster compilations than if the p1 is part of the volounteers.