distcc, VirtualBox, NAT

The following scenario is quite typical. Your family member, spouse, or coworker has this fantastic, powerful, brand-new multicore machine which, for some reasons, has to run Windows or Mac OS X. On the other hand, you probably have some el-cheapo laptop which is also wonderful but it lacks the processing power to do heavy-duty build and compile. Now, what if you can exploit the other box for a distributed compile? Especially if the powerful box seems to be idle once a while, doing nothing.

Many of you already figure it out the solution: let's install Linux on a virtual machine and use tools like distcc or icecream. In fact, this is quite easy to do. I managed to find out the details, even if I have only little idea about network stuff.

Side note: I don't see why icecream would not work. But since I did try distcc, that's what I wrote below.

For the following explanation, refer to the above exemplary network setup. Basically I run virtualized openSUSE inside VirtualBox. You don't even need a full-blown graphical system, you can create a customized openSUSE installation (no X11 but with some development tools like make, gcc, g++, etc) using SUSE Studio.

First of all, we need to configure VirtualBox to do port forwarding. If you read the manual, you see that the default networking is NAT, which is what we use, along with extra forwarded ports. To do this, run the following on the command prompt (in your installation folder, e.g. C:\Program Files\Sun\VirtualBox for Windows, tweak for Mac OS X), changing openSUSE with the name of your virtual machine:

VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/HostPort" 3632
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/GuestPort" 3632
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/Protocol" TCP
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP

The last 3 lines are not important for distcc itself, however it's quite handy since you will be able to ssh to the machine (on the port 2222, i.e. the forwarded one and not port 22) and start or stop distcc remotely.

Side note: I was told that bridged networking mode should work as well (even without the hassle of port forwarding). However, I did not manage to make it working in my setup.

In addition, tweak the settings of your virtual machine so you get more than one CPU cores. For example, on a quad-core machine, giving the virtualized openSUSE two or three cores might be better, considered if the host system does not run need all that burning power.

Next step is to install distcc both on the virtualized openSUSE (anjali) and your laptop (latika). The easiest way is to use the one click install feature, just go to software.opensuse.org, click on Package Search, type in distcc and press Enter and then install both distcc and distcc-server. To do it manually, add http://download.opensuse.org/repositories/home:/dbahi/openSUSE_11.2/ to your repositories and install using:

sudo zypper in distcc distcc-server

After that, on anjali, run the following (as normal user, no need to sudo/root):

/usr/sbin/distccd --daemon --allow 192.168.1.0/24

Now on your own machine, latika in the above example, set first where you want to distribute the compile:

export DISTCC_HOSTS='anjali,lzo'

If you want, do this in your .bashrc or shell startup script for your convenient.

Compiling and building a project is now as easy as replacing the plain make command with:

make -j4 CC=distcc CXX=distcc

For troubleshooting, run distccmon-text 3 (nice for static logging) or watch distccmon-text (useful for real-time view).

Supposed another machine, e.g. naina, wants to join the party, just do the same steps. Don't forget to adjust DISTCC_HOSTS so that the compile will be distributed also to naina.

That's it! For more info (I barely scratch the surface) and better optimization, refer to the manpage of distcc and distccd.

Now the (trivia) question is, the host that runs anjali, who do you think he is? :)