distcc, VirtualBox, NAT

The following scenario is quite typical. Your family member, spouse, or coworker has this fantastic, powerful, brand-new multicore machine which, for some reasons, has to run Windows or Mac OS X. On the other hand, you probably have some el-cheapo laptop which is also wonderful but it lacks the processing power to do heavy-duty build and compile. Now, what if you can exploit the other box for a distributed compile? Especially if the powerful box seems to be idle once a while, doing nothing.

Many of you already figure it out the solution: let's install Linux on a virtual machine and use tools like distcc or icecream. In fact, this is quite easy to do. I managed to find out the details, even if I have only little idea about network stuff.

Side note: I don't see why icecream would not work. But since I did try distcc, that's what I wrote below.

For the following explanation, refer to the above exemplary network setup. Basically I run virtualized openSUSE inside VirtualBox. You don't even need a full-blown graphical system, you can create a customized openSUSE installation (no X11 but with some development tools like make, gcc, g++, etc) using SUSE Studio.

First of all, we need to configure VirtualBox to do port forwarding. If you read the manual, you see that the default networking is NAT, which is what we use, along with extra forwarded ports. To do this, run the following on the command prompt (in your installation folder, e.g. C:\Program Files\Sun\VirtualBox for Windows, tweak for Mac OS X), changing openSUSE with the name of your virtual machine:

VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/HostPort" 3632
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/GuestPort" 3632
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/distcc/Protocol" TCP
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/HostPort" 2222
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/GuestPort" 22
VBoxManage.exe setextradata "openSUSE" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/ssh/Protocol" TCP

The last 3 lines are not important for distcc itself, however it's quite handy since you will be able to ssh to the machine (on the port 2222, i.e. the forwarded one and not port 22) and start or stop distcc remotely.

Side note: I was told that bridged networking mode should work as well (even without the hassle of port forwarding). However, I did not manage to make it working in my setup.

In addition, tweak the settings of your virtual machine so you get more than one CPU cores. For example, on a quad-core machine, giving the virtualized openSUSE two or three cores might be better, considered if the host system does not run need all that burning power.

Next step is to install distcc both on the virtualized openSUSE (anjali) and your laptop (latika). The easiest way is to use the one click install feature, just go to software.opensuse.org, click on Package Search, type in distcc and press Enter and then install both distcc and distcc-server. To do it manually, add http://download.opensuse.org/repositories/home:/dbahi/openSUSE_11.2/ to your repositories and install using:

sudo zypper in distcc distcc-server

After that, on anjali, run the following (as normal user, no need to sudo/root):

/usr/sbin/distccd --daemon --allow 192.168.1.0/24

Now on your own machine, latika in the above example, set first where you want to distribute the compile:

export DISTCC_HOSTS='anjali,lzo'

If you want, do this in your .bashrc or shell startup script for your convenient.

Compiling and building a project is now as easy as replacing the plain make command with:

make -j4 CC=distcc CXX=distcc

For troubleshooting, run distccmon-text 3 (nice for static logging) or watch distccmon-text (useful for real-time view).

Supposed another machine, e.g. naina, wants to join the party, just do the same steps. Don't forget to adjust DISTCC_HOSTS so that the compile will be distributed also to naina.

That's it! For more info (I barely scratch the surface) and better optimization, refer to the manpage of distcc and distccd.

Now the (trivia) question is, the host that runs anjali, who do you think he is? :)

QEMU 0.9 with acceleration on SUSE 10.2

QEMU version 0.9 is already out for some time, but AFAIK no official package is available for openSUSE Linux 10.2 yet. Here are some steps I did to have QEMU 0.9 with accelerator (kqemu) on SUSE 10.2. The motivation is simple: I'd like to have a faster system for filter development.

QEMU 0.9 comes, among others, in binary package. I found out that this works well, so just download qemu-0.9.0-i386.tar.gz then unpack it to /. You'll see some new stuff in /usr/local/.

Next step is to install and use accelerator. This time I did compile it from source. First, get kqemu-1.3.0pre11.tar.gz, unpack it, then run the usual steps: ./configure follows by make and sudo make install. Don't forget to have kernel sources (install kernel-source package from YaST if necessary) in your system before doing this.

Now do the following:

export device="/dev/kqemu"
sudo rm -f $device
sudo mknod --mode=0666 $device c 250 0

After that, load the accelerator (kqemu) kernel module:

sudo /sbin/modprobe kqemu major=250

Now, run your virtual machine using the extra -kernel-kqemu option, e.g:

qemu -hda vdisk.qcow -kernel-kqemu

If you forget to load the kqemu kernel module, you'll get the following error message:

Could not open '/dev/kqemu' - QEMU acceleration layer not activated

otherwise, nothing shows up and your virtual machine is executed as usual.

To verify that kqemu works, activate the monitor using Ctrl+Alt+2, then type in info kqemu followed by Enter. You should see:

kqemu support: enabled for user and kernel code

(Exit from the monitor using Ctrl+Alt+1)

If, for some reason, you don't want to run the virtual machine with acceleration, use the option -no-kqemu, e.g.:

qemu -hda vdisk.qcow -no-kqemu

Now, info kqemu on the monitor would give:

kqemu support: disabled

I tested QEMU 0.9 with Windows NT 4.0 as the guest operating system. Even without acceleration, I've seen performance improvement compared to previous versions of QEMU. With acceleration (kqemu), it was even faster, although the kqemu documentation says only Windows 2000 and XP are supported. Benchmark done using CPUBench showed that the virtual machine runs roughly equal to a 300 Mhz and 1.3 GHz system without and with kqemu, respectively. Now, that's an improvement!

Happy virtualizing...

Sources:
http://fabrice.bellard.free.fr/qemu/kqemu-doc.html
http://fabrice.bellard.free.fr/qemu/kqemu-doc.html
http://kidsquid.com/cgi-bin/moin.cgi/QemuOnLinux