Installing Linux VMs under KVM with cobbler and koan

I spent my weekend working on getting cobbler and koan working on a new server. This server will serve several virtual machines under KVM and I want a way to automatically reprovision VMs while I am working on tuning and testing.

cobbler "is a Linux installation server that allows for rapid setup of network installation environments." It allows for easy management of kickstart scripts, DHCP, and other services needed for provisioning new machines. koan, which stands for "kickstart-over-a-network," is a client for cobbler and is used "for reinstallation and virtualization support."

Directions on how to install cobbler can be found on the site for cobbler. In a nutshell, Fedora users can install it directly from the repository servers and RHEL and CentOS users can install it from the EPEL and EPEL testing repositories. (For this, I am using CentOS 5.4 and the version of cobbler and koan from the EPEL testing repository. cobbler was originally installed from the EPEL repository so this may create some oddities in behavior.) To support some functionality, you will want to install the qspice-libs and yum-utils packages.

For cobbler to work correctly under SELinux, cobbler check suggests the following changes:

/usr/sbin/setsebool -P httpd_can_network_connect true
/usr/sbin/semanage fcontext -a -t public_content_t "/var/lib/tftpboot/.*"
/usr/sbin/semanage fcontext -a -t public_content_t "/var/www/cobbler/images/.*"

This allows the Apache httpd server to connect to the network and assigns the public_content_t file context to files in /var/lib/tftpboot/ and /var/www/cobbler/images/. If the standard firewall is installed, traffic will need to be allowed to TCP ports 80 and 26161. cobbler check will also recommend enabling TFTP and allowing traffic to UDP port 69 but these are not required for provisioning virtual machines with koan. You should also review the settings in /etc/cobbler/settings. At a minimum, the server and next_server settings need to be changed. (If you're using KVM, you probably want to change the default_virt_bridge and default_virt_type settings as well.)

To start working with cobbler, a distribution needs to be created. The man page supports importing a repository. However, this may originally take a while since the base repository tends to be large. (The x86_64 repository for CentOS 5.4 is about 4.4 GB.) An example is:

cobbler import --path=rsync://ftp.linux.ncsu.edu/CentOS/5/os/x86_64/ --name=centos5 --arch=x86_64

This will create a distribution named centos5-x86_64. (You will want to use a mirror local to you.)

cobbler allows identifying other repositories that will be mirrored locally and used for the build process via the repo kickstart option. If you want to install with the newest packages available, you will want to add the updates repository, like so:

cobbler repo add --arch=x86_64 --name=centos5-updates-x86_64 \
    --mirror=rsync://ftp.linux.ncsu.edu/CentOS/5/updates/x86_64/

All this will do is tell cobbler to mirror the repository locally. In order to actually set up the mirror, you will need to run cobbler reposync.

Next, identify an installation profile. This is usually tied to a specific server role. Here, we'll define a profile for a VM for a DNS server:

cobbler profile add --name=centos5-vm_dns --distro=centos5-x86_64 --virt-ram=512 \
    --virt-type=qemu --virt-cpus=1 --repos="centos5-updates-x86_64" \
    --kickstart=/var/lib/cobbler/kickstarts/vm-dns.ks --kopts="serial console=ttyS0,115200" \
    --kopts-post="console=ttyS0,115200"

This specifies that the profile should use the CentOS 5 distribution and the CentOS 5 updates repository created above, configure it to run as a VM under KVM, use 512 MB of RAM, and one virtual CPU. The kickstart file template, which has already been created, resides under /var/lib/cobbler/kickstarts/vm-dns.ks. (Information on setting up the kickstart template can be found on the cobbler website.) The kopts setting specifies that kickstart should be started with the serial and console settings. The kopts-post setting specifies that the installed machine should have the console kernel option defined. (These are needed so that the VM makes a console available that can be used via virsh console.)

Usually each profile will have multiple systems under it. A system corresponds to a machine, physical or virtual. (There is still some benefit in using cobbler when there is only one system per profile like I'm doing on my testing server. However, it will really shine when you have to install many mostly identical machines.) A system is defined in the following manner:

cobbler system add --name example-dns --profile=centos5-vm_dns \
    --ip=192.168.122.3 --gateway=192.168.122.1 --subnet=255.255.255.0 \
    --hostname=dns.example.com --static=1

This creates a system named dns.example.com with the static IP 192.168.122.3.

To install a virtual machine, you use koan like so:

koan --server=192.168.122.1 --virt --system=example-dns --virt-path=/dev/mapper/examplevg-dns

This will install a VM using the example-dns system defined above to /dev/mapper/examplevg-dns, a logical volume created via LVM. This should work properly if using a file or a normal partition.

If you've used virt-install, you're probably used to seeing the console opened immediately once the VM is started. koan does not do this. To see the VM, you will need to open the console manually via virsh console. (For example, to see the console for the VM being created by the koan statement above, use virsh console example-dns.)

A few issues I encountered were:

  • Error message: libvir: QEMU error : internal error cannot parse QEMU version number in ''
    As mentioned here, the error is corrected by installing the qspice-libs package.
  • Error message:
    libvir: QEMU error : internal error unable to start guest: qemu: could not open disk image /dev/mapper/examplevg-dns

    Also, errors like these appear in the audit log:

    type=AVC msg=audit(1266799848.453:623): avc:  denied  { getattr } for  pid=32208 
    comm="qemu-kvm" path="/dev/mapper/examplevg-dns" dev=tmpfs ino=78122 
    scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
    tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file
     
    type=AVC msg=audit(1266799848.453:624): avc:  denied  { read } for  pid=32208 comm="qemu-kvm" 
    name="examplevg-dns" dev=tmpfs ino=78122 scontext=system_u:system_r:qemu_t:s0-s0:c0.c1023 
    tcontext=system_u:object_r:fixed_disk_device_t:s0 tclass=blk_file

    The solution to this is to add the virt_image_t context to the block file with:

    chcon -t virt_image_t /dev/mapper/examplevg-dns

    I tried to set this with semanage but was unsuccessful. This probably means that I need to look at the documentation better.

    This is apparently not an issue in Fedora 11 and later and in the upcoming RHEL 6 due to sVirt. In a mailing list post, Daniel Walsh states that they hope to get this ported into RHEL 5.6.

  • Despite the server and next_server settings in /etc/cobbler/settings, koan still tries to install using URLs referencing 127.0.0.1.

    I don't really know the cause of this. It seemed odd but I couldn't trace it. For now, I used a workaround by setting the server for the profile manually with --server=192.168.122.1.

    It could either be that there is something strange with my setup or that this is intended.

  • Error when running koan:
    exceptions.TypeError
    cannot concatenate 'str' and 'dict' objects
      File "/usr/lib/python2.4/site-packages/koan/app.py", line 215, in main
        k.run()
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 329, in run
        self.virt()
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 652, in virt
        return self.net_install(after_download)
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 571, in net_install
        after_download(self, profile_data)
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 650, in after_download
        self.virt_net_install(profile_data)
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 1103, in virt_net_install
        kextra                        = self.calc_kernel_args(pd)
       File "/usr/lib/python2.4/site-packages/koan/app.py", line 1050, in calc_kernel_args
        kextra = kextra + " " + options

    I'm not certain why I saw this particular error. My solution was to comment out line 1050 and add this line in its place:

    kextra = kextra + " " + utils.hash_to_string(options)

    If I remember, I'll post to the cobbler mailing list.

    Update: I opened ticket #576 for this issue. My email to the cobbler-devel mailing list includes a patch.

If this looks interesting, I definitely suggest looking at the cobbler website. If you install many machines, you may also be interested in looking at the web interface which may make this task even easier.

Previous: So what's it good for?
Next: Atom feed issues

Comments

cannot concatenate 'str' and 'dict' objects issue

I'm glad I found your page 'cause nobody else seemed to be having this issue and although I could see the problem I'm no python coder so didn't know how handle the conversion.

But after trying your suggested fix I get:
'module' object has no attribute 'hash_to_string'

It's probably just a typo but I thought I'd ask.

great blog btw,
Stephane

Old version of koan?

What version of koan do you have installed? According to the git history, hash_to_string was first present in the version 2.0.3 release. If you have an older version of koan, my fix won't work unless you update.

Actually, you are right. I

Actually, you are right. I had to manually update to 2.0.3 on my KVM host and forgot to do it on the second one, on which I did this test today.
Works now. Thanks.

Did you get a chance to post this to the Cobbler devs mailing list? Just to make sure they know as It's kind of a show stopper for KVM deployment.

Thanks for the quick reply.
Stephane

Not yet...

I haven't gotten around to posting to the cobbler mailing list. I'll add it to my to-do list and hopefully get it done today or tomorrow.

Awesome! Thanks!

I've been looking for over a week for a way to fix this, including completely uninstalling and deleting cobbler and starting from scratch. I can't believe that this is a limited problem...

For the record I'm running CentOS 5.4, with EPEL, EPEL-Testing, and RPMForge repositories. I ran a full update, and cobbler stopped working. *sigh*

But your patch fixed it right up. At least, until the next update. ;)

Thanks!

koan using 127.0.0.1

koan finds cobbler using the COBBLER_SERVER environment variable, which is pushed down if the machine
was initially provisioned with Cobbler (it's under /etc/profile.d somewhere if you want to change it).

I didn't know about

I didn't know about COBBLER_SERVER and I do see that on a machine I recently installed via cobbler. However, the machine I was testing with at the time wasn't installed via cobbler so COBBLER_SERVER should have been unset (unless it's set by the cobbler RPM itself).

Post new comment

The content of this field is kept private and will not be shown publicly.