You are here

Archive for No name

Adventures in building a patched kernel for CentOS 6

Recently, I've been trying to build a kernel for CentOS 6 that includes the grsecurity security patch. This was complicated by my desire to build the new kernel as an RPM using the CentOS 6 kernel RPM's spec file.

Why grsecurity?

grsecurity is a kernel patch that adds additional security options and protections. The non-RBAC components of grsecurity will work with SELinux which I will use for now.1

Why use the spec file and not make rpm?

Well, first: Because make rpm didn't work. That was disappointing.

In CentOS kernel install split into separate packages, e.g. kernel, kernel-headers, and kernel-devel. In order to preserve these, I would have to reverse engineer the spec file and it just seems simpler to use what's there. The spec file also makes sure that the new kernel is added to the bootloader's configuration file.

Why use an RPM at all?

Building packages from source doesn't scale.2

One of my current projects is to build a fully functional environment where no packages are installed from source.3 In fact, none of the systems are allowed to have compilers. This is why I've worked on creating my own yum repository.

The process

After downloading the source file for Linux and the grsecurity patch and verifying both4, I changed the spec file to use these. Due to an issue with %setup in the spec file I never fully figured out, I decided to invoke applying the patch manually through the ApplyPatch function defined in the %prep block. I then set about building the kernel initially using rpmbuild -ba SPEC/kernel.spec and ran into problems.

The complications

I ran into six problems during the process of building my RPM. These were:

  1. Fixing an error during the %prep phase
  2. Getting my configuration changes to persist
  3. Signing the modules
  4. Turning off the kABI checker
  5. Setting the kernel version correctly
  6. The system halted on boot

Fixing an error during the %prep phase

During the %prep phase, the spec file tries to verify that all of the configuration options known by the kernel are covered in the configuration files. This relies on the nonint_oldconfig target defined in scripts/kconfig/Makefile in the RPM. However, this option does not exist in Extracting the patch for this (and for the related code in scripts/kconfig/conf.c) from the tarball provided with the RPM, adding it to the spec file, and applying it allowed me to proceed... but not very far. If nonint_oldconfig finds any kernel options that are not covered, like the options added by the grsecurity patch, it returns an error and the build halts.

Getting my configuration changes to persist

The SRPM includes generic and architecture-specific configuration files. These are merged together during the %prep phase. If the configuration changes are not present in these files, they are wiped during the phase.

I decided to keep the configuration files intact and instead placed my overriding kernel options in a separate file. I then modified Makefile.config to check for my files and merge those as well if present.

Signing the modules

CentOS 6 inherits the upstream vendor's patches. One such patch is used to sign the modules with a key so that the file integrity of the modules can be verified later. Any module that fails verification is not loaded. There is a kernel configuration option that requires all modules to be signed. As the spec file is supposed to be used with the distribution's patched kernel source, it verifies that the modules are signed. I could have removed this feature from the spec file but I decided this feature was a good one.

This is not part of the stock kernel and, in fact, does not appear to be included in 3.0.4 either. To use this feature, the code changes have to be isolated from the patched distribution kernel and a new patch needed to be made. After creating a patch from changes made to 54 source files, I added it to the spec file and built signed modules.

Turning off the kABI checker

The CentOS kernel, like the upstream vendor's kernel, is expected to fulfill a given interface contract when built. Anything that would break this contract severely is not allowed to be made as a change to the kernel.5 As a result, the kernel verifies this contract in the spec file.

The standard way to do this is to pass this option to rpmbuild when building the kernel: --without kabichk

As I didn't feel like remembering to specify that every time I build the kernel, I changed the spec file to make sure that the with_kabichk variable set to  . I could have left it in place by copying Module.symvers to the appropriate kABI file in the SOURCES directory and changing the spec file appropriately.

Setting the kernel version correctly

This took me a while to figure out unfortunately. When I built the kernel, the version was being set incorrectly and then this resulted in the system not being able to find modules on boot.

What I figured out in the end was that, for kernel, base_sublevel should be set to 32 and stable_update should be set to 46.

grsecurity also adds a file called localversion-grsec that contains the text -grsec. I wrote a patch to remove the file, added it to the spec file, built it, installed it, and...

The system halted on boot

When a CentOS 6 system boots normally, it uses something called dracut. At some point, it tries to mount a partition inside a chroot'd partition. As the kernel is configured to prevent this (through the grsecurity patch), the system would then halt on boot.

As I didn't want to disable this feature permanently and I wanted everything else to work from boot, I wrote a small patch to disable the feature on boot. Adding kernel.grsecurity.chroot_deny_mount = 1 to /etc/sysctl.conf then enabled this feature later in the boot process.

And now it works

After building the RPMs, I installed the kernel and kernel-firmware packages on a test VM, rebooted it, and it came up using the new kernel.

I did find that a lot of messages were being written to the console. Setting kernel.printk to "6 4 1 7" via sysctl corrected this.

What next?

I want to work on backporting some changes from the CentOS 6 kernel to the package I have built. I know that the CentOS 6 kernel contains some features from 2.6.33, e.g. recvmmsg, and some enhancements to KVM. Unfortunately, since the source tarball in the RPM has already been patched, I have to tease out what those patches are. Once I have those patches, I should then check to see there are any grsecurity changes that need to be backported. Since there is no obviously apparent repository for the grsecurity patch, this requires looking at the test patch (currently for the 3.0.4 kernel) and comparing manually.

So where can I get this?

Once I have some things worked out, I will post the RPMs and the SRPM somewhere that it can be downloaded. For now, you'll just have to be patient.

  • 1. I have grsecurity installed on another server and I've never gotten around to configuring RBAC. Since a lot of work has gone into the CentOS 6 SELinux policies, I'll just rely on those.
  • 2. I've said this before. Some day I'll write a longer post on it.
  • 3. I'll try to write about this in the near future.
  • 4. You do this whenever you download a source file, right?
  • 5. This is why CentOS 5 does not support IPv6 connection tracking without building a custom kernel.

Keeping my repository up to date

The RPMs in my repository are built from a git repository. In the repository, each package is a branch. This lets me keep all of the RPMs together while keeping them separate. To build them, I have a branch named BUILD. I switch to that branch and run make.

The Makefile merges all of the other branches into the BUILD branch, builds the RPMs, copies them into a repository directory, and then rsyncs that directory to my webserver. The source for the Makefile is included at the end of this post.

This method is optimized around a single repository and only needing to enter the password for my GPG key once. The downsides are that the repository layout isn't great and all RPMs are rebuilt even if only one really needs to be. So there are probably better ways to do this. tito may be a way to do this.

Anyway, here's the Makefile:

build : clean
        # Merge branches
        for i in `git branch | grep -v BUILD` ; \
        do \
                git merge $$i ; \
        # Build RPMs
        rpmbuild -ba --sign SPECS/*.spec
        # Create repositories
        mkdir -p repo/{i386,x86_64}
        cp RPMS/noarch/* repo/i386/
        cp RPMS/noarch/* repo/x86_64/
        if [ -d RPMS/i386 ] ; \
        then \
                cp RPMS/i386/* repo/i386/ ; \
        if [ -d RPMS/x86_64 ] ; \
        then \
                cp RPMS/x86_64/* repo/x86_64/ ; \
        createrepo repo/i386/
        createrepo repo/x86_64/
        # Push repositories to server.
        rsync -av -e ssh repo/i386 repo/x86_64 user@host:/path
        # Push git repos to github
        git push --all origin
clean :
        rm -rf repo RPMS/i386/ RPMS/x86_64/ RPMS/noarch/

Creating a yum repository

Now that you have signed RPMs, you have to find a way to make these available for installation.1 If you want to make these available for public use, that means a yum repository. If you're going to use them for private systems only, a yum repository is likely to be the best way to make these available and to handle updates later.

Creating a repository is easy to do: Copy your RPMs to a web-accessible directory and run the createrepo command.

It's traditional to have a separate repository for each combination of OS and architecture. EPEL shows a good example of this.

After creating the repository, you have to tell yum about it. To do this, you need to create a file in /etc/yum.repos.d/ with the .repo extension2 that contains the information for it. For an example repository, you would create a file named example.repo with these contents:

name=Example Repository for Enterprise Linux 5 - $basearch

You can define multiple repositories in a repo file. Each repository definition needs its own section with a unique name. The name of the section is how yum will refer to the repository.

Note that this file says that the GPG key is on the local filesystem. You can use a URL instead like seen here.

After you add this repository file (and put the GPG key in place if you want to keep it on the local filesystem), you should be able to see the RPMs in your repository when you run yum install.

The problem with this approach is that you (or your users) then have to manually maintain the repository files. For example, what if you want to move your repository? If you have the luxury of using a configuration management system, e.g. puppet, this may not be a big deal. However, if you don't or you don't have the access to the systems, there's no easy way to do this.3

How does EPEL deal with this? To make distributing the repo and GPG key easy for the users and to allow for easily updating information later, they package this information in the epel-release RPM. You can do something similar to distribute information for your repository and allow for updating it later.

The easiest place to start will be to download the epel-release SRPM, install it into your RPM build environment, and then make changes to the spec file. For the example repository above, you would end up with this spec file:4

Name:           example-release
Version:        5
Release:        5
Summary:      Example repository for Enterprise Linux configuration
Group:          System Environment/Base
License:        GPL
Source1:        GPL
Source2:        example.repo
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArch:     noarch
Requires:      redhat-release >=  %{version}
Conflicts:     fedora-release
This package contains the Example repository for Enterprise Linux
GPG key as well as configuration for yum.
%setup -q  -c -T
install -pm 644 %{SOURCE0} .
install -pm 644 %{SOURCE1} .
#GPG Key
install -Dpm 644 %{SOURCE0} \
# yum
install -dm 755 $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d
install -pm 644 %{SOURCE2} \
%doc GPL
%config(noreplace) /etc/yum.repos.d/*
* Wed Apr 06 2011 Chris Ess <> - 5-5
- Modified epel-release package for the example repository
 Tue Aug 10 2010 Seth Vidal <skvidal at> - 5-4
- conflict fedora-release so people don't indadvertently do something silly
* Fri Apr 25 2008 Matt Domsch <> - 5-3
- use mirrorlists in epel-testing.repo
- use in (commented out) baseurls
* Mon Apr 02 2007 Michael Stahnke <> - 5-2
- Missed a syntax correction in epel-testing.repo
* Mon Apr 02 2007 Michael Stahnke <> - 5-1
- Hard coded version '5' in epel yum repo files.
* Mon Apr 02 2007 Michael Stahnke <> - 5-0
- Initial Package for RHEL 5

(Please note that this SPEC file is covered under the GPL and I am not the original copyright holder. As a result, I don't think I can remove the changelog for epel-release since it's the only way I see to attribute previous work. I might (and, therefore, you might) be able to do without this by adding a comment that says that this is based on version 5-4 of the epel-release SRPM. I'm not a lawyer and, as such, am not clear on the legal requirements here.)

Once you have the spec file, build (and sign!) the RPM. You can now distribute the RPM to your systems (or your users). To allow for updating this later, add it to your repository as well. Don't forget to add the SRPM to your SRPM repository as well. (If you're making this repository public, you need to have an SRPM repository! It also doesn't hurt to keep one since if your local source trees go away and your version control system5 explodes in a magnificent burst of flame.)

Having this RPM makes provisioning the repository on new machines even easier when using kickstart. Just add a repo line for your repository and add the RPM to the %packages section and it'll get installed automatically.6 You can use this trick to install other RPMs in the repository too.

This covers just about everything I can tell you about building RPMs and making a yum repository. In the future, I'll cover how I'm currently updating the repository I've been setting up. If you have a lot of packages to build, it might make your life easier.

  • 1. If you don't have signed RPMs yet, you might want to read my previous post.
  • 2. I'm not certain that this is required but I've never looked into it.
  • 3. If you're running a public repo, you probably don't want to change the URL if at all possible.
  • 4. Well, not quite. You might be building for a different version of RHEL. If nothing else, the changelog entry is going to be different.
  • 5. You are using version control for your RPM builds, right?
  • 6. Kickstart doesn't check to see if RPMs are signed so you can get away with this.

Signing RPMs

So you've built a shiny new RPM, let's say rubygem-cucumber-0.10.0-1.noarch.rpm, and you want to install it on a system. You enter the command:1

sudo yum install -y ./rubygem-cucumber-0.10.0-1.noarch.rpm

and then get this:

(some output removed)
Dependencies Resolved
 Package          Arch   Version        Repository                         Size
 rubygem-cucumber noarch 0.10.0-1       /rubygem-cucumber-0.10.0-1.noarch 3.3 M
Installing for dependencies:
 rubygem-builder  noarch 2.1.2-2.el5    epel                               81 k
 rubygem-diff-lcs noarch 1.1.2-3.el5    epel                              123 k
 rubygem-gherkin  x86_64 2.3.4-1        oberonproject                     1.2 M
 rubygem-json     x86_64 1.4.6-1        oberonproject                     469 k
                  noarch 1.0.5-1.el5    epel                               42 k
Transaction Summary
Install       6 Package(s)
Upgrade       0 Package(s)
Total size: 5.2 M
Downloading Packages:
Package rubygem-cucumber-0.10.0-1.noarch.rpm is not signed

Well, bummer. yum wants the RPM to be signed.2

To sign your RPM, you first need a GPG key. To create one, run gpg --gen-key and follow the instructions. Once it's created, you should be able to see it by running gpg --list-keys. (For the rest of this, I'm going to assume the key is named "Software Packager". Where you see this, replace it with the name for the key.)

In order for yum to allow using your key, you'll need to import it into the RPM database. First, export the key to a file:

gpg --export -a 'Software Packager' > RPM-GPG-KEY-packager

Now, import it into the RPM database:

sudo rpm --import RPM-GPG-KEY-packager

To tell rpmbuild to use this key, add the following lines to your .rpmmacros file:

%_signature gpg
%_gpg_name Software Packager

Since you have an RPM built, you can add a signature with rpm --addsign, like so:

rpm --addsign ./rubygem-cucumber-0.10.0-1.noarch.rpm

Now, when you run sudo yum install -y ./rubygem-cucumber-0.10.0-1.noarch.rpm, the RPM will install successfully.

If you want to sign RPMs automatically when you build them, which I suggest, add the --sign option to rpmbuild like so:

rpmbuild -ba --sign SPECS/rubygem-cucumber.spec

So now that you have signed RPMs, you surely want to put them in a local repository. I'll show you how to do that (or at least how I do it) in the near future.3

  • 1. You are doing this as a normal user and using sudo for anything that requires root privileges, right?
  • 2. Yes, you could just install it with sudo rpm -i ./rubygem-cucumber-0.10.0-1.noarch.rpm but then you have to manually install the dependencies as well. On a single machine, this may not be too bad, but this won't scale.

    You can also pass --nogpgcheck to yum install but this may be prohibited by your local security policies. For example, the NSA Guide to the Secure Configuration of Red Hat Enterprise Linux 5 recommends ensuring that all yum repositories check the GPG keys.)

  • 3. If you can't wait, check out the createrepo command.

Using Fedora SRPMs with RHEL 5


SRPMs from the Fedora Linux distribution will not install cleanly on a RHEL 5 (or derivative) system.


Use the --nomd5 switch to rpm when installing the SRPM. I.e.:

  1. rpm -ivh --nomd5 php-5.3.5-1.fc14.x86_64.rpm

Alternatively, on a Fedora system, rebuild the SRPM using these parameters to rpmbuild: --define "_source_filedigest_algorithm md5"  --define "_binary_filedigest_algorithm md5"

More Information: 

Starting in Fedora 11, RPMs and SRPMs are signed using SHA256. The rpm packages distributed with RHEL 5 are not compatible with using SHA256 for signing. There is a bug filed with Red Hat but Red Hat declined to update the rpm packages. Since RHEL 6 is out now, there is likely to be no action on this.


Building RPMs from SRPMs

Sometimes you run into a shortcoming in your Linux distribution and you find that a package you want doesn't exist. Perhaps there are is no package for the software available (for example, there is no rubygem-cucumber-nagios package available for CentOS 5). Perhaps the package available is too old (for example, there is no official PHP 5.3.x package for CentOS 5). Perhaps you need functionality built into the package that isn't normally there (for example, patching Apache httpd for mpm-itk). In these cases, you either need to find a package someone else has made or roll your own.

Before you think about installing directly from source, stop. This is a bad habit to get into. It will not scale.

Red Hat Enterprise Linux, Fedora Linux, SuSE, and their derivatives use RPMs for packaging. In this, I hope to show you how to build RPMs from source RPMs. In the future, I hope to show how to add patches to source RPMs to build customized packages.

(You may be able to follow these instructions to build your own RPMs from source. To do this, you will need to skip the step of "installing" the source RPM and will instead need to write your own spec file. I don't have any personal experience with this so I can't help you.)

Before you start anything, make sure you're not doing this as root. If you build RPMs as root, you run a risk of files ending up in system paths rather than in the RPM build path.

Doing it as a normal user? Good. Set up your build environment:

  1. mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
  2. echo '%_topdir %(echo $HOME)/rpmbuild' > ~/.rpmmacros

(For more information, you may want to read the CentOS documentation on setting up the RPM build environment.)

On CentOS and other RHEL derivatives, you will need to install the rpm-build and redhat-rpm-config packages. On other distributions, you may need to install other packages.

Once you have your build directory set up, install your chosen source RPM:

  1. rpm -i rubygem-gem2rpm-0.6.0-1.el5.src.rpm

This will install the source files in ~/rpmbuild/SOURCES and the spec file in ~/rpmbuild/SPECS.

To build the RPM, you can simply run:

  1. rpmbuild -ba ~/rpmbuild/SPECS/rubygem-gem2rpm.spec

This will put the binary RPM under in architecture-dependent directory under ~/rpmbuild/RPMS/. For example, the RPM built for rubygem-gem2rpm is architecture dependent and rubygem-gem2rpm-0.6.0-1.noarch.rpm will be placed under ~/rpmbuild/RPMS/noarch/.

From here, you could simply install the rpm with sudo rpm -i ~/rpmbuild/RPMS/noarch/rubygem-gem2rpm-0.6.0-1.noarch.rpm. I don't recommend this approach. Instead, I recommend signing the RPMs and adding them to a central yum repository. I will cover both of these in the future so stay tuned.

Thoughts on Linux on the desktop

You may disagree with this statement I made earlier today:

Linux on the desktop will never be as polished as OS X, or even Windows, until some company pours hundreds of thousands of dollars (if not more) into building a usable UI on top of it. However, any company that dumps that much into it isn't likely to make those contributions open source.

It is very possible to make Linux desktop-ready. OS X is fundamentally a BSD-clone with a lot of money and time poured into the user interface. So given some vast amount of time and money, it should be possible to do the same thing with Linux.

I could be wrong but: Most organizations that focus on the open source aspects of Linux (well, properly, GNU/Linux and other stuff) tend to not have a lot of money. So providing the required amount of money will probably fall to a company. And a company that is making a such a significant investment is going to want to capitalize on that. Pouring vast amounts of time and money into something and then giving it away is a losing strategy.

It is possible for the software foundations to fund the development but it will take significantly longer. The added time will ensure that desktop Linux UI would always trail the corporate offerings (Windows, OS X, etc.) because it will have never caught up (assuming the corporate offerings do not stagnate).

Enable VM consoles in vSphere Client through a NAT device


When you connect to a VMWare server behind a NAT device via the vSphere client, you are unable to get console access for virtual machines. The vSphere client returns an error like "Unable to connect to the MKS: Failed to connect to server x.x.x.x:903."


Add the following line to /etc/vmware/config:


If the NAT device is also doing firewalling, make sure that port 903/tcp is open for traffic from your local IP to the VMWare server and vice versa.

More Information: 

I originally found the solution on VMWare's forums here. This thread provides more information about the issue. Apparently this is expected to be fixed a future update.


Writing install triggers for cobbler

Cobbler has the ability to run triggers at specific times. Two of those times include before installing a new machine ("pre-install triggers") and after installing a new machine ("post-install triggers").

"Old-style" triggers involve running executable binaries or scripts (e.g. shell scripts) in specific locations. Pre-install triggers are placed in the directory /var/lib/cobbler/triggers/install/pre/ and post-install triggers are placed in the directory /var/lib/cobbler/triggers/install/post/. The trigger will be passed three pieces of information: the object type, e.g. "system," the name of the object, and the object's IP. (A comment in the run_install_triggers method in says this passes the name, MAC, and IP but this does not appear to match the name of the variables.) If the trigger requires more information, it will need to pull it from elsewhere or parse cobbler's output. For all but simple tasks, this is probably not a convenient way to go.

Note: There is a bug in cobbler which prevents running "old-style" triggers. See ticket #530 for more information and a possible fix.

"New-style" triggers are written in Python as modules. They reside in cobbler's module directory. (On my system, this is /usr/lib/python2.4/site-packages/cobbler/modules/.) Each module is required to define at least the functions register and run.

The register function takes no arguments and returns a string corresponding to the directory the module would reside in if it were an "old-style" trigger. For a pre-install trigger, it would be:

  1. def register():
  2.     return "/var/lib/cobbler/triggers/install/post/*"

For a post-install trigger, it would be:

  1. def register():
  2.     return "/var/lib/cobbler/triggers/install/post/*"

The run function is where the actual code for the trigger should reside. It takes three arguments: A cobbler API reference, an array containing arguments, and a logger reference. The argument array contains the same three values as for "old-style" triggers, i.e. the object type, the name, and the IP address. The logger reference may be set to None and the code should handle that. (In cobbler, this will be set to None. This may be fixed when the issue for "old-style" install triggers is.)

For an example of a run function, let's look at one I wrote (based on the trigger in that is included with cobbler) to automatically sign Puppet certificates:

  1. def run(api, args, logger):

This starts the method. Note the signature.

  1.     settings = api.settings()
  3.     if not str(settings.sign_puppet_certs_automatically).lower() in [ "1", "yes", "y", "true"]:
  4.         return 0

This retrieves the settings from /etc/cobbler/settings. To control the trigger, I added another option there named sign_puppet_certs_automatically. If this value either does not exist or is not set to one of the required values showing its enabled, the trigger returns a success code (since it's not supposed to run, it shouldn't return a failure code) and exits.

I also added another option to the cobbler settings called puppetca_path which contains the path to the puppetca command.

  1.     objtype = args[0] # "target" or "profile"
  2.     name    = args[1] # name of target or profile
  4.     if objtype != "system":
  5.         return 0

This retrieves the object type and name from the argument array. If the object type is not a system, it returns a success code and exits.

  1.     system = api.find_system(name)
  2.     system = utils.blender(api, False, system)
  4.     hostname = system[ "hostname" ]

This finds the system in the cobbler API and then flattens it to a dictionary. I'm pretty sure this could be improved upon.

  1.     puppetca_path = settings.puppetca_path
  2.     cmd = [puppetca_path, '--sign', hostname]

This retrieves the path for puppetca and sets up the command to be run to sign the certificate.

  1.     rc = 0
  2.     try:
  3.         rc = utils.subprocess_call(logger, cmd, shell=False)
  4.     except:
  5.         if logger is not None:
  6.             logger.warning("failed to execute %s", puppetca_path)
  8.     if rc != 0:
  9.         if logger is not None:
  10.             logger.warning("signing of puppet cert for %s failed", name)
  12.     return 0

This runs the command and logs a warning if either the command fails to be executed or does not succeed. Finally, at the end, it returns a success code.

According to the cobbler documentation, the return code of post-install triggers is ignored so there's no reason not to return anything other than value. Pre-install triggers apparently can halt the process if they return a non-zero value.

Note: The above code will not run correctly if logger is set to None. This is because utils.subprocess_call tries to call logger without verifying that it is not None and throws an exception. To use this with cobbler, you must either edit change the call to utils.run_triggers in's run_install_triggers method or you must change utils.subprocess_call to properly check for logger being set to None.

Also note: Since the original code is under the GPL, the code above is also under the GPL.

Atom feed issues

Apparently my last post broke the Atom feed, causing it not to validate. I'm looking into how to fix it but it may be related to a known issue with Drupal's Atom module.

I don't expect to have this fixed today. I suggest switching to the RSS feed for now.