Another way to build RPMs with Mock

Thursday night, I wrote about building packages with Mock. After working on copying the built packages into my local repository, I've decided there's a better way.

The old way does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-
mock -r epel-6-i386 rebuild kernel-
mock -r epel-6-x86_64 rebuild kernel- --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/i386/
cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{i386,i686}.rpm $REPOSITORY/i386/

The new way, instead, does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-
mock -r epel-6-x86_64 rebuild kernel- --arch=noarch --no-clean
mock -r epel-6-i386 rebuild kernel-
mock -r epel-6-i386 rebuild kernel- --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.{noarch,x86_64}.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{noarch,i386,i686}.rpm $REPOSITORY/i386/

This builds the noarch packages in both the 32-bit and 64-bit environments. It's a tradeoff between time and having the deployment step make more sense.

Building RPMs with Mock

I haven't been very happy with my way to build packages. I've been looking for a better system for managing it.

Through conversation in #lopsa, I found my way to Mock, a tool that builds packages in chroot environments.

I've been testing it on a VM. So far, it looks promising.

To use Mock, you first need to add your user to the mock group:

sudo /usr/sbin/usermod -a -G mock $user

After that, Mock is simple to use. If you have a source RPM, building a package is as easy as:

mock -r $configuration rebuild $srpm

So, for example, to build my patched kernel RPM:

mock -r epel-6-x86_64 rebuild kernel-

Build environment configurations are defined in /etc/mock. If you're building RPMs for use with RHEL or CentOS, one of the preexisting epel configurations should suffice. There are configurations for Fedora as well. You can configure your own configurations as well.

On 64-bit systems, you can use the 32-bit configurations to build 32-bit packages. If you want to use specify a different architecture, you can use the --arch argument. For example, to build all binary RPMs for my patched kernel:

mock -r epel-6-x86_64 rebuild kernel-
mock -r epel-6-i386 rebuild kernel-
mock -r epel-6-x86_64 rebuild kernel- --arch=noarch --no-clean

The first command builds the 64-bit packages. The second command builds the 32-bit packages. The third command builds the additional packages that don't have an architecture, e.g. kernel-doc- The --no-clean argument tells mock not to clean the build environment first. Without this, the third command will remove the packages generated by the first command.

When the commands are done running, the 64-bit and noarch RPMs can be found in the directory /var/lib/mock/epel-6-x86_64/result/ and the 32-bit RPMs can be found in the directory /var/lib/mock/epel-6-i386/result/.

I haven't tried using Mock in my Makefile but that's next on the list. I also need to simplify my builds so they don't rebuild all RPMs. Since the kernel RPMs take about six hours to build (for both 32-bit and 64-bit) on my VM, this makes builds almost prohibitively long.

I have also thought about building a system that uses Mock, some message queueing system, and some cloud interface to spin up EC2 instances (or the like) for builds. However, that seems pretty close to Koji so I should probably look into that further first.

Cobbler kickstart URL

This is mostly for my reference since I keep forgetting this. As per cobbler's documentation:

The kickstart URL for a system is:

Desired characteristics of a server

I've been working on a body of characteristics that a server should ideally have. So far, I have:

A server should be documented. The documentation should provide enough information to anyone who has to service the server, including allowing them to rebuild the server if necessary. The documentation should also satisfy any auditing requirements of the organization.
A server should have a given known state, described in the documentation, and it should be possible to programmatically determine whether or not the system is in that state.
A server should be protected against any attacks made by unauthorized entities that would disrupt the server or provide the attacker with information they are not authorized to have.
A server should be monitored to ensure that it is in its documented state. Any deviation from that state, either through an uncoordinated change made by operations or through actions of an unauthorized party, should be detected automatically and the operations staff should be notified accordingly.
Backed up
A server should be backed up. Data that may have been lost due to error or tampering should be recoverable within the documented parameters.
A server should be replaceable. If the server fails or significantly goes out of its documented state, any technician should be able to provision and install a replacement while the faulty system can be diagnosed, inspected, and, if possible, corrected outside of the production environment.
A server should be manageable and serviceable by any technician authorized to work on it.
A server should have its statistics measured and recorded on a regular basis. This data should be usable for planning activities and to monitor trends on the server.
A server should fulfill its documented role.

Some of these are admittedly characteristics of the server's environment and its technicians rather than of the server itself.

If you think I've overlooked anything, please let me know.

Edit: I added the last two originally as comments. I've added them to the actual post to make it easier for anyone who visits in the future.

Adventures in building a patched kernel for CentOS 6

Recently, I've been trying to build a kernel for CentOS 6 that includes the grsecurity security patch. This was complicated by my desire to build the new kernel as an RPM using the CentOS 6 kernel RPM's spec file.

Why grsecurity?

grsecurity is a kernel patch that adds additional security options and protections. The non-RBAC components of grsecurity will work with SELinux which I will use for now.1

Why use the spec file and not make rpm?

Well, first: Because make rpm didn't work. That was disappointing.

In CentOS kernel install split into separate packages, e.g. kernel, kernel-headers, and kernel-devel. In order to preserve these, I would have to reverse engineer the spec file and it just seems simpler to use what's there. The spec file also makes sure that the new kernel is added to the bootloader's configuration file.

Why use an RPM at all?

Building packages from source doesn't scale.2

One of my current projects is to build a fully functional environment where no packages are installed from source.3 In fact, none of the systems are allowed to have compilers. This is why I've worked on creating my own yum repository.

The process

After downloading the source file for Linux and the grsecurity patch and verifying both4, I changed the spec file to use these. Due to an issue with %setup in the spec file I never fully figured out, I decided to invoke applying the patch manually through the ApplyPatch function defined in the %prep block. I then set about building the kernel initially using rpmbuild -ba SPEC/kernel.spec and ran into problems.

The complications

I ran into six problems during the process of building my RPM. These were:

  1. Fixing an error during the %prep phase
  2. Getting my configuration changes to persist
  3. Signing the modules
  4. Turning off the kABI checker
  5. Setting the kernel version correctly
  6. The system halted on boot

Fixing an error during the %prep phase

During the %prep phase, the spec file tries to verify that all of the configuration options known by the kernel are covered in the configuration files. This relies on the nonint_oldconfig target defined in scripts/kconfig/Makefile in the RPM. However, this option does not exist in Extracting the patch for this (and for the related code in scripts/kconfig/conf.c) from the tarball provided with the RPM, adding it to the spec file, and applying it allowed me to proceed... but not very far. If nonint_oldconfig finds any kernel options that are not covered, like the options added by the grsecurity patch, it returns an error and the build halts.

Getting my configuration changes to persist

The SRPM includes generic and architecture-specific configuration files. These are merged together during the %prep phase. If the configuration changes are not present in these files, they are wiped during the phase.

I decided to keep the configuration files intact and instead placed my overriding kernel options in a separate file. I then modified Makefile.config to check for my files and merge those as well if present.

Signing the modules

CentOS 6 inherits the upstream vendor's patches. One such patch is used to sign the modules with a key so that the file integrity of the modules can be verified later. Any module that fails verification is not loaded. There is a kernel configuration option that requires all modules to be signed. As the spec file is supposed to be used with the distribution's patched kernel source, it verifies that the modules are signed. I could have removed this feature from the spec file but I decided this feature was a good one.

This is not part of the stock kernel and, in fact, does not appear to be included in 3.0.4 either. To use this feature, the code changes have to be isolated from the patched distribution kernel and a new patch needed to be made. After creating a patch from changes made to 54 source files, I added it to the spec file and built signed modules.

Turning off the kABI checker

The CentOS kernel, like the upstream vendor's kernel, is expected to fulfill a given interface contract when built. Anything that would break this contract severely is not allowed to be made as a change to the kernel.5 As a result, the kernel verifies this contract in the spec file.

The standard way to do this is to pass this option to rpmbuild when building the kernel: --without kabichk

As I didn't feel like remembering to specify that every time I build the kernel, I changed the spec file to make sure that the with_kabichk variable set to  . I could have left it in place by copying Module.symvers to the appropriate kABI file in the SOURCES directory and changing the spec file appropriately.

Setting the kernel version correctly

This took me a while to figure out unfortunately. When I built the kernel, the version was being set incorrectly and then this resulted in the system not being able to find modules on boot.

What I figured out in the end was that, for kernel, base_sublevel should be set to 32 and stable_update should be set to 46.

grsecurity also adds a file called localversion-grsec that contains the text -grsec. I wrote a patch to remove the file, added it to the spec file, built it, installed it, and...

The system halted on boot

When a CentOS 6 system boots normally, it uses something called dracut. At some point, it tries to mount a partition inside a chroot'd partition. As the kernel is configured to prevent this (through the grsecurity patch), the system would then halt on boot.

As I didn't want to disable this feature permanently and I wanted everything else to work from boot, I wrote a small patch to disable the feature on boot. Adding kernel.grsecurity.chroot_deny_mount = 1 to /etc/sysctl.conf then enabled this feature later in the boot process.

And now it works

After building the RPMs, I installed the kernel and kernel-firmware packages on a test VM, rebooted it, and it came up using the new kernel.

I did find that a lot of messages were being written to the console. Setting kernel.printk to "6 4 1 7" via sysctl corrected this.

What next?

I want to work on backporting some changes from the CentOS 6 kernel to the package I have built. I know that the CentOS 6 kernel contains some features from 2.6.33, e.g. recvmmsg, and some enhancements to KVM. Unfortunately, since the source tarball in the RPM has already been patched, I have to tease out what those patches are. Once I have those patches, I should then check to see there are any grsecurity changes that need to be backported. Since there is no obviously apparent repository for the grsecurity patch, this requires looking at the test patch (currently for the 3.0.4 kernel) and comparing manually.

So where can I get this?

Once I have some things worked out, I will post the RPMs and the SRPM somewhere that it can be downloaded. For now, you'll just have to be patient.

  • 1. I have grsecurity installed on another server and I've never gotten around to configuring RBAC. Since a lot of work has gone into the CentOS 6 SELinux policies, I'll just rely on those.
  • 2. I've said this before. Some day I'll write a longer post on it.
  • 3. I'll try to write about this in the near future.
  • 4. You do this whenever you download a source file, right?
  • 5. This is why CentOS 5 does not support IPv6 connection tracking without building a custom kernel.


Subscribe to Ithiriel RSS