Ithiriel // Thoughts From a Sysadmin And Occasional Coder

"You know what you believe in by observing yourself in a crisis."
— Robert C. Martin, The Clean Coder

Almost three years ago, I wrote a rather presumptuous post about moving away from Perl. I specifically cited Perl's object system as a major reason for dropping it. chromatic called me out on that and for good reason. I also cited maintainability issues with my Perl scripts but those are a result of my own coding and not anything inherent in Perl. (Allowing me to write sloppy code does not mean the same thing as making me write sloppy code.)

While I talked about moving on to Python, I have made almost no progress on this. And when I feel stressed or pressed for time (which is just about always), I always fall back to Perl. It is simply what I know best.

I'm sure I could still correct this. I could immerse myself in Python (or Ruby) and learn either to the point of being able to dream in it. But I haven't done it in the past three years. What likelihood is there that I'll be able to do this in the next three years?

Through observation, and no matter how what rational or irrational reasons I could put forth otherwise, I can only conclude that I believe in Perl.

In lieu of a better term, behavior-driven system administration is the application of BDD principles to system administration, primarily building and maintaining systems.

The RSpec Book by David Chelimsky et al presents BDD as using the tools Cucumber and RSpec. Cucumber is used to describe and test the feature to be implemented and then RSpec is used to test the implementation while it is being built.

In Test-Driven Infrastructure with Chef, Stephen Nelson-Smith states that RSpec is not used in his example in the book because "there's no point in unit testing a declarative system." On one hand, I can agree with the statement. Somewhere, I read software developers saying that you should test your application but not worry about testing the platform.1

On the other hand, I disagree. I think RSpec can be useful when testing a declarative system but my rationale has less to do with testing than with auditing.2 I see Cucumber and RSpec tests as filling two different roles: Cucumber verifies the system's behavior. RSpec verifies the system's state.3 This also lets me use the Cucumber features to document the system's behavior and RSpec scenarios to document the system's state. (As pointed out in one of the BoF sessions at LISA 2011, Puppet manifests don't themselves work as documentation of a system.)

I am currently working on building a new system where I hope to test this. I have RSpec scripts written for making sure Puppet is set up and correctly configured but I haven't completed the Cucumber feature for Puppet to verify it's behavior so I can't show how this works in practice. When I have a full example, including a working Cucumber feature and Puppet config, I'll make another post and walk through it. (Or if it doesn't work out, I'll point that out too.)

• 1. I don't like this amorphous "somewhere" but I can't remember or find where I read this. If I find it, I'll add a reference.
• 2. Although, yes, I do appreciate knowing whether or not the change I have made to Puppet's manifests was the correct change to make. This may be something that goes away as I get more experience with Puppet.
• 3. This may be a misuse of RSpec since it's also intended to verify behavior rather than state. I use RSpec since I'm more familiar with it than Test::Unit or other testing methods.

Thursday night, I wrote about building packages with Mock. After working on copying the built packages into my local repository, I've decided there's a better way.

The old way does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/i386/
cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{i386,i686}.rpm $REPOSITORY/i386/

The new way, instead, does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.{noarch,x86_64}.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{noarch,i386,i686}.rpm $REPOSITORY/i386/

This builds the noarch packages in both the 32-bit and 64-bit environments. It's a tradeoff between time and having the deployment step make more sense.

I haven't been very happy with my way to build packages. I've been looking for a better system for managing it.

Through conversation in #lopsa, I found my way to Mock, a tool that builds packages in chroot environments.

I've been testing it on a VM. So far, it looks promising.

To use Mock, you first need to add your user to the mock group:

sudo /usr/sbin/usermod -a -G mock $user

After that, Mock is simple to use. If you have a source RPM, building a package is as easy as:

mock -r $configuration rebuild $srpm

So, for example, to build my patched kernel RPM:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm

Build environment configurations are defined in /etc/mock. If you're building RPMs for use with RHEL or CentOS, one of the preexisting epel configurations should suffice. There are configurations for Fedora as well. You can configure your own configurations as well.

On 64-bit systems, you can use the 32-bit configurations to build 32-bit packages. If you want to use specify a different architecture, you can use the --arch argument. For example, to build all binary RPMs for my patched kernel:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

The first command builds the 64-bit packages. The second command builds the 32-bit packages. The third command builds the additional packages that don't have an architecture, e.g. kernel-doc-2.6.32.46-1.el6.oberon.noarch.rpm. The --no-clean argument tells mock not to clean the build environment first. Without this, the third command will remove the packages generated by the first command.

When the commands are done running, the 64-bit and noarch RPMs can be found in the directory /var/lib/mock/epel-6-x86_64/result/ and the 32-bit RPMs can be found in the directory /var/lib/mock/epel-6-i386/result/.

I haven't tried using Mock in my Makefile but that's next on the list. I also need to simplify my builds so they don't rebuild all RPMs. Since the kernel RPMs take about six hours to build (for both 32-bit and 64-bit) on my VM, this makes builds almost prohibitively long.

I have also thought about building a system that uses Mock, some message queueing system, and some cloud interface to spin up EC2 instances (or the like) for builds. However, that seems pretty close to Koji so I should probably look into that further first.

This is mostly for my reference since I keep forgetting this. As per cobbler's documentation:

The kickstart URL for a system is:
http://$server/cblr/svc/op/ks/system/$name_of_system