Ithiriel // Thoughts From a Sysadmin And Occasional Coder

In lieu of a better term, behavior-driven system administration is the application of BDD principles to system administration, primarily building and maintaining systems.

The RSpec Book by David Chelimsky et al presents BDD as using the tools Cucumber and RSpec. Cucumber is used to describe and test the feature to be implemented and then RSpec is used to test the implementation while it is being built.

In Test-Driven Infrastructure with Chef, Stephen Nelson-Smith states that RSpec is not used in his example in the book because "there's no point in unit testing a declarative system." On one hand, I can agree with the statement. Somewhere, I read software developers saying that you should test your application but not worry about testing the platform.1

On the other hand, I disagree. I think RSpec can be useful when testing a declarative system but my rationale has less to do with testing than with auditing.2 I see Cucumber and RSpec tests as filling two different roles: Cucumber verifies the system's behavior. RSpec verifies the system's state.3 This also lets me use the Cucumber features to document the system's behavior and RSpec scenarios to document the system's state. (As pointed out in one of the BoF sessions at LISA 2011, Puppet manifests don't themselves work as documentation of a system.)

I am currently working on building a new system where I hope to test this. I have RSpec scripts written for making sure Puppet is set up and correctly configured but I haven't completed the Cucumber feature for Puppet to verify it's behavior so I can't show how this works in practice. When I have a full example, including a working Cucumber feature and Puppet config, I'll make another post and walk through it. (Or if it doesn't work out, I'll point that out too.)

• 1. I don't like this amorphous "somewhere" but I can't remember or find where I read this. If I find it, I'll add a reference.
• 2. Although, yes, I do appreciate knowing whether or not the change I have made to Puppet's manifests was the correct change to make. This may be something that goes away as I get more experience with Puppet.
• 3. This may be a misuse of RSpec since it's also intended to verify behavior rather than state. I use RSpec since I'm more familiar with it than Test::Unit or other testing methods.

Thursday night, I wrote about building packages with Mock. After working on copying the built packages into my local repository, I've decided there's a better way.

The old way does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-x86_64/result/*.noarch.rpm $REPOSITORY/i386/
cp /var/lib/mock/epel-6-x86_64/result/*.x86_64.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{i386,i686}.rpm $REPOSITORY/i386/

The new way, instead, does this to build the RPMs:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

And then this to copy the files into the repository:

cp /var/lib/mock/epel-6-x86_64/result/*.{noarch,x86_64}.rpm $REPOSITORY/x86_64/
cp /var/lib/mock/epel-6-i386/result/*.{noarch,i386,i686}.rpm $REPOSITORY/i386/

This builds the noarch packages in both the 32-bit and 64-bit environments. It's a tradeoff between time and having the deployment step make more sense.

I haven't been very happy with my way to build packages. I've been looking for a better system for managing it.

Through conversation in #lopsa, I found my way to Mock, a tool that builds packages in chroot environments.

I've been testing it on a VM. So far, it looks promising.

To use Mock, you first need to add your user to the mock group:

sudo /usr/sbin/usermod -a -G mock $user

After that, Mock is simple to use. If you have a source RPM, building a package is as easy as:

mock -r $configuration rebuild $srpm

So, for example, to build my patched kernel RPM:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm

Build environment configurations are defined in /etc/mock. If you're building RPMs for use with RHEL or CentOS, one of the preexisting epel configurations should suffice. There are configurations for Fedora as well. You can configure your own configurations as well.

On 64-bit systems, you can use the 32-bit configurations to build 32-bit packages. If you want to use specify a different architecture, you can use the --arch argument. For example, to build all binary RPMs for my patched kernel:

mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-i386 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm
mock -r epel-6-x86_64 rebuild kernel-2.6.32.46-1.el6.oberon.src.rpm --arch=noarch --no-clean

The first command builds the 64-bit packages. The second command builds the 32-bit packages. The third command builds the additional packages that don't have an architecture, e.g. kernel-doc-2.6.32.46-1.el6.oberon.noarch.rpm. The --no-clean argument tells mock not to clean the build environment first. Without this, the third command will remove the packages generated by the first command.

When the commands are done running, the 64-bit and noarch RPMs can be found in the directory /var/lib/mock/epel-6-x86_64/result/ and the 32-bit RPMs can be found in the directory /var/lib/mock/epel-6-i386/result/.

I haven't tried using Mock in my Makefile but that's next on the list. I also need to simplify my builds so they don't rebuild all RPMs. Since the kernel RPMs take about six hours to build (for both 32-bit and 64-bit) on my VM, this makes builds almost prohibitively long.

I have also thought about building a system that uses Mock, some message queueing system, and some cloud interface to spin up EC2 instances (or the like) for builds. However, that seems pretty close to Koji so I should probably look into that further first.

This is mostly for my reference since I keep forgetting this. As per cobbler's documentation:

The kickstart URL for a system is:
http://$server/cblr/svc/op/ks/system/$name_of_system

I've been working on a body of characteristics that a server should ideally have. So far, I have:

Documented

A server should be documented. The documentation should provide enough information to anyone who has to service the server, including allowing them to rebuild the server if necessary. The documentation should also satisfy any auditing requirements of the organization.

Verifiable

A server should have a given known state, described in the documentation, and it should be possible to programmatically determine whether or not the system is in that state.

Secure

A server should be protected against any attacks made by unauthorized entities that would disrupt the server or provide the attacker with information they are not authorized to have.

Monitored

A server should be monitored to ensure that it is in its documented state. Any deviation from that state, either through an uncoordinated change made by operations or through actions of an unauthorized party, should be detected automatically and the operations staff should be notified accordingly.

Backed up

A server should be backed up. Data that may have been lost due to error or tampering should be recoverable within the documented parameters.

Replaceable

A server should be replaceable. If the server fails or significantly goes out of its documented state, any technician should be able to provision and install a replacement while the faulty system can be diagnosed, inspected, and, if possible, corrected outside of the production environment.

Manageable

A server should be manageable and serviceable by any technician authorized to work on it.

Measured

A server should have its statistics measured and recorded on a regular basis. This data should be usable for planning activities and to monitor trends on the server.

Functional

A server should fulfill its documented role.

Some of these are admittedly characteristics of the server's environment and its technicians rather than of the server itself.

If you think I've overlooked anything, please let me know.

Edit: I added the last two originally as comments. I've added them to the actual post to make it easier for anyone who visits in the future.