That Time I Was Screwed Over By Red Hat Network

Red Hat Network doesn't exist any more, thank God. But let me tell you what it was and why it absolutely and totally sucked.

In 2014 I started my first real sysadmin job. I'd been doing devops for three years already but never worked this close to the OS and hardware. Now I would be managing a bunch of virtual machines we used for building and testing. This was old school; there was hardly any automation in place for stuff like that and the project in question was winding down to be replaced by a complete re-write of the system so there wasn't much point in spending too much time fixing it either.

We had two releases a year if I recall correctly, and some patch releases in between. For every new release we set a baseline of packages to be installed on the host machine (and of course all test servers). This was the setup we would support.

Red Hat Network was a web portal where we registered all our VMs and saved our baselines as templates. Then whenever we needed a test server to conform to a specific baseline we would apply the template to it in the web portal. It would then tell us what the difference between the actual installed packages and the template was and we would run a CLI command to sync it up.

This process could take anywhere from one to eight hours. Why?

Because the sync never ever worked. You would see a difference in version in some hundred packages or whatever. Then run the sync only to find that some fifty or something packages were still on the wrong version. Then run the sync again to reduce that number a bit more. Then again. Then again.

It was a nightmare, but it was the process in place when I started there and I didn't really know RHEL enough to see a better way. This was an official tool made for the job! I just assumed it was the best one for it.

Until it royally screwed up my entire day once.

In the middle of a sync the test server just completely broke down. This had just never happened before and it took me quite some time to figure it out.

When trying to adapt to the template RHN had inadvertently removed yum and bash from the system. It rendered the VM entirely unusable. After a lot of hours I just had to revert the entire server from the backup and was set on having to go through the entire procedure again and hope for the best.

But then I just decided that no, I never ever wanted to torment myself with that crap again.

Instead I logged into another VM that was already running the baseline I needed. I ran

rpm -qa

to get a list of all the installed packages. Then I read the rpm man page a bit, and guess what? I can't remember the exact arguments, but you can actually just give it a list of packages and say "this is what I want; make it so" and it will install all those packages with those exact versions and remove everything else from the system.

It never failed. It never took more than half an hour. It never had to be run more than once.

After that I just checked in a text file with all the packages for the baseline into our version control and never logged into RHN again.

I was not surprised when Red Hat retired that service.

-- CC0 Björn Wärmedal