Fri, 26 Jun 2015

Release Tools and Infrastructure FAD

The Release Tools and Infrastructure Fedora Activity Day happened recently at the Red Hat office in Westford, Massachusetts. The goal was to bring our release tooling and processes up to speed with the current and future demands of the Fedora Project. Since there are a ton of moving parts of the Fedora Release Engineering community that need work, many of us split out into groups to tackle various components.

Even though there were a ton of shiny tasks that I would have loved to dive into, like pungi4, createrepo_c, koji 2.0, etc, I felt like my cycles were best spent working on Bodhi 2.0.

Bodhi is a critical piece of infrastructure that has been in production for over 7 years now, and it's initial incarnation was deployed internally to Red Hat for Fedora Core prior to that. It is currently a cross-section between releng, qa, and infra, and is responsible for shipping updated bits to users while facilitating community feedback. It started with pushing out updated RPMs initially, but since F21 it has also started updating Atomic OSTrees as well. With the number of requirements for updated deliverables growing each release, it's clear that our current architecture does not scale with this appropriately.

The next generation of Bodhi has been in the works for a while now, and is almost ready for testing in staging. So I started off the FAD by getting the current bodhi2 stack built for RHEL7. This turned out to be a non-trivial task, as I had to build a bunch of complex dependencies like python-cryptography, python-six, and python-py/pytest (which have circular deps on eachother). Thankfully, I now have a bodhi2 copr setup for building the extra pieces of the stack until they are available in RHEL7+EPEL.

I then began writing a new bodhi2 ansible role and got an instance deployed to staging. There is still a bunch of work to be done before the staging instance is usable, but that mostly revolves around configuration and hopefully shouldn't be too far off. Once we have something we can start playing with, there is going to be a bunch of work needed to port the existing tools over to use the new API. I'm hoping we can keep the python-fedora bindings "stable", but there are still plenty of other things that hit the JSON API directly that will most likely need tweaking.

As far as future goals for the Bodhi architecture goes, I'd really like to see us split it up into a couple of different components. Ideally, I think the majority of it would revolve around the web frontend & RESTful JSON API that facilitates community testing. There's already a pretty awesome ecosystem that has been built around this, with client-side QA tools like fedora-easy-karma and fedora-gooey-karma which make it easy people test and give feedback to things in updates-testing that they have installed.

Then there is "The Masher" backend, which currently does most of the heavy lifting. Most of this could ideally be pushed into pungi4 and koji plugins, for things like mashing updates repos, composing ostrees, generating updated images/installers, etc. This way, bodhi would simply handle tagging things in koji and then orchestrate the creation of the updated bits with pungi4, update/close bugs, generate updateinfo metadata, etc. Then we can finally leverage our build farm to perform the long-running composition tasks in parallel, instead of doing them all linearly on a single releng machine.

So bodhi would still need to handle the orchestration of these processes, but I could also see us pushing this stuff into the theoretical "ComposeDB" idea that we have been tossing around as well. This could then potentially take care of rebuilding/composing whatever needs to be updated based on what has changed. I wasn't directly involved with the detailed requirements discussions between Dennis, Ralph, etc.. but from my perspective I am really excited about the idea of a central tool that knows about all release artifacts, what is contained within them, and how to produce them. Then when something like a critical security update gets released, it can then kick off rebuilds of only the things that need to be rebuilt, as opposed to having to rebuild the world by hand each time.

Another goal of bodhi2, which has yet to be implemented, is to support additional artifacts such as docker containers, OSTrees, and anything else we may want in the future. This would entail kicking off the process to build and "push" these updated artifacts, as well as encourage & facilitate community testing and feedback around them. I think once we have Koji plugins to create these new artifacts, then making bodhi flexible enough to handle the testing/karma cycle for them shouldn't be very difficult.

Optimizing the workflow of pushing updates is also another big task. After talking with Jon Disnard (masta) about the package signing process, I ended up hacking together a tool that spits out a list of packages that are not in the current push but are ready to be signed for the next one. This will allow releng signers to kick off a push and then start prepping for the next one. I am hopeful that someday we can figure out a better way to streamline the gpg signing in the push process, but this is something that we have yet to properly flesh out yet. Ideally, we'd enter the signing passphrase once at the beginning of the process, and then end up with gpg signed rpms, repo metadata, ostrees, etc.

It's also worth noting that the releng git repo has been migrated from fedorahosted to Pagure. Since Fedora prides itself on being comprised of only free and open source software, I think we'll start to see more and more of our project repos moving over to this new platform away from GitHub in the future. The few issues that it hit with my usual workflow were quickly fixed and deployed by pingou. On the topic of Fedora Hosted, I wrote a little script to figure out what percent of projects have been active (code pushes, wiki edits, ticket activity) since 2014-01-01, and the answer is 129 out of 660 (19.55%).

So overall I though this was a very productive FAD. Along with getting a lot of code written, it was also a great opportunity to sync-up with the stakeholders of the various releng components and help flesh-out the next generation of our release engineering stack.

For other perspectives, checkout the varous writeups from Ralph and Adam.

posted at: 22:54 | link | Tags: | 1 comments

Posted by Tsing at Wed Dec 30 12:05:40 2015

Wow, that's cool!