Mon, 16 Jun 2014

Bodhi2 Fedora Activity Day

The Bodhi2/Taskotron Fedora Activity Day happened earlier this month! A bunch of us gathered in Denver for a few days and worked on some of our critical releng & qa infrastructure. The hackfest was held in a conference room in my apartment building, which worked out quite nicely for the amount of people that we had. The hotel was right up the road, and we were able to walk to a lot of awesome spots, like the 1UP Barcade :).

It was great to have folks from various corners of Fedora together in the same room for a few days. As it is, we get a lot done remotely, but being able to leverage the high-bandwidth face-to-face time is extremely valuable, especially when coming to consensus on difficult architectural decisions. We used Gobby to collaborate on a long list of action items, and chipped away most of it. Thankfully, Bodhi has enough layers that we were all able to split up and dive into different corners of the code without stepping on each others' toes.

Up until now, our releng stack in staging has always been less than useful. We've never able to do a full build->update->push->run, and have had to rely on testing certain codepaths in production. Not only that, but Bodhi's "masher" never really had proper unit tests, so pushing out major changes to that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our staging setup and made it so we can actually use it. I made a lot of headway on porting the Bodhi masher to a new fedmsg-driven architecture, while writing unit tests for it along the way. I'm hopeful that we can write tests for every part of the "push" process, and then optimize the hell out of it :)

While Mathieu and I mainly focused on back-end hacking, Ralph and Ricky made some fantastic headway on the front-end. Ralph started working on the revamped New Update Form, which is already significantly better than the original. The idea here is that the maintainer should only need to provide the package name(s), and Bodhi will automatically find the corresponding candidate builds, related bugs, and eventually it will pull in all candidate deps as well (or tell if you if any need to be rebuilt). It would also be very convenient to be able to "push this entire stack again". Ideally, I'd love to make it so that folks maintaining large stacks, like GNOME, shouldn't need to use a Google doc to coordinate their mega-updates ;)

Ralph also started revamping the new karma system (check out his screencast here). We don't have any of the policy in place to enforce it yet, but eventually we'd like the maintainers to be able to define custom policy constraints for their updates. For example, you could say "only allow this update to go to the stable repo once this specific bug or test case has been confirmed as fixed/passing".

Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on the front page using the fedmsg datagrepper widget that Ralph blogged about recently. Other front-end improvements include libravatar support for all users, proper markdown rendering with previews and image support, and of course a konami code easter-egg ;)

I was going to post a bunch of screenshots here, but Ralph just deployed a development instance of Bodhi2 that folks can play around with instead: (it's a dev instance, so don't expect it to always be up/functional).

Some other corners of Bodhi that have seen improvements recently:

The API. The Bodhi webapp is written in Python using the excellent Pyramid web framework. On top of that we are using Cornice, which makes it really easy to build & document standards-compliant web services with Pyramid. Thanks to colander validation/deserialization schemas and our custom ACLs and validators, we are able to write dead-simple views that can safely assume that all of the data we are dealing with is valid and in the right format, and that the user has the appropriate permissions. Cornice also has a Sphinx plugin that auto-generates our API documentation. So not only do we now have a semi-RESTful standards-compliant self-documenting API, but Ralph also added a support for rendering almost every service as an RSS feed as well.

Regarding the Bodhi Python APIs, I've begun porting them to the new python-fedora OpenIDBaseClient (see bodhi/ Since a large percentage of the API usage is through the current python-fedora BodhiClient, I'd like to try our best to maintain compatibility--at least for a version or two with deprecation warnings if we have to. I am really looking forward to finally being able to trash our old TurboGears1 FAS visit/identity layer in favor of FedOAuth.

On top of the Python API lies the bodhi-client. I recently ported the basic functionality over using the click module, which makes it really easy to write complex command-line tools (see bodhi/ Since the current bodhi-client is an absolute mess, this is one area that I'm actually okay with breaking backwards-compatibility to a certain extent. Having a proper command structure, similar to the Koji cli, is well worth some cli flag changes in my opinion.

In a similar fashion, Mathieu implemented a great release management tool for Bodhi admins. Currently, creating a release and changing it's pre-release status involves using the TurboGears Python shell, creating SQLObject instances by hand, editing config files, etc. This tool will make it dead simple for releng to create new releases and manage all of the pre-GA state changes that happen along the way.

Performance was another key area of development. The app is fairly snappy now, but there is still a ton of room for improvement. The pyramid-debugtoolbar has been amazingly useful for us so far. It let's us analyze every SQL statement made, it does full-stack profiling, and it lets us execute commands in every layer of a traceback. Along with that, Ralph added a SQLAlchemy event hook to our unit tests to ensure that certain changes don't drastically change how many SQL statements are getting executed. With regard to Masher performance, there is still a lot of low-hanging fruit there. We saw a drastic boost in mash speed recently when Kevin discovered that a couple of releng machines weren't using virtio. This brought the mash time of EPEL5 updates from 45 minutes down to around 15. In the new version of the masher, updates are grouped by tag and then processed in a separate threads. Security updates and critical bugfixes will take priority over enhancements, and there are also things that we can do to make the bits hit the mirrors faster once we're done mashing.

Another corner of Bodhi that was the topic of discussion was around notifications. Bodhi currently sends way too much email, most of which I'm assuming gets ignored. There are a couple of mails that are a bit more important, like the update announcement emails that get sent to package-announce, and the updates-testing digest that goes to the test-list. The consensus that we came to was that we are going to attempt to use FMN to allow people to configure what messages they want to receive, and where (IRC, email, etc). This alleviates the need to build an email layer into Bodhi2, and allows us to focus on publishing fedmsgs only, letting FMN do the rest.

As far as the transition to Bodhi2 goes, we're going to try our best to not break the world in the process. Ralph mentioned the potential timeline in his blog post, and we still have lots of work to do before then. In order to help ease this transition, I created a wiki page to track the consumers of Bodhi's APIs, so we can make the appropriate changes to those codebases before launch. Please feel free to update it with any that I left out.

Overall, it was a very successful FAD. We got a ton of stuff done, ate a bunch of great food, and had a lot of fun in the process. I didn't cover everything that we worked on, so checkout the blog posts from threebean, bochecha, and nirik for more details on other things that got done. If you're interested in getting involved with Bodhi2, grab the code, checkout the open issues, and hop in #fedora-apps on Freenode.

Also, I'll be giving a presentation at Flock in Prague this year on "Evolving the Fedora Updates process", which will cover the history of pushing updates as well as an in-depth dive into the new bodhi2 stack.

posted at: 16:49 | link | Tags: , , | 1 comments

Posted by Valeriy at Fri Dec 5 20:15:24 2014

It's rather more about the time-of-day chgenas suddenly landing very close to the final change deadline, with little warning that they were coming at all; it certainly completely surprised me. Maybe if the spins owners had been aware that the change was coming, and if it had landed earlier, we would have been able to come up with a better resolution.