Mon, 26 Jan 2009

bodhi updates push process

Bodhi's push process is something that is usually quite opaque to Fedora package maintainers. Once an update request goes into bodhi, the developer sits back and waits for the update to go to where it needs to go. The ball is then in releng's court, as they must sign the packages, and tell bodhi to begin the push. From there, bodhi does it's thing for a while, and then updates magically end up on our users machines. Yay!

Pushing updates used to take the better part of a day, mostly due to dumb code and lots of filesystem churn over NFS. Thankfully, a lot of the code is now much smarter, and people like jkeating and mmcgrath have been helping to address the NFS & infrastructure bottlenecks.

Hopefully I can help shed some light on one of the dark corners of bodhi known as The Masher. Here are some statistics of the last updates push that happened earlier today.

Initial push request from releng
Check koji tag / bodhi status consistency38s
Move all of the build tags in Koji 9m32s
Update the comps CVS module 11s
Mash f9-updates-testing 4m16s
Mash f9-updates 1h3m8s
Mash f10-updates-testing 12m43s
Mash f10-updates 37m51s
Set update ids, state modifications, updates-testing digest generation 1m57s
Generate updateinfo.xml 5m55s
Repo sanity checks & symlinking to go live 1m4s
Cache latest repodata, and remove old 1m14s
Wait for updates to hit the master mirror 1h1s
Send update notices, update/close bugs, notify developers/commenters 11m11s
Total3h49m42s


So we've obviously made some great improvements here, and once the signing server is deployed, you can probably expect a much more frequent/consistent flow of updates. However, I definitely think there is still a lot of low-hanging fruit in this process, and many steps can probably be done in parallel. We're going to be adding DeltaRPM generation into the mix in the near future, so I'll give an update a bit later with some details as to how that effects the process.

Anyway... if you know Python, and enjoy optimizing code -- come talk to me :)



posted at: 23:15 | link | Tags: , , , | 0 comments

Sat, 13 Dec 2008

Time spent in updates-testing purgatory

Will Woods asked me on IRC earlier today how easy it would be to determine the amount of time Fedora updates spend in testing within bodhi. It turned out to be fairly easy to calculate, so I thought I would share the code and results.

from datetime import timedelta
from bodhi.model import PackageUpdate

deltas = []
occurrences = {}
accumulative = timedelta()

for update in PackageUpdate.select():
    for comment in update.comments:
        if comment.text == 'This update has been pushed to testing':
            for othercomment in update.comments:
                if othercomment.text == 'This update has been pushed to stable':
                    delta = othercomment.timestamp - comment.timestamp
                    deltas.append(delta)
                    occurrences[delta.days] = occurrences.setdefault(delta.days, 0) + 1
                    accumulative += deltas[-1]
                    break
            break

deltas.sort()
all = PackageUpdate.select().count()
percentage = int(float(len(deltas)) / float(all) * 100)
mode = sorted(occurrences.items(), cmp=lambda x, y: cmp(x[1], y[1]))[-1][0]

print "%d out of %d updates went through testing (%d%%)" % (len(deltas), all, percentage)
print "mean = %d days" % (accumulative.days / len(deltas))
print "median = %d days" % deltas[len(deltas) / 2].days
print "mode = %d days" % mode

4878 out of 10829 updates went through testing (45%)
mean = 17 days
median = 11 days
mode = 6 days

So, it seems that the majority of updates leave updates-testing in less than a week. This is interesting when taking into consideration the testing workflow mechanisms that bodhi employs. An update can go from testing to stable in two ways: 1) The update's karma can reach an optional stable threshold, and automatically get pushed to the stable repository based on positive community feedback. 2) The developer can request that the update be marked as stable. After an update sits in testing for two weeks, bodhi will send the developer nagmail, which seems to help mitigate stale updates. When initially deploying bodhi, I thought that we would get bogged down with a ton of stale testing updates and would have to implement a timeout to have them automatically get marked as stable. This is still a viable option (which would require FESCo rubberstamping), but I'm quite surprised to see how effective this community-driven workflow is already. Now we just need to encourage more people to use it :)

Due to the limitations of the current model I couldn't figure out an easy way to determine which updates were marked as stable by positive community feedback. This issue will be assessed with the long-awaited SQLAlchemy port that I will hopefully finish up at some point early next year.



posted at: 02:13 | link | Tags: , , , | 1 comments

Wed, 10 Sep 2008

bodhi 0.5!

As many have already noticed, I performed a large bodhi upgrade recently. A few weeks ago, during The Incident, I was forced to perform what was originally going to be a week long bodhi migration and upgrade, overnight. During the past two weeks I've pushed out 24 revisions of bodhi to our infrastructure, fixing various show-stoppers, and helping to make sure that updates got out the door.

One of the most noticable changes is that bodhi is much more responsive. Previously, bodhi was a single python process, running on a single server. This single server was also responsible for composing the updates repositories, and rawhide, among lots of other bodhi-related churn. This lead to much pain and suffering for all.

The bodhi deployment has since changed. All bodhi requests are now load balanced to a bunch of app servers, each running mod_wsgi with multiple bodhi processes, each with multiple threads. All of the hard work is now done on an isolated releng server. This separate bodhi "masher" is now responsible for composing repositories, updating bugs, generating update notices, sending emails, extended metadata generation, and calculating metrics. I also added support for inter-bodhi communication, which allows our bodhi web frontends to kick off push requests to our bodhi-masher instance.

Some of the new features in this release:

Bodhi is far from being feature complete. Some new features in the pipeline:

As always,

Also, if you're currently having issues with the bodhi client, a fixed version will be going out with the next batch of updates. For the impatient, you can pull fixed versions from koji (also, make sure your Makefile.common is up to date): 

    koji download-build --arch=noarch python-fedora-0.3.5-1.fc10
    koji download-build --arch=noarch bodhi-0.5.2-1.fc9



posted at: 16:20 | link | Tags: , , | 1 comments

Mon, 16 Jun 2008

Fedora 7 Update Metrics

Fedora 7 reached End of Life on Friday, June 13th. Here are some graphs that I generated with bodhi.









posted at: 15:24 | link | Tags: , , | 2 comments

Sat, 08 Dec 2007

Fedora update metrics

Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.

I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.

When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.



posted at: 19:05 | link | Tags: , , , , , | 0 comments

Tue, 20 Nov 2007

make update

I updated Fedora's Makefile.common to support an 'update' target, which has now been comitted. This gives developers the ability to push an update into bodhi with a single command.

Once your package has been built, jump into your branch and do the following:

[lmacken@tomservo F-8]$ make update
This will drop you into a template where you can specify various fields. Any bug numbers mentioned in the most recent RPM ChangeLog will be pre-populated as well. 
# [ nethack-3.4.3-15.fc8 ]
# type=[S|B|E] (S=security, B=bugfix, E=enhancement) (required)
# request=[T|S] (T=testing, S=stable) (default: testing)
# bug=123,456
# all other text will be considered to be part of the update notes
type=B
bug=221948,221692

Make and save your changes, and then your update will be submitted to bodhi. 

Creating new update for nethack-3.4.3-15.fc8
Update successfully created

================================================================================
     nethack-3.4.3-15.fc8
================================================================================
    Release: Fedora 8
     Status: pending
       Type: bugfix
      Karma: 0
    Request: testing
       Bugs: 221948 - nethack-recover looks in the wrong directory
           : 221692 - Nethack font warning
  Submitter: lmacken
  Submitted: 2007-11-20 14:25:55

  http://admin.fedoraproject.org/updates/F8/pending/nethack-3.4.3-15.fc8

See the UpdatingPackageHowTo for more information on updating your packages. You can find more details on the bodhi console client here.



posted at: 00:00 | link | Tags: , | 2 comments

Tue, 13 Nov 2007

bodhi command-line client

The bodhi-client package should be making its way to an updates-testing repository near you! Not only does this command-line tool give developers easier access to bodhi, but also provides some new features to help people get more involved with testing updates and providing useful feedback.

I wrote up some documentation on various usage examples of the tool, which can be found on the bodhi wiki. I also submitted a Makefile.common patch that, once applied, will allow you to run `make update` from your package branch. This will drop you into a new update template, and will then submit your update straight to bodhi.

Some noteworthy features in the bodhi-client, aside from the normal bodhi functionality:

I also upgraded our production bodhi instance yesterday, which pulled in a ton of bugfixes and some new features, such as:

As always, patches/questions/criticisms/comments are welcome. You can file tickets in the usual place. Happy hacking!



posted at: 09:42 | link | Tags: , , | 0 comments

Mon, 01 Oct 2007

Use your Nose!

Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.

So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.

For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update. 

import urllib, cherrypy
from turbogears import update_config, database, testutil, url

update_config(configfile='dev.cfg', modulename='bodhi.config')
database.set_db_uri("sqlite:///:memory:")

class TestControllers(testutil.DBTest):

    def test_unauthenticated_update(self):
        params = {
                'builds'  : 'TurboGears-1.0.2.2-2.fc7',
                'release' : 'Fedora 7',
                'type'    : 'enhancement',
                'bugs'    : '1234 5678',
                'cves'    : 'CVE-2020-0001',
                'notes'   : 'foobar'
        }
        path = url('/save?' + urllib.urlencode(params))
        testutil.createRequest(path, method='POST')
        assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'. 
[lmacken@tomservo bodhi]$ nosetests
.................................
----------------------------------------------------------------------
Ran 33 tests in 16.798s

OK
Now, for the fun part. Nose comes equipped with a profiling plugin that will profile your test cases using Python's hotshot module. So, I went ahead and added a 'profile' target to bodhi's Makefile: 
profile:
    nosetests --with-profile --profile-stats-file=nose.prof
    python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
Now, typing 'make profile' will execute and profile all of our unit tests, and spit out the top 20 method calls -- ordered by internal time and call count. 
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile --profile-stats-file=nose.prof
.................................
----------------------------------------------------------------------
Ran 33 tests in 42.878s

OK
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
         800986 function calls (702850 primitive calls) in 42.878 CPU seconds

   Ordered by: internal time, call count
   List reduced from 3815 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       14   13.675    0.977   13.675    0.977 /usr/lib/python2.5/socket.py:71(ssl)
       31   10.683    0.345   10.683    0.345 /usr/lib/python2.5/httplib.py:994(_read)
2478/2429    9.297    0.004    9.677    0.004 :1()
        1    0.604    0.604    0.604    0.604 /usr/lib/python2.5/commands.py:50(getstatusoutput)
     2999    0.536    0.000    0.539    0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/sqliteconnection.py:177(_executeRetry)
   105899    0.448    0.000    0.773    0.000 Modules/pyexpat.c:871(Default)
       60    0.327    0.005    1.102    0.018 /usr/lib/python2.5/site-packages/kid/parser.py:343(_buildForeign)
   105899    0.325    0.000    0.325    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:452(_default)
     3396    0.280    0.000    0.420    0.000 /usr/lib/python2.5/site-packages/cherrypy/config.py:107(get)
     2965    0.263    0.000    0.263    0.000 /usr/lib/python2.5/logging/__init__.py:364(formatTime)
44964/6587    0.238    0.000    0.252    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:156(_pull)
       60    0.116    0.002    0.116    0.002 /usr/lib/python2.5/site-packages/kid/compiler.py:38(py_compile)
     8127    0.114    0.000    0.114    0.000 /usr/lib/python2.5/site-packages/cherrypy/_cputil.py:311(lower_to_camel)
     8982    0.110    0.000    0.137    0.000 /usr/lib/python2.5/site-packages/sqlobject/dbconnection.py:902(__getattr__)
13740/4044    0.108    0.000    2.176    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:209(_coalesce)
24353/4026    0.107    0.000    2.143    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:174(_track)
     3170    0.093    0.000    0.398    0.000 /usr/lib/python2.5/logging/__init__.py:405(format)
        1    0.082    0.082    0.082    0.082 /usr/lib/python2.5/site-packages/rpm/__init__.py:5()
     4777    0.081    0.000    1.320    0.000 /usr/lib/python2.5/site-packages/kid/serialization.py:564(generate)
  759/176    0.074    0.000    0.210    0.001 /usr/lib/python2.5/sre_parse.py:385(_parse)



posted at: 09:40 | link | Tags: , , , , | 1 comments

Sat, 15 Sep 2007

bodhi 0.2.0

I'm pleased to announce that bodhi 0.2.0 has been released and deployed. This release has fixed a ton of issues and introduces many new features, such as:

This release introduces many database changes from the previous version, so it will be much easier to jump back into the release-early-release often cycle.

Soon to come:

I would also like to thank Till Maas and Tim Lauridsen for taking the time to help out and do some great work.

There is still much to be done with bodhi, so if you're interested in helping out, you can setup a local bodhi development playground with just a few commands and dive in.

As always, please file any bugs or enhancement requests here.



posted at: 10:44 | link | Tags: , , | 0 comments

Wed, 15 Aug 2007

Bodhi Presentation at Virtual FUDCon

I'm going to be giving a presentation on Bodhi at the Virtual FUDCon August 15th at 1900-2000 UTC (3pm EST). I'll be covering the past, present, and future of Fedora updates, as well as diving into some technical aspects of the Bodhi project. If all goes to plan, this will be a live audio-enabled presentation, where people can follow the slides as well as converse on IRC. I try to post my slides, and [hopefully] a recording of the presentation next week.

*Update* You can find the IRC transcripts from my presentation here, and my slides here.



posted at: 23:49 | link | Tags: , , | 0 comments

Wed, 14 Feb 2007

bodhi

So I've been spending the majority of my free time recently getting the new Fedora Updates System ready for Fedora 7, which is quickly approaching. Since the code is going to exist in multiple instances for different projects (Fedora, RH, etc), and is already fairly modular and distro-independent, I decided it would be best to give the code a new name, and a new home: Bodhi emerged. Being hardcore into Zen recently, I feel like the name seems to fit the goals of the project nicely.

A Buddhist term for the wisdom by which one attains enlightenment. Bodhi is the opposite of ignorance, the insight into reality which destroys mental afflictions and brings peace.

Previously, the updates system code was in CVS, and bugs were to be filed in OTRS, which ended up being extremely painful to work with. After switching to our new shiny hosting setup, Bodhi is now using Mercurial for source control, and Trac for managing the project milestones, bugs, and documentation. Overall, I've been extremely impressed with Mercurial and Trac so far.

Here are some screenshots of the current development version. More screenshots can be found on the bodhi wiki.

So what's going to change for Fedora developers? Well, currently any [extras] developer can push out updates to their packages for any release with ease (cvs commit && make tag build). If your updated package fixes 4 bugs and 2 CVE's, the end user who is blindly installing this update has no idea. Not only is there no differentiation between bugfix/enhancement/security updates, but none of them go through any sort of QA whatsoever. This is what bodhi is aiming to change.

I've been designing this system completely independent of any Buildsystem (mainly because the unveiling of brew is still churning slowly), so the process of building packages is going to remain the same (for now). Once your packages is built, you will enter it into bodhi via a web form, or [eventually] a command-line tool (Ideally, I'd like to see the process of preparing/testing/releasing a package to be a single point of interaction; either completely from command-line or in web interface). From here, your update will undergo various checks (and will eventually dispatch tests to the beaker test lab). If all goes well, your update will get signed and pushed, all referenced Bugzillas will be updated/closed, and an update notification mail will be sent to the appropriate list.

updates-testing is currently our "testing" repository that developers can choose to push their updates out to before going to 'final'. This is not very appealing for devs, as it requires them to 'Move to final' after a certain amount of time when they feel it is necessary, and the system will nag them if they don't. The updates-testing repo is not widely used, and gives testers no incentive or way to give feedback, other than filing a bug. One of the goals of bodhi is to require all updates to go through the updates-testing process, and provide a simple interface for developers/testers/users to provide positive/negative feedback regarding an update, and also make it trivial to submit bugs about them as well (we threw around some ideas about making a reusable QA feedback widget for this at a recent Fedora QA meeting). After a certain number of positive responses (or after a given length of time with no negative responses), an update will then able to make it's way to the stable updates repository.

Hackery
I've written a ton of code so far, but there is still much work to be done to accomplish all of the goals mentioned above. I layed out a few milestones for bodhi, 1.0 being the minimal functionality needed for the release of Fedora 7. I also added a 1.1 milestone for features that aren't crucial for minimal functionality, but are definitely important, and also a wishlist milestone for features that might be nice to have someday (metrics, rss feeds, etc).

With FUDCon and the Fedora Infrastructure HackFest coming up, I'm encouraging anyone interested to dive in help out in any way they want. If you can't make it out to Boston, just hop on IRC. If you would like to see a feature in bodhi, feel free to add a ticket to any of the milestones.

This system holds much potential for revolutionizing the way Fedora releases evolve, and how developers and testers interact with it. Since this is going to be utilized by all package maintainers, it's important that the system is molded to fit the needs of the developers and testers; so if you have any suggestions/improvements/fixes, don't hesitate: Contribute.



posted at: 21:12 | link | Tags: , , | 5 comments