Mon, 26 Jan 2009
bodhi updates push process
Bodhi's push process is something that is usually quite opaque to Fedora package maintainers. Once an update request goes into bodhi, the developer sits back and waits for the update to go to where it needs to go. The ball is then in releng's court, as they must sign the packages, and tell bodhi to begin the push. From there, bodhi does it's thing for a while, and then updates magically end up on our users machines. Yay!
Pushing updates used to take the better part of a day, mostly due to dumb code and lots of filesystem churn over NFS. Thankfully, a lot of the code is now much smarter, and people like jkeating and mmcgrath have been helping to address the NFS & infrastructure bottlenecks.
Hopefully I can help shed some light on one of the dark corners of bodhi known as The Masher. Here are some statistics of the last updates push that happened earlier today.
| Initial push request from releng | |
| Check koji tag / bodhi status consistency | 38s |
| Move all of the build tags in Koji | 9m32s |
| Update the comps CVS module | 11s |
| Mash f9-updates-testing | 4m16s |
| Mash f9-updates | 1h3m8s |
| Mash f10-updates-testing | 12m43s |
| Mash f10-updates | 37m51s |
| Set update ids, state modifications, updates-testing digest generation | 1m57s |
| Generate updateinfo.xml | 5m55s |
| Repo sanity checks & symlinking to go live | 1m4s |
| Cache latest repodata, and remove old | 1m14s |
| Wait for updates to hit the master mirror | 1h1s |
| Send update notices, update/close bugs, notify developers/commenters | 11m11s |
| Total | 3h49m42s |
So we've obviously made some great improvements here, and once the signing server is deployed, you can probably expect a much more frequent/consistent flow of updates. However, I definitely think there is still a lot of low-hanging fruit in this process, and many steps can probably be done in parallel. We're going to be adding DeltaRPM generation into the mix in the near future, so I'll give an update a bit later with some details as to how that effects the process.
Anyway... if you know Python, and enjoy optimizing code -- come talk to me :)
posted at: 23:15 | link | | 0 comments
Sat, 13 Dec 2008
Time spent in updates-testing purgatory
Will Woods asked me on IRC earlier today how easy it would be to determine the amount of time Fedora updates spend in testing within bodhi. It turned out to be fairly easy to calculate, so I thought I would share the code and results.
from datetime import timedelta
from bodhi.model import PackageUpdate
deltas = []
occurrences = {}
accumulative = timedelta()
for update in PackageUpdate.select():
for comment in update.comments:
if comment.text == 'This update has been pushed to testing':
for othercomment in update.comments:
if othercomment.text == 'This update has been pushed to stable':
delta = othercomment.timestamp - comment.timestamp
deltas.append(delta)
occurrences[delta.days] = occurrences.setdefault(delta.days, 0) + 1
accumulative += deltas[-1]
break
break
deltas.sort()
all = PackageUpdate.select().count()
percentage = int(float(len(deltas)) / float(all) * 100)
mode = sorted(occurrences.items(), cmp=lambda x, y: cmp(x[1], y[1]))[-1][0]
print "%d out of %d updates went through testing (%d%%)" % (len(deltas), all, percentage)
print "mean = %d days" % (accumulative.days / len(deltas))
print "median = %d days" % deltas[len(deltas) / 2].days
print "mode = %d days" % mode
4878 out of 10829 updates went through testing (45%)
mean = 17 days
median = 11 days
mode = 6 days
So, it seems that the majority of updates leave updates-testing in less than a week. This is interesting when taking into consideration the testing workflow mechanisms that bodhi employs. An update can go from testing to stable in two ways: 1) The update's karma can reach an optional stable threshold, and automatically get pushed to the stable repository based on positive community feedback. 2) The developer can request that the update be marked as stable. After an update sits in testing for two weeks, bodhi will send the developer nagmail, which seems to help mitigate stale updates. When initially deploying bodhi, I thought that we would get bogged down with a ton of stale testing updates and would have to implement a timeout to have them automatically get marked as stable. This is still a viable option (which would require FESCo rubberstamping), but I'm quite surprised to see how effective this community-driven workflow is already. Now we just need to encourage more people to use it :)
Due to the limitations of the current model I couldn't figure out an easy way to determine which updates were marked as stable by positive community feedback. This issue will be assessed with the long-awaited SQLAlchemy port that I will hopefully finish up at some point early next year.
posted at: 02:13 | link | | 1 comments
Wed, 10 Sep 2008
bodhi 0.5!
As many have already noticed, I performed a large bodhi upgrade recently. A few weeks ago, during The Incident, I was forced to perform what was originally going to be a week long bodhi migration and upgrade, overnight. During the past two weeks I've pushed out 24 revisions of bodhi to our infrastructure, fixing various show-stoppers, and helping to make sure that updates got out the door.
One of the most noticable changes is that bodhi is much more responsive. Previously, bodhi was a single python process, running on a single server. This single server was also responsible for composing the updates repositories, and rawhide, among lots of other bodhi-related churn. This lead to much pain and suffering for all.
The bodhi deployment has since changed. All bodhi requests are now load balanced to a bunch of app servers, each running mod_wsgi with multiple bodhi processes, each with multiple threads. All of the hard work is now done on an isolated releng server. This separate bodhi "masher" is now responsible for composing repositories, updating bugs, generating update notices, sending emails, extended metadata generation, and calculating metrics. I also added support for inter-bodhi communication, which allows our bodhi web frontends to kick off push requests to our bodhi-masher instance.
Some of the new features in this release:
- A much more flexible karma automatism scheme. Stable/unstable karma thresholds are now fully configurable
- Support for bug aliases
- A 'newpackage' update type
- Newer updates which obsolete older ones will now inherit their bugs and notes.
- A shiny new API in the fedora.client.bodhi module
- Lots of improved releng and security team support, making our lives little easier
- An improved `make update` template (be sure to update your Makefile.common)
- Some new bodhi-client features
- Creating updates for multiple releases using a single form Note: This is not perfect yet. You can use the "New Update Form" to add any number of builds for any number of releases, but bodhi will still create a single update for each. This issue will be resolved in the next major bodhi release, which will contain a full model redesign.
- updateinfo.xml generation takes about 20 seconds, instead of 20 minutes.
- A lot of metrics enhancements
- A ton of bug and usability fixes
Bodhi is far from being feature complete. Some new features in the pipeline:
- DeltaRPM generation
- Security issue (CVE) tracking and triaging
- Dependency closure verification, utilizing the power of rpmgrok.
- A complete remodeling from SQLObject to SQLAlchemy, which is almost complete, will give us a lot more flexibility, speed, and power over our update model. This will also allow for things such as having multi-builds for multiple releases in a single update.
As always,
- File tickets here
- Help out here
- Subscribe to the bodhi mailing list here
- You can always find bodhi here
Also, if you're currently having issues with the bodhi client, a fixed version will be going out with the next batch of updates. For the impatient, you can pull fixed versions from koji (also, make sure your Makefile.common is up to date):
koji download-build --arch=noarch python-fedora-0.3.5-1.fc10
koji download-build --arch=noarch bodhi-0.5.2-1.fc9
posted at: 16:20 | link | | 1 comments
Mon, 16 Jun 2008
Fedora 7 Update Metrics
Fedora 7 reached End of Life on Friday, June 13th.
Here are some graphs that I generated with bodhi.






- Fedora 7 updates receieved a total of 501 pieces of feedback from testers, 33 of which were anonymous.
posted at: 15:24 | link | | 2 comments
Sat, 08 Dec 2007
Fedora update metrics
Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.
I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.
When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.
posted at: 19:05 | link | | 0 comments
Tue, 20 Nov 2007
make update
I updated Fedora's Makefile.common to support an 'update' target, which has now been comitted. This gives developers the ability to push an update into bodhi with a single command.
Once your package has been built, jump into your branch and do the following:
[lmacken@tomservo F-8]$ make update
This will drop you into a template where you can specify various fields. Any bug numbers mentioned in the most recent RPM ChangeLog will be pre-populated as well.
# [ nethack-3.4.3-15.fc8 ]
# type=[S|B|E] (S=security, B=bugfix, E=enhancement) (required)
# request=[T|S] (T=testing, S=stable) (default: testing)
# bug=123,456
# all other text will be considered to be part of the update notes
type=B
bug=221948,221692
Make and save your changes, and then your update will be submitted to bodhi.
Creating new update for nethack-3.4.3-15.fc8
Update successfully created
================================================================================
nethack-3.4.3-15.fc8
================================================================================
Release: Fedora 8
Status: pending
Type: bugfix
Karma: 0
Request: testing
Bugs: 221948 - nethack-recover looks in the wrong directory
: 221692 - Nethack font warning
Submitter: lmacken
Submitted: 2007-11-20 14:25:55
http://admin.fedoraproject.org/updates/F8/pending/nethack-3.4.3-15.fc8
See the UpdatingPackageHowTo for more information on updating your packages. You can find more details on the bodhi console client here.
posted at: 00:00 | link | | 2 comments
Tue, 13 Nov 2007
bodhi command-line client
The bodhi-client package should be making its way to an updates-testing repository near you! Not only does this command-line tool give developers easier access to bodhi, but also provides some new features to help people get more involved with testing updates and providing useful feedback.
I wrote up some documentation on various usage examples of the tool, which can be found on the bodhi wiki. I also submitted a Makefile.common patch that, once applied, will allow you to run `make update` from your package branch. This will drop you into a new update template, and will then submit your update straight to bodhi.
Some noteworthy features in the bodhi-client, aside from the normal bodhi functionality:
- Ability to view all updates-testing packages that you currently have
installed on your local machine, that you *could* be testing and providing
useful feedback for
- bodhi --testable
- Ability to view your update candidates (this is a fairly expensive
operation -- please use sparingly):
- bodhi --candidates
I also upgraded our production bodhi instance yesterday, which pulled in a ton of bugfixes and some new features, such as:
- Updates by default will now get submitted to into testing. This can easily be modified when using the web form, the bodhi client, and `make update`.
- Thanks to the new security bug tracking policy, we're now tracking CVEs using Bugzilla, thus bodhi no longer will ask you for CVE IDs. The less information that the developer has to type, the better. Read the policy for more details. Bodhi is not yet 100% compliant to the proposed changes, as it does not know about parent/tracking bugs, but should be soon.
- If you try and submit an update that is older than something already pending/testing, you will be prompted with a dialog that will give you the ability to instantly obsolete those updates.
As always, patches/questions/criticisms/comments are welcome. You can file tickets in the usual place. Happy hacking!
posted at: 09:42 | link | | 0 comments
Mon, 01 Oct 2007
Use your Nose!
Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.
So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.
For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update.
import urllib, cherrypy
from turbogears import update_config, database, testutil, url
update_config(configfile='dev.cfg', modulename='bodhi.config')
database.set_db_uri("sqlite:///:memory:")
class TestControllers(testutil.DBTest):
def test_unauthenticated_update(self):
params = {
'builds' : 'TurboGears-1.0.2.2-2.fc7',
'release' : 'Fedora 7',
'type' : 'enhancement',
'bugs' : '1234 5678',
'cves' : 'CVE-2020-0001',
'notes' : 'foobar'
}
path = url('/save?' + urllib.urlencode(params))
testutil.createRequest(path, method='POST')
assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'.
[lmacken@tomservo bodhi]$ nosetests
.................................
----------------------------------------------------------------------
Ran 33 tests in 16.798s
OK
Now, for the fun part. Nose comes equipped with a profiling plugin that will profile your test cases using Python's hotshot module.
So, I went ahead and added a 'profile' target to bodhi's Makefile:
profile:
nosetests --with-profile --profile-stats-file=nose.prof
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
Now, typing 'make profile' will execute and profile all of our unit tests, and spit out the top 20 method calls -- ordered by internal time and call count.
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile --profile-stats-file=nose.prof
.................................
----------------------------------------------------------------------
Ran 33 tests in 42.878s
OK
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
800986 function calls (702850 primitive calls) in 42.878 CPU seconds
Ordered by: internal time, call count
List reduced from 3815 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
14 13.675 0.977 13.675 0.977 /usr/lib/python2.5/socket.py:71(ssl)
31 10.683 0.345 10.683 0.345 /usr/lib/python2.5/httplib.py:994(_read)
2478/2429 9.297 0.004 9.677 0.004 :1()
1 0.604 0.604 0.604 0.604 /usr/lib/python2.5/commands.py:50(getstatusoutput)
2999 0.536 0.000 0.539 0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/sqliteconnection.py:177(_executeRetry)
105899 0.448 0.000 0.773 0.000 Modules/pyexpat.c:871(Default)
60 0.327 0.005 1.102 0.018 /usr/lib/python2.5/site-packages/kid/parser.py:343(_buildForeign)
105899 0.325 0.000 0.325 0.000 /usr/lib/python2.5/site-packages/kid/parser.py:452(_default)
3396 0.280 0.000 0.420 0.000 /usr/lib/python2.5/site-packages/cherrypy/config.py:107(get)
2965 0.263 0.000 0.263 0.000 /usr/lib/python2.5/logging/__init__.py:364(formatTime)
44964/6587 0.238 0.000 0.252 0.000 /usr/lib/python2.5/site-packages/kid/parser.py:156(_pull)
60 0.116 0.002 0.116 0.002 /usr/lib/python2.5/site-packages/kid/compiler.py:38(py_compile)
8127 0.114 0.000 0.114 0.000 /usr/lib/python2.5/site-packages/cherrypy/_cputil.py:311(lower_to_camel)
8982 0.110 0.000 0.137 0.000 /usr/lib/python2.5/site-packages/sqlobject/dbconnection.py:902(__getattr__)
13740/4044 0.108 0.000 2.176 0.001 /usr/lib/python2.5/site-packages/kid/parser.py:209(_coalesce)
24353/4026 0.107 0.000 2.143 0.001 /usr/lib/python2.5/site-packages/kid/parser.py:174(_track)
3170 0.093 0.000 0.398 0.000 /usr/lib/python2.5/logging/__init__.py:405(format)
1 0.082 0.082 0.082 0.082 /usr/lib/python2.5/site-packages/rpm/__init__.py:5()
4777 0.081 0.000 1.320 0.000 /usr/lib/python2.5/site-packages/kid/serialization.py:564(generate)
759/176 0.074 0.000 0.210 0.001 /usr/lib/python2.5/sre_parse.py:385(_parse)
posted at: 09:40 | link | | 1 comments
Sat, 15 Sep 2007
bodhi 0.2.0
I'm pleased to announce that bodhi 0.2.0 has been released and deployed.
This release has fixed a ton of issues and introduces many new features,
such as:
- Multi-build updates. You can add as many builds to a single update as you want. Bodhi will treat it as a single update, but will still send individual update notification mails for each build.
- New homepage widget that allows you to keep up with the happenings inside of bodhi. You can see the latest comments, testing/security/stable updates, and keep track of your own business.
- Enhanced notifications. If you commented on an update, you'll receive notifications when others comment on that update, and when that update is modified or changes states.
- Automatic closing of bugs is now optional.
- Build-completion improvements. Package names will be automagically completed, and if you type '-' after, it will complete versions as well.
- Positive/negative comments effect an updates 'karma'. After an update achieves a karma of 3, it will automatically be pushed to the stable updates repository. This will hopefully encourage testers to get involved with the updates-testing process a bit more, and will add some automation to the workflow
- Extended metadata (updateinfo.xml.gz) should start appearing in the repodata, which will allow tools like pup and the yum-security plugin to take advantage of it and do some nifty stuff.
- Reminders. You'll get nagged when your update sits around in testing for too long, and so on..
This release introduces many database changes from the previous version, so it will be much easier to jump back into the release-early-release often cycle.
Soon to come:- bodhi command-line client is almost ready to go. It needs to be polished up a bit, but should be released soon.
- RSS feeds and public details.
- Better build-completion based on koji tags.
- More sanity checking (koji buildroot verification, dependency closure, etc)
- Better handling of obsolete updates
I would also like to thank Till Maas and Tim Lauridsen for taking the time to help out and do some great work.
There is still much to be done with bodhi, so if you're interested in helping out, you can setup a local bodhi development playground with just a few commands and dive in.
As always, please file any bugs or enhancement requests here.
posted at: 10:44 | link | | 0 comments
Wed, 15 Aug 2007
Bodhi Presentation at Virtual FUDCon
I'm going to be giving a presentation on Bodhi at the Virtual FUDCon August 15th at 1900-2000 UTC (3pm EST). I'll be covering the past, present, and future of Fedora updates, as well as diving into some technical aspects of the Bodhi project. If all goes to plan, this will be a live audio-enabled presentation, where people can follow the slides as well as converse on IRC. I try to post my slides, and [hopefully] a recording of the presentation next week.
posted at: 23:49 | link | | 0 comments
Wed, 14 Feb 2007
bodhi
So I've been spending the majority of my free time recently getting the new Fedora Updates System ready for Fedora 7, which is quickly approaching. Since the code is going to exist in multiple instances for different projects (Fedora, RH, etc), and is already fairly modular and distro-independent, I decided it would be best to give the code a new name, and a new home: Bodhi emerged. Being hardcore into Zen recently, I feel like the name seems to fit the goals of the project nicely.
A Buddhist term for the wisdom by which one attains enlightenment. Bodhi is the opposite of ignorance, the insight into reality which destroys mental afflictions and brings peace.
Previously, the updates system code was in CVS, and bugs were to be filed in OTRS, which ended up being extremely painful to work with. After switching to our new shiny hosting setup, Bodhi is now using Mercurial for source control, and Trac for managing the project milestones, bugs, and documentation. Overall, I've been extremely impressed with Mercurial and Trac so far.
Here are some screenshots of the current development version. More screenshots can be found on the bodhi wiki.
|
|
|
|
|
|
So what's going to change for Fedora developers?
Well, currently any [extras] developer can push out updates to their packages for any release with ease (cvs commit && make tag build). If your updated package fixes 4 bugs and 2 CVE's, the end user who is blindly installing this update has no idea. Not only is there no differentiation between bugfix/enhancement/security updates, but none of them go through any sort of QA whatsoever. This is what bodhi is aiming to change.
I've been designing this system completely independent of any Buildsystem (mainly because the unveiling of brew is still churning slowly), so the process of building packages is going to remain the same (for now). Once your packages is built, you will enter it into bodhi via a web form, or [eventually] a command-line tool (Ideally, I'd like to see the process of preparing/testing/releasing a package to be a single point of interaction; either completely from command-line or in web interface). From here, your update will undergo various checks (and will eventually dispatch tests to the beaker test lab). If all goes well, your update will get signed and pushed, all referenced Bugzillas will be updated/closed, and an update notification mail will be sent to the appropriate list.
updates-testing is currently our "testing" repository that developers can choose to push their updates out to before going to 'final'. This is not very appealing for devs, as it requires them to 'Move to final' after a certain amount of time when they feel it is necessary, and the system will nag them if they don't. The updates-testing repo is not widely used, and gives testers no incentive or way to give feedback, other than filing a bug. One of the goals of bodhi is to require all updates to go through the updates-testing process, and provide a simple interface for developers/testers/users to provide positive/negative feedback regarding an update, and also make it trivial to submit bugs about them as well (we threw around some ideas about making a reusable QA feedback widget for this at a recent Fedora QA meeting). After a certain number of positive responses (or after a given length of time with no negative responses), an update will then able to make it's way to the stable updates repository.
Hackery
I've written a ton of code so far, but there is still much work to be done to accomplish all of the goals mentioned above. I layed out a few milestones for bodhi, 1.0 being the minimal functionality needed for the release of Fedora 7. I also added a 1.1 milestone for features that aren't crucial for minimal functionality, but are definitely important, and also a wishlist milestone for features that might be nice to have someday (metrics, rss feeds, etc).
With FUDCon and the Fedora Infrastructure HackFest coming up, I'm encouraging anyone interested to dive in help out in any way they want. If you can't make it out to Boston, just hop on IRC. If you would like to see a feature in bodhi, feel free to add a ticket to any of the milestones.
This system holds much potential for revolutionizing the way Fedora releases evolve, and how developers and testers interact with it. Since this is going to be utilized by all package maintainers, it's important that the system is molded to fit the needs of the developers and testers; so if you have any suggestions/improvements/fixes, don't hesitate: Contribute.
posted at: 21:12 | link | | 5 comments
