Wed, 16 Jan 2008
FUDCon
So FUDCon happened over the weekend in Raleigh, North Carolina. It was a great chance to meet a bunch of new people, catch up with some old friends, and kick around in the south for a bit. I was amazed to see such a huge turnout for the first day of the hackfest. It was nice to see a ton of new contributors looking to dive in head first into projects. My goal for the day was to hack on the MyFedora framework, and solidify our architecture and base widget classes, making it easy to create and display your own widgets. It's probably safe to say that we exceeded those expectations.
I sat down with J5, Toshio, and Douglas Warner, fired up a Gobby instance, and started hacking. Thanks to the wonders of distributed source control (git!), TurboGears, and Gobby, we were all able to simultaneously run, commit, and hack on the code. The result of our days work turns out to be a pretty solid architecture for writing, configuring, and displaying reusable Python widgets (based on ToscaWidgets) that can pull from various data sources. For example, writing a widget to display the latest entries in an RSS feed couldn't really be much easier:
class FedoraPeopleWidget(RSSWidget):
url = 'http://planet.fedoraproject.org/rss20.xml'
title = 'Fedora People'
The next day during the MyFedora session we got a chance to show off some of the work we did, and get some more ideas from various types of contributors.
This project has the potential to make a lot of peoples lives easier, so if you're interested in helping out, grab the code and dive in:
$ git clone git://git.fedorahosted.org/git/myfedora.git
Toshio and I gave a session on TurboGears, which seemed to go pretty well. Lots of good discussion and code examples. You can checkout the slides for my presentation here: http://tg.lewk.org.
I was going to be giving a session on bodhi, which we eventually merged with the TurboGears talk. However, the TG session went a lot longer than expected, and bodhi never emerged. So, for those who were interested, you can find my bodhi slides here, and some transcripts from our last virtual fudcon.
The PackageKit session went well too. People definitely were interested, and also had some interesting ideas.
Saturday night was FUDPub, where we had the back room of the Flying Saucer all to ourselves. People kept feeding me drinks, and I didn't complain. Good times :)
Sunday was the second day of the hackfests. I decided to context-switch a bit and get my func on. I wrote a patch that adds a "mem" method to the ProcessModule that returns per-program memory usage from your minion in the format of [[Private, Shared, Total RAM used, Program], ...]. This allows you to do something like,
[lmacken@crow ~]$ sudo func "*" call process mem
on https://tomservo:51234 running process mem ()
[['16.8 MiB', '6.5 MiB', '23.4 MiB', 'Xorg'],
['21.7 MiB', '8.3 MiB', '30.1 MiB', 'tomboy'],
['33.6 MiB', '2.3 MiB', '35.9 MiB', 'ssh (5)'],
['23.2 MiB', '14.3 MiB', '37.5 MiB', 'deskbar-applet'],
['139.9 MiB', '9.9 MiB', '149.8 MiB', 'firefox-bin']]
I also discussed a potential TurboGears FuncWeb implementation with Michael DeHaan. I got a chance to create create a skeleton project, and jot some ideas down. Just as I was about to dive in, I got a phone call notifying me of my flight cancellation. I then had to immediately sketch off to catch a 2:20pm flight and head back to Boston.
Last night I got a little bit A.D.D. and re-wrote some chunks of the func minion module_loader/server to make writing func modules a lot easier.
So, the moral of the story is: FUDCon rocks. Feeding large quantities of geeks caffeine, beer, and barbeque can result in amazing things.
Of course there are no ups without downs, so I was stuck dealing with a nasty cold most of the time there, and my laptop power adapter melted as well. Thankfully, both of those issue have since been resolved :)
posted at: 11:21 | link | | 0 comments
Wed, 19 Dec 2007
TurboFlot 0.0.1
In an effort to clean up bodhi's metrics code a bit, I wrote a TurboFlot plugin that allows you to wield the jQuery plugin flot inside of TurboGears applications. The code is quite trivial -- it's essentially just a TurboGears JSON proxy to the jQuery flot plugin. Breaking this code out into it's own widget makes it really easy to generate shiny graphs in a Pythonic fashon, without having to write a line of javascript.
Check out the README to see the code for the example above.
To use TurboFlot in your own application, you just pass your data and graph options to the widget, and then throw it up to your template. Read the flot API documentation for details on all of the arguments. Here is a simple usage example:
flot = TurboFlot([
{
'data' : [[0, 3], [4, 8], [8, 5], [9, 13]],
'lines' : { 'show' : True, 'fill' : True }
}],
{
'grid' : { 'backgroundColor' : '#fffaff' },
'yaxis' : { 'max' : '850' }
}
)
Then, to display the widget in your template, you simply use:
${flot.display()}
The code for the widget itself is pretty simple. It just takes your data and graph options, encodes them as JSON and tosses them at flot.
class TurboFlot(Widget):
"""
A TurboGears Flot Widget.
"""
template = """
<div xmlns:py="http://purl.org/kid/ns#" id="turboflot"
style="width:${width};height:${height};">
<script>
$.plot($("#turboflot"), ${data}, ${options});
</script>
</div>
"""
params = ["data", "options", "height", "width"]
javascript = [JSLink('turboflot', 'excanvas.js'),
JSLink("turboflot", "jquery.js"),
JSLink("turboflot", "jquery.flot.js")]
def __init__(self, data, options={}, height="300px", width="600px"):
self.data = simplejson.dumps(data)
self.options = simplejson.dumps(options)
self.height = height
self.width = width
You can download the latest releases from the Python Package Index:
http://pypi.python.org/pypi/TurboFlotOr you can grab my latest development tree out of mercurial:
http://hg.lewk.org/TurboFlotAs always, patches are welcome :)
posted at: 14:21 | link | | 1 comments
Sat, 08 Dec 2007
Fedora update metrics
Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.
I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.
When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.
posted at: 19:05 | link | | 0 comments
Mon, 01 Oct 2007
Use your Nose!
Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.
So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.
For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update.
import urllib, cherrypy
from turbogears import update_config, database, testutil, url
update_config(configfile='dev.cfg', modulename='bodhi.config')
database.set_db_uri("sqlite:///:memory:")
class TestControllers(testutil.DBTest):
def test_unauthenticated_update(self):
params = {
'builds' : 'TurboGears-1.0.2.2-2.fc7',
'release' : 'Fedora 7',
'type' : 'enhancement',
'bugs' : '1234 5678',
'cves' : 'CVE-2020-0001',
'notes' : 'foobar'
}
path = url('/save?' + urllib.urlencode(params))
testutil.createRequest(path, method='POST')
assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'.
[lmacken@tomservo bodhi]$ nosetests
.................................
----------------------------------------------------------------------
Ran 33 tests in 16.798s
OK
profile:
nosetests --with-profile --profile-stats-file=nose.prof
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile --profile-stats-file=nose.prof
.................................
----------------------------------------------------------------------
Ran 33 tests in 42.878s
OK
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
800986 function calls (702850 primitive calls) in 42.878 CPU seconds
Ordered by: internal time, call count
List reduced from 3815 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
14 13.675 0.977 13.675 0.977 /usr/lib/python2.5/socket.py:71(ssl)
31 10.683 0.345 10.683 0.345 /usr/lib/python2.5/httplib.py:994(_read)
2478/2429 9.297 0.004 9.677 0.004 :1()
1 0.604 0.604 0.604 0.604 /usr/lib/python2.5/commands.py:50(getstatusoutput)
2999 0.536 0.000 0.539 0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/sqliteconnection.py:177(_executeRetry)
105899 0.448 0.000 0.773 0.000 Modules/pyexpat.c:871(Default)
60 0.327 0.005 1.102 0.018 /usr/lib/python2.5/site-packages/kid/parser.py:343(_buildForeign)
105899 0.325 0.000 0.325 0.000 /usr/lib/python2.5/site-packages/kid/parser.py:452(_default)
3396 0.280 0.000 0.420 0.000 /usr/lib/python2.5/site-packages/cherrypy/config.py:107(get)
2965 0.263 0.000 0.263 0.000 /usr/lib/python2.5/logging/__init__.py:364(formatTime)
44964/6587 0.238 0.000 0.252 0.000 /usr/lib/python2.5/site-packages/kid/parser.py:156(_pull)
60 0.116 0.002 0.116 0.002 /usr/lib/python2.5/site-packages/kid/compiler.py:38(py_compile)
8127 0.114 0.000 0.114 0.000 /usr/lib/python2.5/site-packages/cherrypy/_cputil.py:311(lower_to_camel)
8982 0.110 0.000 0.137 0.000 /usr/lib/python2.5/site-packages/sqlobject/dbconnection.py:902(__getattr__)
13740/4044 0.108 0.000 2.176 0.001 /usr/lib/python2.5/site-packages/kid/parser.py:209(_coalesce)
24353/4026 0.107 0.000 2.143 0.001 /usr/lib/python2.5/site-packages/kid/parser.py:174(_track)
3170 0.093 0.000 0.398 0.000 /usr/lib/python2.5/logging/__init__.py:405(format)
1 0.082 0.082 0.082 0.082 /usr/lib/python2.5/site-packages/rpm/__init__.py:5()
4777 0.081 0.000 1.320 0.000 /usr/lib/python2.5/site-packages/kid/serialization.py:564(generate)
759/176 0.074 0.000 0.210 0.001 /usr/lib/python2.5/sre_parse.py:385(_parse)
posted at: 09:40 | link | | 0 comments
Wed, 14 Feb 2007
break
So my Thanksgiving break was far from a break. I spent a couple of days last week at Red Hat's westford office before heading back up to RIT to start a new quarter. In my two days in the office I was able to touch base with a bunch of people, and get a bunch of stuff done as well. I had a long discussion with dmalcom about integrating the Fedora Updates System with Beaker/TableCloth. He also gave me a quick rundown on a bunch of the Red Hat QA infrastructure that is currently being used. Ideally we'd like to be able to crunch all package updates through an automated test system before pushing them out to the world. Involvement needed: FedoraTesting.
Later that day I met with jrb and jkeating about getting a package updating system in place for a new Red Hat product that is going out the door very soon. This means that much work will be going into the new UpdatesSystem in the near future, which means I get to dig deeper into the world of TurboGears :)
On thursday I cranked a bunch of code out, but was fairly distracted most of the time by the OLPC laptops that were lying around the office. I must say, it is an absolutely incredible machine. The screen is gorgeous, and it's camera is very impressive. I hung around later at the office for an OLPC hackfest that was going down.
|
|
I was busy working on the updates system most of the time, but then later on I started looking into some Python start-up issues, which can be seen by doing:
strace python 2>&1 | grep ENOENT
stat64("/usr/lib/python24.zip/posixpath", 0xbfdb5094) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpathmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (N o such file or directory)
stat64("/usr/lib/python2.4/posixpath", 0xbfdb5094) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpath.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No su ch file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpathmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpath.py", O_RDONLY|O_LARGEFILE) = 5
So it's obvious that modules could exist in multiple locations, but if you are repeatedly going to check a series of directories, such as /usr/lib/python24.zip, wouldn't it be a *bit* smarter to check if they exists first, and then avoid checking there in the future? Doing so would help cut down from the 233+ syscalls python makes while starting up looking for modules. I really don't have any free cycles to try and add some sense into Python, so I really hope someone can beat me to a patch.
TurboGears 1.0b2
I came back home to find the new TurboGears book in my mailbox, which has been extremely informative, aside from the fact that the project has awesome online docs as well. I pushed out the latest TurboGears release, 1.0b2, for FC6 and rawhide yesterday as well.
posted at: 21:12 | link | | 0 comments
