Sat, 07 May 2011

Red Hat OpenShift Express & The Leafy Miracle

Red Hat made a lot of awesome announcements this week at The Red Hat Summit, one of which being OpenShift.

I had the opportunity to play with the internal beta for a little while now, and I must say that as a developer I am extremely impressed with the service. Just being able to git push my code into to the cloud drastically simplifies large-scale software deployment, and makes it so I don't even have to leave my development environment.

I figured out a way to get TurboGears2 and Pyramid running on OpenShift Express, and documented it here and here. After that, I proceeded to write my very first Pyramid application.

[ The Leafy Miracle ]

In memory of the proposed [and rejected] Fedora 16 codename "Beefy Miracle", this little app is called "Leafy Miracle".

leafy-miracle.rhcloud.com


[ Features & Tech ]

[ Running ]
sudo yum -y install python-virtualenv
git clone git://fedorapeople.org/~lmacken/leafymiracle && cd leafymiracle
virtualenv env && source env/bin/activate
python setup.py develop
python leafymiracle/populate.py
paster serve development.ini
[ Code ]

git clone git://fedorapeople.org/~lmacken/leafymiracle

[ Props ]

Mad props go out to RJ Bean, who helped me write this app. He is responsible for writing a ton of amazing Python widgets for various JavaScript visualization libraries. You can see some demos of them here: tw2-demos.threebean.org.


posted at: 20:20 | link | Tags: , , , , , , , , , , | 0 comments

Mon, 18 Jan 2010

nose 0.11

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module.

lmacken@tomservo ~/bodhi $ nosetests
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 725.111s

OK

lmacken@tomservo ~/bodhi $ nosetests --processes=50
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 10.915s

OK

Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month


posted at: 22:58 | link | Tags: , , , , | 4 comments

Thu, 10 Dec 2009

FUDCon Toronto 2009

Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion.

My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on.

Moksha

Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later.

One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker.

I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing:

"I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is https://fedorahosted.org/moksha/ and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well http://meetbot.fedoraproject.org/fudcon-room-3/2009-12-05/fudcon-room-3.2009-12-05-22.07.log.html ... presentation slides found: http://lmacken.fedorapeople.org/moksha-FUDConToronto-2009.odp"

Fedora Community

So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform.

After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen.

During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them).

I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to come. My main concern was a blocker issue that we were hitting with our flot graphs when you quickly bounce between tabs. I ended up "fixing" the bug, so I'll be pushing what we have of the stats branch into production in the near future.

TurboGears2

TurboGears has definitely been our favorite web framework within Fedora's Infrastructure for many years now. TurboGears2, a complete re-invention of itself, has been released recently, and is catching on *very* quickly in the community. Tons of people are working on awesome new apps, and loving every minute of it. I was also able to convert a rails hacker over to it, after he was able to quickly dive into one of the tutorials with ease. See my previous blog post about getting up and running with TG2 in Fedora/EPEL.

python-fedora

One of my main tasks during the hackfests was to pull the authentication layer in Fedora Community that authenticates against the Fedora Account System, and port it over to python-fedora, so we can use it in any TurboGears2 application. I committed the initial port to python-fedora-devel, and have started working on integrating it into a default TG2 quickstart and document the process. There are still a couple of minor things I want to fix/clean up before releasing it, so expect a blog about it soon.

Bodhi

It seems like yesterday that I was an intern at Red Hat working on an internal updates system for Fedora Core. Coming up on 5 years later, and I am now working on my 3rd implementation of an updates system, Bodhi v2.0. What's wrong with the current Bodhi you ask? Well, if you talk to any user of it, you'll probably get a pretty long list. Bodhi is the first TurboGears application written & deployed in Fedora Infrastructure, and uses the vanilla components (SQLObject, kid, CherryPy2). The TG1 stack has been holding up quite nicely over the years, and is still supported upstream, but bodhi's current implemention and design does not make it easy to grow.

Bodhi v2.0 will be implemented in TurboGears2, using SQLAlchemy for an ORM, Mako for templates, and ToscaWidgets2 for re-usable widgets. It will be hook-based and plugin-driven, and will be completely distribution agnostic. Another important goal will be AMQP message-bus integration, which will allow other services or users to react to various events inside of the system as they happen.

So far I've ported the old DB model from SQLObject to SQLAlchemy, and have begun porting the old unit tests, and writing new ones. Come the new year, I'll be giving this much more of my focus.

During the hackfests I got a chance to talk to Dennis Gilmore about various improvements that we need to make with regard to the update push process. It was also great to talk to many different users of bodhi, who expressed various concerns, some of which I've already fixed. I also got a chance to talk to Xavier Lamien about deploying Bodhi for rpmfusion. On the bus ride home I helped explain to Mel how Bodhi & Koji fit into the big picture of things.

During the BarCamp sessions I also attended a session about the Update Experience, where we discussed many important issues surrounding updates.

liveusb-creator

So I got a chance to finally meet Sebastian Dziallas, of Sugar on a Stick fame, and was able to fix a few liveusb-creator issues on his laptop. I ended up pushing out a new release a couple of days ago that contains some of those fixes, along with a new version of Sugar on a Stick.

The liveusb-creator has been catching a lot of press recently (see the front page for a list). Not only did it have a 2 page spread in Linux Format, but it was also featured in this weeks Wired.com article New Sugar on a Stick Brings Much Needed Improvements. Rock.

Python

There was lot of brainstorming done by Dave Malcolm, Colin Walters, Toshio Kuratomi, Bernie Innocenti, I, and many others about various improvements that we could make to the Python interpreter. From speeding up startup time by doing some clever caching to potentially creating a new optimized compiled binary format. We also looked into how WebError/abrt gather tracebacks, and discussed ways of enabling interactive traceback debugging for vanilla processes, without requiring a layer of WSGI middleware.

There was also work done on adding SystemTap probes to Python, which is very exciting. There are many ideas for various probe points, including one that I blogged about previously.

Intel iMac8,1 support

My iMac sucks at Linux. This has been something that has been nagging me for a long time, and I've been slowly trying to chip away at the problems. First, I've been doing work on a Mac port of the liveusb-creator. I also started to work on a kernel patch for getting the EFI framebuffer working, and discussed how to do it with ajax and pjones. The screen doesn't display anything after grub, and since we don't know the base address of the framebuffer, it involves writing code to iterate over memory trying to find some common pixel patterns. I'm still trying to wrap my head around all of it, but I'll probably end up just buying them beer to fix it for me.

Thincrust

Thincrust is a project that I've been excited about for a while, and I actually have some appliances deployed in a production cloud. I was able to run some ideas for various virtual appliances by one of the authors over some beers. Some pre-baked virtual appliances that you can easily throw into a cloud that I would like to see:

dogtail

I'm glad to see that dogtail is still exciting people in the community. It still has a lot of potential to improve not only the way we test graphical software, but we also discussed ways of using it to teach people and automate various desktop tasks. What if you logged in after a fresh install and got the following popup bubble:

Hi, welcome to Fedora, what can I help you do today?

Each task would then allow Fedora to take the wheel and walk the user through various steps. I had this idea a while ago, when dogtail first came out, and I still think it would be totally awesome. Anyway, this was not a focus of the hackfests, but merely a conversation that I had while walking to lunch :)


posted at: 17:49 | link | Tags: , , , , , , , | 10 comments

Thu, 19 Nov 2009

TurboGears2 in Fedora & EPEL

I'm excited to announce that the TurboGears2 web application stack is now available in Fedora 12, 11 and EPEL-5.

What is TurboGears2?

TurboGears 2 is the built on top of the experience of several next generation web frameworks including TurboGears 1 (of course), Django, and Rails. All of these frameworks had limitations which were frustrating in various ways, and TG2 is an answer to that frustration. We wanted something that had:
  • Real multi-database support
  • Horizontal data partitioning (sharding)
  • Support for a variety of JavaScript toolkits, and new widget system to make building ajax heavy apps easier
  • Support for multiple data-exchange formats.
  • Built in extensibility via standard WSGI components

Installing the TurboGears2 stack & development tools

Fedora 12
yum install TurboGears2 python-tg-devtools
Fedora 11
yum --enablerepo=updates-testing install TurboGears2 python-tg-devtools
Red Hat Enterprise Linux 5 (with EPEL)
yum --enablerepo=epel-testing install TurboGears2 python-tg-devtools

Creating your first TG2 app

paster quickstart

Run your test suite

nosetests

Run your application

paster serve development.ini

Read the documentation

http://www.turbogears.org/2.0/docs

Contribute

If you're interested in helping maintain and improve the TG2/Pylons stack within Fedora/EPEL, please let me know. We're always looking for new Python hackers to join the team. There are still a few more components that need to be packaged and reviewed (eg: chameleon.genshi), so please take a look at the TurboGears2 page on the Fedora wiki for more details..


posted at: 06:00 | link | Tags: , , | 1 comments

Sun, 14 Dec 2008

>>> from fedora.client import Wiki

I created a simple Python API for interacting with Fedora's MediaWiki a while back, in an attempt to gather various metrics. I just went ahead and committed it to the python-fedora modules. Here is how to use it:

>>> from fedora.client import Wiki
>>> wiki = Wiki()
>>> wiki.print_recent_changes()
From 2008-12-07 20:59:01.187363 to 2008-12-14 20:59:01.187363
500 wiki changes in the past week

== Most active wiki users ==
 Bbbush............................................ 230
 Konradm........................................... 25
 Duffy............................................. 22
 Jreznik........................................... 21
 Ianweller......................................... 14
 Jjmcd............................................. 14
 Geroldka.......................................... 10
 Gdk............................................... 9
 Anouar............................................ 7
 Gomix............................................. 6

== Most edited pages ==
 Features/KDE42.................................... 21
 SIGs/SciTech/SAGE................................. 15
 FUDCon/FUDConF11.................................. 14
 Special:Log/upload................................ 13
 How to be a release notes beat writer............. 12
 Special:Log/move.................................. 11
 Design/SETroubleshootUsabilityImprovements........ 10
 PackageMaintainers/FEver.......................... 9
 User:Gomix........................................ 6
 Zh/主要配置文件..................................... 5

>>> for event in wiki.send_request('api.php', req_params={
...         'action': 'query',
...         'list': 'logevents',
...         'format': 'json',
...         })['query']['logevents']:
...     print '%-10s %-15s %s' % (event['action'], event['user'], event['title'])
...
patrol     Ianweller       User:Ianweller/How to create a contributor business card
move       Nippur          REvanderLuit
patrol     Ianweller       Project Leader
move       Ianweller       FPL
upload     Anouar          Image:AnouarAbtoy.JPG
move       Liangsuilong    ZH/Docs/FetionOnFedora
move       Liangsuilong    FetionOnFedora
patrol     Ianweller       User:Ianweller

It uses the fedora.client.BaseClient, which is a class that simplifies interacting with arbitrary web services. Toshio and I created it a while back as a the core client for talking with our various TurboGears-based Fedora Services (bodhi, pkgdb, fas, etc.), but it has now seemed to morph into a much more flexible client for talking JSON with web applications.

from datetime import datetime, timedelta
from collections import defaultdict
from fedora.client import BaseClient

class Wiki(BaseClient):

    def __init__(self, base_url='http://fedoraproject.org/w/', *args, **kwargs):
        super(Wiki, self).__init__(base_url, *args, **kwargs)

    def get_recent_changes(self, now, then, limit=500):
        """ Get recent wiki changes from `now` until `then` """
        data = self.send_request('api.php', req_params={
                'list'    : 'recentchanges',
                'action'  : 'query',
                'format'  : 'json',
                'rcprop'  : 'user|title',
                'rcend'   : then.isoformat().split('.')[0] + 'Z',
                'rclimit' : limit,
                })
        if 'error' in data:
            raise Exception(data['error']['info'])
        return data['query']['recentchanges']

    def print_recent_changes(self, days=7, show=10):
        now = datetime.utcnow()
        then = now - timedelta(days=days)
        print "From %s to %s" % (then, now)
        changes = self.get_recent_changes(now=now, then=then)
        num_changes = len(changes)
        print "%d wiki changes in the past week" % num_changes

        users = defaultdict(list) # {username: [change,]}
        pages = defaultdict(int)  # {pagename: # of edits}

        for change in changes:
            users[change['user']].append(change['title'])
            pages[change['title']] += 1

        print '\n== Most active wiki users =='
        for user, changes in sorted(users.items(),
                                    cmp=lambda x, y: cmp(len(x[1]), len(y[1])),
                                    reverse=True)[:show]:
            print ' %-50s %d' % (('%s' % user).ljust(50, '.'), len(changes))

        print '\n== Most edited pages =='
        for page, num in sorted(pages.items(),
                                cmp=lambda x, y: cmp(x[1], y[1]),
                                reverse=True)[:show]:
            print ' %-50s %d' % (('%s' % page).ljust(50, '.'), num)

I added a Wiki.login method to the latest version, but it isn't quite working yet. This is due to some minor limitations in the ProxyClient, so we currently cannot handle authenticated requests. However, this shouldn't be very difficult to implement. The reason for this is that we need to be able to run authenticated queries as a 'bot' account in order to mitigate the 500 entry API return limit.

This module makes it easy to talk to MediaWiki's API, so if you do anything cool with it feel free to send patches here. It's currently not being shipped in a python-fedora release, so you'll have to grab the code from Bazaar:

bzr branch bzr://bzr.fedorahosted.org/bzr/python-fedora/python-fedora-devel


posted at: 23:12 | link | Tags: , , , | 13 comments

Tue, 28 Oct 2008

TurboGears2 on Fedora

So, I was finally able to force myself to take a break from hacking on a few different TG2 applications to actually sit down and package everything up, submit it all for review, and make sure everything Just Works on Fedora.

You can start playing with TurboGears2 by throwing my tg2.repo file in /etc/yum.repos.d, and then by installing the TurboGears2 package. It should work fine alongside of any existing TurboGears 1.x installs, however it will pull in SQLAlchemy 0.5, which may or may not cause problems with older models. Once installed, checkout the fantastic TurboGears2 documentation for details on diving into the framework.

I already have two TurboGears2 applications currently in production, and have had nothing but amazing experiences with the new stack. I still love working on my older TG1.0 apps (such as bodhi), which will still be supported for a long time, but with the ridiculous amounts of power that WSGI/Pylons/Paste/SQLAlchemy/ToscaWidgets/Mako gives you, I'm extremely tempted to start porting :)

Aside from testing the packages, you can also help by reviewing various parts of the stack so we can get them into Fedora as soon as possible. The TG2 dependency tree of pending reviews can be found here.


posted at: 06:18 | link | Tags: , , | 1 comments

Wed, 16 Jan 2008

FUDCon

So FUDCon happened over the weekend in Raleigh, North Carolina. It was a great chance to meet a bunch of new people, catch up with some old friends, and kick around in the south for a bit. I was amazed to see such a huge turnout for the first day of the hackfest. It was nice to see a ton of new contributors looking to dive in head first into projects. My goal for the day was to hack on the MyFedora framework, and solidify our architecture and base widget classes, making it easy to create and display your own widgets. It's probably safe to say that we exceeded those expectations.

I sat down with J5, Toshio, and Douglas Warner, fired up a Gobby instance, and started hacking. Thanks to the wonders of distributed source control (git!), TurboGears, and Gobby, we were all able to simultaneously run, commit, and hack on the code. The result of our days work turns out to be a pretty solid architecture for writing, configuring, and displaying reusable Python widgets (based on ToscaWidgets) that can pull from various data sources. For example, writing a widget to display the latest entries in an RSS feed couldn't really be much easier:

class FedoraPeopleWidget(RSSWidget):
    url = 'http://planet.fedoraproject.org/rss20.xml'
    title = 'Fedora People'

The next day during the MyFedora session we got a chance to show off some of the work we did, and get some more ideas from various types of contributors. This project has the potential to make a lot of peoples lives easier, so if you're interested in helping out, grab the code and dive in: $ git clone git://git.fedorahosted.org/git/myfedora.git

Toshio and I gave a session on TurboGears, which seemed to go pretty well. Lots of good discussion and code examples. You can checkout the slides for my presentation here: http://tg.lewk.org.

I was going to be giving a session on bodhi, which we eventually merged with the TurboGears talk. However, the TG session went a lot longer than expected, and bodhi never emerged. So, for those who were interested, you can find my bodhi slides here, and some transcripts from our last virtual fudcon.

The PackageKit session went well too. People definitely were interested, and also had some interesting ideas.

Saturday night was FUDPub, where we had the back room of the Flying Saucer all to ourselves. People kept feeding me drinks, and I didn't complain. Good times :)

Sunday was the second day of the hackfests. I decided to context-switch a bit and get my func on. I wrote a patch that adds a "mem" method to the ProcessModule that returns per-program memory usage from your minion in the format of [[Private, Shared, Total RAM used, Program], ...]. This allows you to do something like,

[lmacken@crow ~]$ sudo func "*" call process mem
on https://tomservo:51234 running process mem ()
[['16.8 MiB', '6.5 MiB', '23.4 MiB', 'Xorg'],
 ['21.7 MiB', '8.3 MiB', '30.1 MiB', 'tomboy'],
 ['33.6 MiB', '2.3 MiB', '35.9 MiB', 'ssh (5)'],
 ['23.2 MiB', '14.3 MiB', '37.5 MiB', 'deskbar-applet'],
 ['139.9 MiB', '9.9 MiB', '149.8 MiB', 'firefox-bin']]

I also discussed a potential TurboGears FuncWeb implementation with Michael DeHaan. I got a chance to create create a skeleton project, and jot some ideas down. Just as I was about to dive in, I got a phone call notifying me of my flight cancellation. I then had to immediately sketch off to catch a 2:20pm flight and head back to Boston.

Last night I got a little bit A.D.D. and re-wrote some chunks of the func minion module_loader/server to make writing func modules a lot easier.

So, the moral of the story is: FUDCon rocks. Feeding large quantities of geeks caffeine, beer, and barbeque can result in amazing things.

Of course there are no ups without downs, so I was stuck dealing with a nasty cold most of the time there, and my laptop power adapter melted as well. Thankfully, both of those issue have since been resolved :)


posted at: 17:21 | link | Tags: , , , | 0 comments

Wed, 19 Dec 2007

TurboFlot 0.0.1

In an effort to clean up bodhi's metrics code a bit, I wrote a TurboFlot plugin that allows you to wield the jQuery plugin flot inside of TurboGears applications. The code is quite trivial -- it's essentially just a TurboGears JSON proxy to the jQuery flot plugin. Breaking this code out into it's own widget makes it really easy to generate shiny graphs in a Pythonic fashon, without having to write a line of javascript.

Check out the README to see the code for the example above.

To use TurboFlot in your own application, you just pass your data and graph options to the widget, and then throw it up to your template. Read the flot API documentation for details on all of the arguments. Here is a simple usage example:

flot = TurboFlot([
    {
        'data'  : [[0, 3], [4, 8], [8, 5], [9, 13]],
        'lines' : { 'show' : True, 'fill' : True }
    }],
    {
        'grid'  : { 'backgroundColor' : '#fffaff' },
        'yaxis' : { 'max' : '850' }
    }
)
Then, to display the widget in your template, you simply use:
${flot.display()}

The code for the widget itself is pretty simple. It just takes your data and graph options, encodes them as JSON and tosses them at flot.
class TurboFlot(Widget):
    """
        A TurboGears Flot Widget.
    """
    template = """
      <div xmlns:py="http://purl.org/kid/ns#" id="turboflot"
           style="width:${width};height:${height};">
        <script>
          $.plot($("#turboflot"), ${data}, ${options});
        </script>
      </div>
    """
    params = ["data", "options", "height", "width"]
    javascript = [JSLink('turboflot', 'excanvas.js'),
                  JSLink("turboflot", "jquery.js"),
                  JSLink("turboflot", "jquery.flot.js")]

    def __init__(self, data, options={}, height="300px", width="600px"):
        self.data = simplejson.dumps(data)
        self.options = simplejson.dumps(options)
        self.height = height
        self.width = width

You can download the latest releases from the Python Package Index:

http://pypi.python.org/pypi/TurboFlot
Or you can grab my latest development tree out of mercurial:
http://hg.lewk.org/TurboFlot
As always, patches are welcome :)


posted at: 20:21 | link | Tags: , , , | 5 comments

Sun, 09 Dec 2007

Fedora update metrics

Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.

I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.

When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.


posted at: 01:05 | link | Tags: , , , , , | 2 comments

Mon, 01 Oct 2007

Use your Nose!

Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.

So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.

For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update.

import urllib, cherrypy
from turbogears import update_config, database, testutil, url

update_config(configfile='dev.cfg', modulename='bodhi.config')
database.set_db_uri("sqlite:///:memory:")

class TestControllers(testutil.DBTest):

    def test_unauthenticated_update(self):
        params = {
                'builds'  : 'TurboGears-1.0.2.2-2.fc7',
                'release' : 'Fedora 7',
                'type'    : 'enhancement',
                'bugs'    : '1234 5678',
                'cves'    : 'CVE-2020-0001',
                'notes'   : 'foobar'
        }
        path = url('/save?' + urllib.urlencode(params))
        testutil.createRequest(path, method='POST')
        assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'.
[lmacken@tomservo bodhi]$ nosetests
.................................
----------------------------------------------------------------------
Ran 33 tests in 16.798s

OK
Now, for the fun part. Nose comes equipped with a profiling plugin that will profile your test cases using Python's hotshot module. So, I went ahead and added a 'profile' target to bodhi's Makefile:
profile:
    nosetests --with-profile --profile-stats-file=nose.prof
    python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
Now, typing 'make profile' will execute and profile all of our unit tests, and spit out the top 20 method calls -- ordered by internal time and call count.
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile --profile-stats-file=nose.prof
.................................
----------------------------------------------------------------------
Ran 33 tests in 42.878s

OK
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
         800986 function calls (702850 primitive calls) in 42.878 CPU seconds

   Ordered by: internal time, call count
   List reduced from 3815 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       14   13.675    0.977   13.675    0.977 /usr/lib/python2.5/socket.py:71(ssl)
       31   10.683    0.345   10.683    0.345 /usr/lib/python2.5/httplib.py:994(_read)
2478/2429    9.297    0.004    9.677    0.004 :1()
        1    0.604    0.604    0.604    0.604 /usr/lib/python2.5/commands.py:50(getstatusoutput)
     2999    0.536    0.000    0.539    0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/sqliteconnection.py:177(_executeRetry)
   105899    0.448    0.000    0.773    0.000 Modules/pyexpat.c:871(Default)
       60    0.327    0.005    1.102    0.018 /usr/lib/python2.5/site-packages/kid/parser.py:343(_buildForeign)
   105899    0.325    0.000    0.325    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:452(_default)
     3396    0.280    0.000    0.420    0.000 /usr/lib/python2.5/site-packages/cherrypy/config.py:107(get)
     2965    0.263    0.000    0.263    0.000 /usr/lib/python2.5/logging/__init__.py:364(formatTime)
44964/6587    0.238    0.000    0.252    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:156(_pull)
       60    0.116    0.002    0.116    0.002 /usr/lib/python2.5/site-packages/kid/compiler.py:38(py_compile)
     8127    0.114    0.000    0.114    0.000 /usr/lib/python2.5/site-packages/cherrypy/_cputil.py:311(lower_to_camel)
     8982    0.110    0.000    0.137    0.000 /usr/lib/python2.5/site-packages/sqlobject/dbconnection.py:902(__getattr__)
13740/4044    0.108    0.000    2.176    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:209(_coalesce)
24353/4026    0.107    0.000    2.143    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:174(_track)
     3170    0.093    0.000    0.398    0.000 /usr/lib/python2.5/logging/__init__.py:405(format)
        1    0.082    0.082    0.082    0.082 /usr/lib/python2.5/site-packages/rpm/__init__.py:5()
     4777    0.081    0.000    1.320    0.000 /usr/lib/python2.5/site-packages/kid/serialization.py:564(generate)
  759/176    0.074    0.000    0.210    0.001 /usr/lib/python2.5/sre_parse.py:385(_parse)


posted at: 14:40 | link | Tags: , , , , | 8 comments

Thu, 15 Feb 2007

break

So my Thanksgiving break was far from a break. I spent a couple of days last week at Red Hat's westford office before heading back up to RIT to start a new quarter. In my two days in the office I was able to touch base with a bunch of people, and get a bunch of stuff done as well. I had a long discussion with dmalcom about integrating the Fedora Updates System with Beaker/TableCloth. He also gave me a quick rundown on a bunch of the Red Hat QA infrastructure that is currently being used. Ideally we'd like to be able to crunch all package updates through an automated test system before pushing them out to the world. Involvement needed: FedoraTesting.

Later that day I met with jrb and jkeating about getting a package updating system in place for a new Red Hat product that is going out the door very soon. This means that much work will be going into the new UpdatesSystem in the near future, which means I get to dig deeper into the world of TurboGears :)

On thursday I cranked a bunch of code out, but was fairly distracted most of the time by the OLPC laptops that were lying around the office. I must say, it is an absolutely incredible machine. The screen is gorgeous, and it's camera is very impressive. I hung around later at the office for an OLPC hackfest that was going down.

I was busy working on the updates system most of the time, but then later on I started looking into some Python start-up issues, which can be seen by doing:

	strace python 2>&1 | grep ENOENT
You'll notice a ton of syscalls like the following, which try to open/stat modules in locations that do not exist:
stat64("/usr/lib/python24.zip/posixpath", 0xbfdb5094) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpathmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python24.zip/posixpath.pyc", O_RDONLY|O_LARGEFILE) = -1 ENOENT (N o such file or directory)
stat64("/usr/lib/python2.4/posixpath", 0xbfdb5094) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpath.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No su ch file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpathmodule.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
PrivoxyWindowOpen("/usr/lib/python2.4/posixpath.py", O_RDONLY|O_LARGEFILE) = 5 

So it's obvious that modules could exist in multiple locations, but if you are repeatedly going to check a series of directories, such as /usr/lib/python24.zip, wouldn't it be a *bit* smarter to check if they exists first, and then avoid checking there in the future? Doing so would help cut down from the 233+ syscalls python makes while starting up looking for modules. I really don't have any free cycles to try and add some sense into Python, so I really hope someone can beat me to a patch.


TurboGears 1.0b2


I came back home to find the new TurboGears book in my mailbox, which has been extremely informative, aside from the fact that the project has awesome online docs as well. I pushed out the latest TurboGears release, 1.0b2, for FC6 and rawhide yesterday as well.


posted at: 03:12 | link | Tags: , , , , | 1 comments