Mon, 16 Jun 2014

Bodhi2 Fedora Activity Day

The Bodhi2/Taskotron Fedora Activity Day happened earlier this month! A bunch of us gathered in Denver for a few days and worked on some of our critical releng & qa infrastructure. The hackfest was held in a conference room in my apartment building, which worked out quite nicely for the amount of people that we had. The hotel was right up the road, and we were able to walk to a lot of awesome spots, like the 1UP Barcade :).

It was great to have folks from various corners of Fedora together in the same room for a few days. As it is, we get a lot done remotely, but being able to leverage the high-bandwidth face-to-face time is extremely valuable, especially when coming to consensus on difficult architectural decisions. We used Gobby to collaborate on a long list of action items, and chipped away most of it. Thankfully, Bodhi has enough layers that we were all able to split up and dive into different corners of the code without stepping on each others' toes.

Up until now, our releng stack in staging has always been less than useful. We've never able to do a full build->update->push->run, and have had to rely on testing certain codepaths in production. Not only that, but Bodhi's "masher" never really had proper unit tests, so pushing out major changes to that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our staging setup and made it so we can actually use it. I made a lot of headway on porting the Bodhi masher to a new fedmsg-driven architecture, while writing unit tests for it along the way. I'm hopeful that we can write tests for every part of the "push" process, and then optimize the hell out of it :)

While Mathieu and I mainly focused on back-end hacking, Ralph and Ricky made some fantastic headway on the front-end. Ralph started working on the revamped New Update Form, which is already significantly better than the original. The idea here is that the maintainer should only need to provide the package name(s), and Bodhi will automatically find the corresponding candidate builds, related bugs, and eventually it will pull in all candidate deps as well (or tell if you if any need to be rebuilt). It would also be very convenient to be able to "push this entire stack again". Ideally, I'd love to make it so that folks maintaining large stacks, like GNOME, shouldn't need to use a Google doc to coordinate their mega-updates ;)

Ralph also started revamping the new karma system (check out his screencast here). We don't have any of the policy in place to enforce it yet, but eventually we'd like the maintainers to be able to define custom policy constraints for their updates. For example, you could say "only allow this update to go to the stable repo once this specific bug or test case has been confirmed as fixed/passing".

Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on the front page using the fedmsg datagrepper widget that Ralph blogged about recently. Other front-end improvements include libravatar support for all users, proper markdown rendering with previews and image support, and of course a konami code easter-egg ;)

I was going to post a bunch of screenshots here, but Ralph just deployed a development instance of Bodhi2 that folks can play around with instead: http://bodhi.dev.fedoraproject.org (it's a dev instance, so don't expect it to always be up/functional).

Some other corners of Bodhi that have seen improvements recently:

The API. The Bodhi webapp is written in Python using the excellent Pyramid web framework. On top of that we are using Cornice, which makes it really easy to build & document standards-compliant web services with Pyramid. Thanks to colander validation/deserialization schemas and our custom ACLs and validators, we are able to write dead-simple views that can safely assume that all of the data we are dealing with is valid and in the right format, and that the user has the appropriate permissions. Cornice also has a Sphinx plugin that auto-generates our API documentation. So not only do we now have a semi-RESTful standards-compliant self-documenting API, but Ralph also added a support for rendering almost every service as an RSS feed as well.

Regarding the Bodhi Python APIs, I've begun porting them to the new python-fedora OpenIDBaseClient (see bodhi/client.py). Since a large percentage of the API usage is through the current python-fedora BodhiClient, I'd like to try our best to maintain compatibility--at least for a version or two with deprecation warnings if we have to. I am really looking forward to finally being able to trash our old TurboGears1 FAS visit/identity layer in favor of FedOAuth.

On top of the Python API lies the bodhi-client. I recently ported the basic functionality over using the click module, which makes it really easy to write complex command-line tools (see bodhi/cli.py). Since the current bodhi-client is an absolute mess, this is one area that I'm actually okay with breaking backwards-compatibility to a certain extent. Having a proper command structure, similar to the Koji cli, is well worth some cli flag changes in my opinion.

In a similar fashion, Mathieu implemented a great release management tool for Bodhi admins. Currently, creating a release and changing it's pre-release status involves using the TurboGears Python shell, creating SQLObject instances by hand, editing config files, etc. This tool will make it dead simple for releng to create new releases and manage all of the pre-GA state changes that happen along the way.

Performance was another key area of development. The app is fairly snappy now, but there is still a ton of room for improvement. The pyramid-debugtoolbar has been amazingly useful for us so far. It let's us analyze every SQL statement made, it does full-stack profiling, and it lets us execute commands in every layer of a traceback. Along with that, Ralph added a SQLAlchemy event hook to our unit tests to ensure that certain changes don't drastically change how many SQL statements are getting executed. With regard to Masher performance, there is still a lot of low-hanging fruit there. We saw a drastic boost in mash speed recently when Kevin discovered that a couple of releng machines weren't using virtio. This brought the mash time of EPEL5 updates from 45 minutes down to around 15. In the new version of the masher, updates are grouped by tag and then processed in a separate threads. Security updates and critical bugfixes will take priority over enhancements, and there are also things that we can do to make the bits hit the mirrors faster once we're done mashing.

Another corner of Bodhi that was the topic of discussion was around notifications. Bodhi currently sends way too much email, most of which I'm assuming gets ignored. There are a couple of mails that are a bit more important, like the update announcement emails that get sent to package-announce, and the updates-testing digest that goes to the test-list. The consensus that we came to was that we are going to attempt to use FMN to allow people to configure what messages they want to receive, and where (IRC, email, etc). This alleviates the need to build an email layer into Bodhi2, and allows us to focus on publishing fedmsgs only, letting FMN do the rest.

As far as the transition to Bodhi2 goes, we're going to try our best to not break the world in the process. Ralph mentioned the potential timeline in his blog post, and we still have lots of work to do before then. In order to help ease this transition, I created a wiki page to track the consumers of Bodhi's APIs, so we can make the appropriate changes to those codebases before launch. Please feel free to update it with any that I left out.

Overall, it was a very successful FAD. We got a ton of stuff done, ate a bunch of great food, and had a lot of fun in the process. I didn't cover everything that we worked on, so checkout the blog posts from threebean, bochecha, and nirik for more details on other things that got done. If you're interested in getting involved with Bodhi2, grab the code, checkout the open issues, and hop in #fedora-apps on Freenode.

Also, I'll be giving a presentation at Flock in Prague this year on "Evolving the Fedora Updates process", which will cover the history of pushing updates as well as an in-depth dive into the new bodhi2 stack.


posted at: 16:49 | link | Tags: , , | 0 comments

Thu, 07 Mar 2013

Keeping your finger on the pulse of the Fedora community

For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout fedmsg.com. Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close.

This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop!

Introducing fedmsg-notify

fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME.

For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e fedmsg@lmacken-redhat.com` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus:

For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer.

The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package.

The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB.

Under the hood

The fedmsg-notify-daemon itself is fairly minimal (see daemon.py). At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue.

class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer):
    topic = 'org.fedoraproject.*'

    def consume(self, msg): 

The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop.

The API for writing custom filters is dead simple (see filters.py). Here is an example of one:

class MyPackageFilter(Filter):
    """ Matches messages regarding packages that a given user has ACLs on """
    __description__ = 'Packages that these users maintain'
    __user_entry__ = 'Usernames'

    def __init__(self, settings):
        self.usernames = settings.replace(',', ' ').split()
        self.packages = set()
        reactor.callInThread(self._query_pkgdb)

    def _query_pkgdb(self):
        for username in self.usernames:
            log.info("Querying the PackageDB for %s's packages" % username)
            for pkg in PackageDB().user_packages(username)['pkgs']:
                self.packages.add(pkg['name'])

    def match(self, msg, processor):
        packages = processor.packages(msg)
        for package in self.packages:
            if package in packages:
                return True
The fedmsg-notify-config interface (see gui.py), automatically introspects the filters and populates the Advanced tab with the appropriate labels, switches, and text entries.

Consuming fedmsg over DBus

Let's say you want to write an application that listens to fedmsg, but you don't want to deal with spinning up your own connection, or you're not using Python, etc. For these cases, fedmsg-notify supports relaying messages over DBus. This functionality can be enabled by running `gsettings set org.fedoraproject.fedmsg.notify emit-dbus-signals true`. You can then easily listen for the MessageReceived DBus signal, like so:
import json, dbus

from gi.repository import GObject
from dbus.mainloop.glib import DBusGMainLoop

def consume(topic, body):
    print(topic)
    print(json.loads(body))

DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus()
bus.add_signal_receiver(consume, signal_name='MessageReceived',
                        dbus_interface='org.fedoraproject.fedmsg.notify',
                        path='/org/fedoraproject/fedmsg/notify')
loop = GObject.MainLoop()
loop.run()

Contributing

If you're interested in helping out with any layer of the fedmsg stack, hop in #fedora-apps, and fork it on GitHub:

Hop on the bus!


posted at: 17:30 | link | Tags: , , , | 4 comments

Sat, 21 Apr 2012

Wielding the ANU Quantum Random Number Generator

Last week Science Daily published an article that caught my attention titled 'Sounds of Silence' Proving a Hit: World's Fastest Random Number Generator. The tl;dr is that researchers at the ANU ARC Centre of Excellence for Quantum Computation and Communication Technology created a blazing fast random number generator based on quantum fluctuations in a vacuum. Thankfully, these awesome scientists are giving their data away for free, and they even provide a JSON API.

In an effort to make it simple to leverage this data, I created a new project: quantumrandom. It provides a qrandom command-line tool, a Python API, and also a /dev/qrandom Linux character device.

Installing

$ virtualenv env
$ source env/bin/activate
$ pip install quantumrandom

Using the command-line tool

$ qrandom --int --min 5 --max 15
7
$ qrandom --binary
���I�%��e(�1��c��Ee�4�������j�Կ��=�^H�c�u
oq��G��Z�^���fK�0_��h��s�b��AE=�rR~���(�^
+a�a̙�IB�,S�!ꀔd�2H~�X�Z����R��.f
...
$ qrandom --hex
1dc59fde43b5045120453186d45653dd455bd8e6fc7d8c591f0018fa9261ab2835eb210e8
e267cf35a54c02ce2a93b3ec448c4c7aa84fdedb61c7b0d87c9e7acf8e9fdadc8d68bcaa5a
...

Creating /dev/qrandom

quantumrandom comes equipped with a multi-threaded character device in userspace. When read from, this device fires up a bunch of threads to fetch data. Not only can you utilize this as a rng, but you can also feed this data back into your system's entropy pool.

In order to build it's dependencies, you'll need the following packages installed: svn gcc-c++ fuse-devel gccxml libattr-devel. On Fedora 17 and newer, you'll also need the kernel-modules-extra package installed for the cuse module.

pip install ctypeslib hg+https://cusepy.googlecode.com/hg
sudo modprobe cuse
sudo chmod 666 /dev/cuse
qrandom-dev -v
sudo chmod 666 /dev/qrandom
By default it will use 3 threads, which can be changed by passing '-t #' into the qrandom-dev.

Testing the randomness for FIPS 140-2 compliance

$ cat /dev/qrandom | rngtest --blockcount=1000
rngtest: bits received from input: 20000032
rngtest: FIPS 140-2 successes: 1000
rngtest: FIPS 140-2 failures: 0
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 0
rngtest: FIPS 140-2(2001-10-10) Long run: 0
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=17.696; avg=386.711; max=4882812.500)Kibits/s
rngtest: FIPS tests speed: (min=10.949; avg=94.538; max=161.640)Mibits/s
rngtest: Program run time: 50708319 microseconds

Adding entropy to the Linux random number generator

sudo rngd --rng-device=/dev/qrandom --random-device=/dev/random --timeout=5 --foreground

Monitoring your available entropy levels

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

Python API

The quantumrandom Python module contains a low-level get_data function, which is modelled after the ANU Quantum Random Number Generator's JSON API. It returns variable-length lists of either uint16 or hex16 data.

>>> quantumrandom.get_data(data_type='uint16', array_length=5)
[42796, 32457, 9242, 11316, 21078]
>>> quantumrandom.get_data(data_type='hex16', array_length=5, block_size=2)
['f1d5', '0eb3', '1119', '7cfd', '64ce']

Based on this get_data function, quantumrandom also provides a bunch of higher-level helper functions that make easy to perform a variety of tasks.

>>> quantumrandom.randint(0, 20)
5
>>> quantumrandom.hex()[:10]
'8272613343'
>>> quantumrandom.binary()[0]
'\xa5'
>>> len(quantumrandom.binary())
10000
>>> quantumrandom.uint16()
numpy.array([24094, 13944, 22109, 22908, 34878, 33797, 47221, 21485, 37930, ...], dtype=numpy.uint16)
>>> quantumrandom.uint16().data[:10]
'\x87\x7fY.\xcc\xab\xea\r\x1c`'

Follow quantumrandom on GitHub: https://github.com/lmacken/quantumrandom


posted at: 16:30 | link | Tags: , , , , | 6 comments

Sat, 07 May 2011

Red Hat OpenShift Express & The Leafy Miracle

Red Hat made a lot of awesome announcements this week at The Red Hat Summit, one of which being OpenShift.

I had the opportunity to play with the internal beta for a little while now, and I must say that as a developer I am extremely impressed with the service. Just being able to git push my code into to the cloud drastically simplifies large-scale software deployment, and makes it so I don't even have to leave my development environment.

I figured out a way to get TurboGears2 and Pyramid running on OpenShift Express, and documented it here and here. After that, I proceeded to write my very first Pyramid application.

[ The Leafy Miracle ]

In memory of the proposed [and rejected] Fedora 16 codename "Beefy Miracle", this little app is called "Leafy Miracle".

leafy-miracle.rhcloud.com


[ Features & Tech ]

[ Running ]
sudo yum -y install python-virtualenv
git clone git://fedorapeople.org/~lmacken/leafymiracle && cd leafymiracle
virtualenv env && source env/bin/activate
python setup.py develop
python leafymiracle/populate.py
paster serve development.ini
[ Code ]

git clone git://fedorapeople.org/~lmacken/leafymiracle

[ Props ]

Mad props go out to RJ Bean, who helped me write this app. He is responsible for writing a ton of amazing Python widgets for various JavaScript visualization libraries. You can see some demos of them here: tw2-demos.threebean.org.


posted at: 20:20 | link | Tags: , , , , , , , , , , | 0 comments

Thu, 24 Mar 2011

git clone all of your Fedora packages

After doing a fresh Fedora 15 install on my laptop last night, I wanted to quickly clone all of the packages that I maintain. Here is a single command that does the job:

python -c "import pyfedpkg; from fedora.client.pkgdb import PackageDB; [pyfedpkg.clone(pkg['name'], '$USER') for pkg in PackageDB().user_packages('$USER')['pkgs']]"


posted at: 16:10 | link | Tags: , , | 4 comments

Sun, 13 Mar 2011

Fedora Photobooth @ SXSW

This is the first year that Fedora will have a booth at SXSW! Sadly, I am not going to be attending since it conflicts with PyCon. However, my code will be running at our booth. Usually the Fedora booth at conferences is comprised of a bunch of flyers, media, swag, and some people to help answer questions and tell the Fedora story. However at SXSW, things are going to be a little different.

Aside from the amazing flyers that Máirín created, there will also be a Fedora Photobooth. Someone (probably Spot or Jared) will be dressed in a full Tux costume, and people can come and get their photo taken with them. Spot came to me the other day and asked if I could write some code to streamline the whole process.

An hour or so later, photobooth.py was born. There are definitely lots of improvements that can be made, but here is what it currently does in its initial incarnation:

In Action
See Mo's blog for photos of this code in action at the Fedora SXSW booth!
* SXSW Expo Day 1 from the show floor
* SXSW Expo Day 2
* A Beefy, Miraculous Day at SXSW (Expo Day 3)


The Code
I threw this in a git repo and tossed it up on GitHub:
github.com/lmacken/photobooth.py
#!/usr/bin/python
# photobooth.py - version 0.3
# Requires: python-imaging, qrencode, gphoto2, surl
# Author: Luke Macken <lmacken@redhat.com>
# License: GPLv3

import os
import surl
import Image
import subprocess

from uuid import uuid4
from os.path import join, basename, expanduser

# Where to spit out our qrcode, watermarked image, and local html
out = expanduser('~/Desktop/sxsw')

# The watermark to apply to all images
watermark_img = expanduser('~/Desktop/fedora.png')

# This assumes ssh-agent is running so we can do password-less scp
ssh_image_repo = 'fedorapeople.org:~/public_html/sxsw/'

# The public HTTP repository for uploaded images
http_image_repo = 'http://lmacken.fedorapeople.org/sxsw/'

# Size of the qrcode pixels
qrcode_size = 10

# Whether or not to delete the photo after uploading it to the remote server
delete_after_upload = True

# The camera configuration
# Use gphoto2 --list-config and --get-config for more information
gphoto_config = {
    '/main/imgsettings/imagesize': 3, # small
    '/main/imgsettings/imagequality': 0, # normal
    '/main/capturesettings/zoom': 70, # zoom factor
}

# The URL shortener to use
shortener = 'tinyurl.com'

class PhotoBooth(object):

    def initialize(self):
        """ Detect the camera and set the various settings """
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 --auto-detect ' +
                        ' '.join(cfg), shell=True)

    def capture_photo(self):
        """ Capture a photo and download it from the camera """
        filename = join(out, '%s.jpg' % str(uuid4()))
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 ' +
                        '--capture-image-and-download ' +
                        '--filename="%s" ' % filename,
                        shell=True)
        return filename

    def process_image(self, filename):
        print "Processing %s..." % filename
        print "Applying watermark..."
        image = self.watermark(filename)
        print "Uploading to remote server..."
        url = self.upload(image)
        print "Generating QRCode..."
        qrcode = self.qrencode(url)
        print "Shortening URL..."
        tiny = self.shorten(url)
        print "Generating HTML..."
        html = self.html_output(url, qrcode, tiny)
        subprocess.call('firefox "%s"' % html, shell=True)
        print "Done!"

    def watermark(self, image):
        """ Apply a watermark to an image """
        mark = Image.open(watermark_img)
        im = Image.open(image)
        if im.mode != 'RGBA':
            im = im.convert('RGBA')
        layer = Image.new('RGBA', im.size, (0,0,0,0))
        position = (im.size[0] - mark.size[0], im.size[1] - mark.size[1])
        layer.paste(mark, position)
        outfile = join(out, basename(image))
        Image.composite(layer, im, layer).save(outfile)
        return outfile

    def upload(self, image):
        """ Upload this image to a remote server """
        subprocess.call('scp "%s" %s' % (image, ssh_image_repo), shell=True)
        if delete_after_upload:
            os.unlink(image)
        return http_image_repo + basename(image)

    def qrencode(self, url):
        """ Generate a QRCode for a given URL """
        qrcode = join(out, 'qrcode.png')
        subprocess.call('qrencode -s %d -o "%s" %s' % (
            qrcode_size, qrcode, url), shell=True)
        return qrcode

    def shorten(self, url):
        """ Generate a shortened URL """
        return surl.services.supportedServices()[shortener].get({}, url)

    def html_output(self, image, qrcode, tinyurl):
        """ Output HTML with the image, qrcode, and tinyurl """
        html = """
            <html>
              <center>
                <table>
                  <tr>
                    <td colspan="2">
                        <b><a href="%(tinyurl)s">%(tinyurl)s</a></b>
                    </td>
                  </tr>
                  <tr>
                    <td><img src="%(image)s" border="0"/></td>
                    <td><img src="%(qrcode)s" border="0"/></td>
                  </tr>
                </table>
              </center>
          </html>
        """ % {'image': image, 'qrcode': qrcode, 'tinyurl': tinyurl}
        outfile = join(out, basename(image) + '.html')
        output = file(outfile, 'w')
        output.write(html)
        output.close()
        return outfile

if __name__ == "__main__":
    photobooth = PhotoBooth()
    try:
        photobooth.initialize()
        while True:
            raw_input("Press enter to capture photo.")
            filename = photobooth.capture_photo()
            photobooth.process_image(filename)
    except KeyboardInterrupt:
        print "\nExiting..."


posted at: 02:57 | link | Tags: , , , | 21 comments

Wed, 16 Feb 2011

FUDCon 2011 Tempe

I meant to get this summary out the door weeks ago, however, I didn't want to distract from my post-FUDCon productivity :)

Another FUDCon has come and gone. This time it was in gorgeous Tempe, Arizona. It was my first time in AZ, and I must say that I was thoroughly impressed. It was truly a great location for FUDCon. Thanks to the epic snowstorm of doom, I also was stuck there for a few extra days

Live Widgetry
Before the conference I decided to throw together a little live widget that scrolls all of the new identi.ca posts tagged with #FUDCon. Thanks to the mad design skillz of mizmo, and the feed aggregation and real-time web sockets of Moksha, I was able to throw it together pretty quickly. I plan on taking this code and integrating it in the existing fedoracommunity dashboard and hooking up many different fedora-related feeds to it.

Sessions

AutoQA
Day 1 of the FUDCon sessions were quite interesting. I got a chance to learn a bit more about the exciting AutoQA project, which is coming along nicely. You'll be seeing AutoQA commenting on bodhi updates soon enough.

Security Lab
I caught Joerg Simon's session on the Fedora Security Lab. It was exciting to see how the Security Spin has evolved ever since I created it back in 2007 for a project in my forensics class. It was also interesting to learn more about OSSTMM.

Spins
The next session I attended was about the future of spins. Almost everyone agreed that Spins are useful and a valuable part of Fedora. The problems seem to mostly lie in governance/policy and a lack of communication and coordination between the Spins SIG and QA/Releng/Infrastructure. Once of my ideas below may help with this a little bit.

The Next Big Project

Day 2 was comprised of more sessions, including my team's "Next Big Project" proposals. Of course, the Fedora RPG that Spot and Mo talked about was definitely a hot topic, and got a lot of people excited.

I talked about a handful of project ideas that I would like to work on in the future (actually, I have code written for most of them already). Here is a quick rundown:

Real-time Infrastructure
I want to see us deploy an AMQP message broker inside our production infrastructure. Then, we hook up all of our existing services and have them fire off messages when various events occur (koji builds, bodhi updates, pkgdb additions/removals/changes, git hooks, planet feeds, wiki edits, etc). From here, using some realtime web technology that we created, we could easily expose these message queues via a live dashboard that lets you filter and navigate the stream of activity, along with providing real-time metrics. We can also create desktop notification widgets, so you can get popup bubbles for things that you care about.

This is also key to the whole RPG as well. In order to build a game based on Fedora workflows, we need an underlying expert system that knows what actions can be taken within fedora, how they are accomplished, and who is getting them done.

Meeting app
Currently after a meeting, our Meetbot spits out the logs and an overview in txt/html/rst to a directory on meetbot.fedoraproject.org. Trying to track down who agreed to what when, or even to see what a given team has been up to over the past couple of months, is very tedious.

The data is there, now we just need to make it useful. I would love to see a frontend for this sytem that tracked meetings by team/people/topics/projects, kept track of actions and held people accountable for what they say they are going to do (and make it easy for people to say "I need help with this", or "I don't have time to finish this"), making it simple to go from an #idea in a meeting to implementation. There is so much great data in these logs, and I think we can do some awesome things with it.

Improved upstream monitoring
Most Fedora developers probably don't even know that we already have an Upstream release monitoring service buried in the wiki. I added almost every package I maintain to it, and it will automatically open a bug when a new upstream version is released. Extremely useful.

I wouldn't call this a "big" project, but I would like to see us integrate this service into our existing infrastructure. We could potentially store this per-package upstream data in the pkgdb/bodhi, and when a new release comes out write some code to automatically try doing a simple specfile bump, throw a scratch build at koji, run it through AutoQA, queue up for testing in bodhi, etc. Ideally, this would minimize the massive amounts of effort that our maintainers have to do to keep our packages up to speed with upstream.

Discussions app
Last year, Máirín Duffy and I came up with some interesting ideas for improving our mailing lists. I would like to make this a reality. Since then, I have already written code that can successfully parse all of fedora-devel.mbox (sounds much easier than it really is), populate it into a SQLAlchemy database model, expose a JSON API for quering, and visualize threads and various statistics with a basic widget.

Spin Master
One of the biggest problems with spins that we have right now is that there is no easy way to track how they are evolving. I would like to see us create a system that took the nightly spins and analyzed them, tracking what packages have been added/removed, which packages have grown/shrunk, etc. We could potentially get AutoQA involved here and make sure all spins pass a certain level of sanity checks before they can even be released. There are scripts floating around that can do a lot of this already -- but I want to streamline it and build a frontend.

Source-level package diff viewer
Fedora churns at such a fast pace, yet I can only imagine that a small subset of maintainers actually look at the complete code changes between upstream package releases. The recent sourceforge intrusion should be seen as a reality check to us distributions, and I think we need to step up our game quite a bit to ensure we don't let any malicious code slip into Fedora.

I would like to see a web/cli interface for viewing full source diffs of package updates in Fedora, allowing people to annotate/flag lines of code. Having more eyes view the code changes that go into our distribution is definitely a Good Thing, not just for Fedora, but for Open Source in general.

Hackfests

Now, on to my favorite part of any conference -- The Hackfests. First off, I felt that the hackfests were a bit unorganized this year. There was no opportunity to pitch hackfests, and it was not easy to figure out who was doing what in which rooms. Anyway, I had a fairly productive day of hacking...

bodhi
I integrated our package test cases into bodhi, which will now query the wiki for tests and display them in your updates, like so:

Thanks to James Laska for the code to query the wiki, and to Ian Weller for python-simplemediawiki API.

I also sat down with Máirín Duffy and talked about Bodhi v2.0 interaction design. We discussed what actions we want people to take when they arrive at bodhi's homepage, which essentially boils down to submitting, searching, browsing, and testing updates. Mo quickly threw together an awesome mockup that portrays some of our initial ideas.


Fedora Community Discussions
As mentioned above, I have already started implementing the mailing list interface that mizmo and I designed last year. I worked with Casey Dahlin during the hackfests and helped him get a working fedoracommunity/moksha development environment up and running and become familiar with the existing code.

Even though he wasn't at FUDCon, Jan Hutar has also been working on a couple of great graphs/grids of mailing list statistics for fedoracommunity as well. We'll be hacking away at this stuff over the next few months, so stay tuned.

kernel EFI framebuffer
I spun up a quick kernel patch to enable the EFI framebuffer on a handfull of Macs. I already wrote a patch that got applied upstream that enables this framebuffer on 14 different mac models, but this new patch adds 5 more. A lot of people, especially Sugar on a Stick users, are desperate to get Fedora running on their mactel machines (assuming found in may school labs), so I spun up a fresh livecd with my kernel patch for testing. See Bug #528232 for more information.

liveusb-creator
I did a bunch of work on porting the liveusb-creator from HAL to UDisks. Thankfully, Ubuntu's cleverly-named "usb-creator" already has UDisks support, so I've been happily borrowing ideas from their code :)

I also had some great discussions with Peter Robinson and Bernie Innocenti about solving the persistent overlay problem with our Live USBs. Right now, they are essentially a ticking time bomb, and real world LiveUSB use-cases are getting bit by this all of the time. Over the past few years, the "solution" has been to "wait for unionfs to get merged into the kernel". However, there are a variety of other potential solutions that we are going to look into as well.

Pyramid
The future of Python web development is extremely exciting, innovative, and still evolving at a rapid pace. As a TurboGears developer, I'm still very impressed with TG2, which along with TG1 will be supported for a long time to come -- but I'm also very eager for the next generation framework that has just emerged.

Recently, the Pylons project has merged with repoze.bfg to form Pyramid, which just released version 1.0. I'm quite amazed by the quality of the code, docs, and tests, and benchmarks already show it blowing rails/django/tg/pylons out of the water. I'm looking forward to the PyCon sprints, where all 3 communitites are going to be sitting at the same table working together on this project.

So while I was stuck in Tempe during the snow storm, I wrote 7 RPMs for Pyramid and it's dependencies, which are currently waiting to be reviewed. I plan on writing Bodhi v2.0 using Pyramid, so if this is something that interests you, let me know (and start reading the 600+ pages of docs ;)

Good times
Technical stuff aside, I had a blast at FUDCon, and I feel like it was one of the best. FUDPub was great, as usual. I played a lot of poker [poorly], ate a lot of tasty food, and had some great conversations. I had caught a cold prior to coming to FUDCon, so instead of nursing it with some nyquil and sleep, I decided to try to nurse it with a bottle of Jack, which turned out to be a bad idea.

Also, no FUDCon would be complete without mentioning TheGreatestGameYouWillEverPlay.com. Once the hackfests started winding down, I gave a quick session on nethack, where I tought people various ways to steal from shops, using a bunch of screecasts that I had on my laptop.


posted at: 21:55 | link | Tags: , , | 0 comments

Thu, 06 Jan 2011

liveusb-creator 3.9.3 windows release

I spent the majority of yesterday at a DOS prompt. Thankfully, it wasn't as painful as it sounds, as git, vim and Python make Windows development quite tolerable.

Anyway, I was finally able to track down and fix a couple of major bugs in the liveusb-creator on Windows XP and 7, and I pushed out a new build yesterday with the following changes:

Windows users, download it here: http://liveusb-creator.fedorahosted.org


posted at: 15:34 | link | Tags: , , , | 7 comments

Tue, 08 Jun 2010

Fedora Updates Report

I recently wrote some code to generate detailed statistics of Fedora & EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics.

This report definitely conveys the shortcomings in how we currently utilize bodhi for "testing" updates, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria, I think we'll see these numbers drastically improve in the future.

You can find the code that generates these statistics here: metrics.py, log_stats.py. If you have any ideas or suggestions for different types of metrics to generate, or if you find any bugs in my code, please let me know.

Bodhi Statistics Report (Generated on June 8th, 2010)
=====================================================

Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
 * notting (424)
 * mclasen (366)
 * jkeating (321)
 * adamwill (283)
 * cwickert (161)
 * rdieter (159)
 * pbrobinson (141)
 * kevin (141)
 * cweyl (122)
 * tomspur (119)
 * mtasaka (110)
 * xake (97)
 * cschwangler (86)
 * kwright (84)
 * peter (83)
 * hadess (80)
 * michich (72)
 * tagoh (69)
 * pfrields (69)
 * bpepple (69)
 * iarnell (68)
 * lkundrak (66)
 * shinobi (65)
 * sundaram (64)
 * spot (62)
 * pravins (62)
 * markmc (62)
 * thomasj (61)
 * smooge (60)
 * fab (59)

================================================================================
     Fedora 13
================================================================================

 * 3562 updates
 * 3065 stable updates
 * 427 testing updates
 * 62 pending updates
 * 8 obsolete updates
 * 2371 bugfix updates (66.56%)
 * 745 enhancement updates (20.92%)
 * 89 security updates (2.50%)
 * 357 newpackage updates (10.02%)
 * 410 critical path updates (11.51%)
 * 333 critical path updates approved
 * 1155 updates received feedback (32.43%)
 * 12120 +0 comments
 * 2477 +1 comments
 * 155 -1 comments
 * 595 unique authenticated karma submitters
 * 133 anonymous users gave feedback (1.57%)
 * 2261 out of 3562 updates went through testing (63.48%)
 * 1317 testing updates were pushed *without* karma (58.25%)
 * 21 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 11 days
   * median = 9 days
   * mode = 7 days
 * 4 updates automatically unpushed due to karma (0.11%)
   * 0 of which were critical path updates
 * 231 updates automatically pushed due to karma (6.49%)
   * 2 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2445 packages updated (top 10 shown)
    * selinux-policy: 13
    * jd: 12
    * openoffice.org: 12
    * gdb: 12
    * ibus-pinyin: 11
    * nautilus: 10
    * kernel: 10
    * evolution: 9
    * libfm: 9
    * libmx: 9

================================================================================
     Fedora 12
================================================================================

 * 4844 updates
 * 4291 stable updates
 * 371 testing updates
 * 113 pending updates
 * 69 obsolete updates
 * 2905 bugfix updates (59.97%)
 * 1054 enhancement updates (21.76%)
 * 201 security updates (4.15%)
 * 684 newpackage updates (14.12%)
 * 407 critical path updates (8.40%)
 * 960 updates received feedback (19.82%)
 * 16311 +0 comments
 * 1899 +1 comments
 * 554 -1 comments
 * 758 unique authenticated karma submitters
 * 576 anonymous users gave feedback (5.33%)
 * 2873 out of 4844 updates went through testing (59.31%)
 * 2138 testing updates were pushed *without* karma (74.42%)
 * 188 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 14 days
   * median = 13 days
   * mode = 17 days
 * 12 updates automatically unpushed due to karma (0.25%)
   * 4 of which were critical path updates
 * 133 updates automatically pushed due to karma (2.75%)
   * 13 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2902 packages updated (top 10 shown)
    * qbittorrent: 25
    * gdb: 25
    * selinux-policy: 22
    * kernel: 15
    * xorg-x11-server: 14
    * ibus: 13
    * jd: 13
    * abrt: 11
    * gvfs: 11
    * gtk2: 11

================================================================================
     Fedora 11
================================================================================

 * 6987 updates
 * 6381 stable updates
 * 183 testing updates
 * 99 pending updates
 * 324 obsolete updates
 * 3649 bugfix updates (52.23%)
 * 1566 enhancement updates (22.41%)
 * 350 security updates (5.01%)
 * 1422 newpackage updates (20.35%)
 * 383 critical path updates (5.48%)
 * 729 updates received feedback (10.43%)
 * 23427 +0 comments
 * 1197 +1 comments
 * 448 -1 comments
 * 782 unique authenticated karma submitters
 * 481 anonymous users gave feedback (3.58%)
 * 4129 out of 6987 updates went through testing (59.10%)
 * 3620 testing updates were pushed *without* karma (87.67%)
 * 278 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 15 days
   * median = 14 days
   * mode = 17 days
 * 7 updates automatically unpushed due to karma (0.10%)
   * 0 of which were critical path updates
 * 64 updates automatically pushed due to karma (0.92%)
   * 11 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 3787 packages updated (top 10 shown)
    * libguestfs: 30
    * jd: 24
    * selinux-policy: 23
    * kdebase-workspace: 19
    * kernel: 18
    * gdb: 16
    * dovecot: 16
    * qemu: 16
    * kdebase-runtime: 16
    * kdenetwork: 16

================================================================================
     Fedora EPEL 5
================================================================================

 * 1572 updates
 * 1255 stable updates
 * 198 testing updates
 * 43 pending updates
 * 76 obsolete updates
 * 734 bugfix updates (46.69%)
 * 236 enhancement updates (15.01%)
 * 93 security updates (5.92%)
 * 509 newpackage updates (32.38%)
 * 20 critical path updates (1.27%)
 * 103 updates received feedback (6.55%)
 * 6076 +0 comments
 * 156 +1 comments
 * 19 -1 comments
 * 243 unique authenticated karma submitters
 * 41 anonymous users gave feedback (1.22%)
 * 1176 out of 1572 updates went through testing (74.81%)
 * 1092 testing updates were pushed *without* karma (92.86%)
 * 19 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 24 days
   * median = 18 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 10 updates automatically pushed due to karma (0.64%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 1060 packages updated (top 10 shown)
    * libguestfs: 26
    * znc: 10
    * vrq: 8
    * cherokee: 8
    * 389-ds-base: 8
    * viewvc: 8
    * 389-admin: 7
    * pki-ca: 7
    * wordpress-mu: 7
    * Django: 7

================================================================================
     Fedora EPEL 4
================================================================================

 * 447 updates
 * 359 stable updates
 * 40 testing updates
 * 11 pending updates
 * 37 obsolete updates
 * 222 bugfix updates (49.66%)
 * 68 enhancement updates (15.21%)
 * 40 security updates (8.95%)
 * 117 newpackage updates (26.17%)
 * 5 critical path updates (1.12%)
 * 11 updates received feedback (2.46%)
 * 1592 +0 comments
 * 11 +1 comments
 * 2 -1 comments
 * 85 unique authenticated karma submitters
 * 2 anonymous users gave feedback (0.24%)
 * 320 out of 447 updates went through testing (71.59%)
 * 311 testing updates were pushed *without* karma (97.19%)
 * 5 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 18 days
   * median = 16 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 1 updates automatically pushed due to karma (0.22%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 313 packages updated (top 10 shown)
    * cherokee: 8
    * globus-common: 7
    * R: 6
    * voms: 6
    * globus-gsi-proxy-ssl: 5
    * globus-openssl-module: 5
    * globus-gsi-proxy-core: 5
    * bitlbee: 5
    * flashrom: 5
    * viewvc: 5


posted at: 20:41 | link | Tags: , , , | 0 comments

Tue, 25 May 2010

liveusb-creator trojan in the wild

I've been noticing many different copies of my Windows liveusb-creator popping up on various sketchy-looking download sites. The majority of these copies contain a variant of the Vundo Trojan.

"Vundo, or the Vundo Trojan (also known as Virtumonde or Virtumondo and sometimes referred to as MS Juan) is a Trojan horse that is known to cause popups and advertising for rogue antispyware programs, and sporadically other misbehavior including performance degradation and denial of service with some websites including Google and Facebook."

So, if you downloaded a copy of the Windows liveusb-creator from anywhere other than https://fedorahosted.org/liveusb-creator -- you could be infected. Apparently the latest variation of this trojan is undetectable by most antivirus (although, clamav was able to recognize the one that I found), so you may need to look around for some of the common symptoms. There is apparently a tool that will remove this trojan which can be found here, however I have not tested it and cannot vouch for its validity.

If anyone was actually hit by this, I'd be interested to hear about it.

Also, to state the blatantly obvious: only download the liveusb-creator from the homepage!


posted at: 21:11 | link | Tags: , , , | 4 comments

Sat, 27 Mar 2010

Fedora Community Statistics

I'm pleased to announce that version 0.4.0 of the Fedora Community dashboard has just hit production. Along with the usual batch of bugfixes, this release contains a new 'Statistics' section that contains metrics from a variety of different pieces of Fedora Infrastructure.

Thanks goes to Ian Weller for the wiki stats, Seth Vidal for the torrent stats, and Matt Domsch & Jef Spaleta for the map generation code. I ended up writing the updates metrics, package stats, and users/mirrors widgets. Enjoy!








posted at: 03:48 | link | Tags: , , | 4 comments

Mon, 18 Jan 2010

nose 0.11

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module.

lmacken@tomservo ~/bodhi $ nosetests
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 725.111s

OK

lmacken@tomservo ~/bodhi $ nosetests --processes=50
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 10.915s

OK

Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month


posted at: 22:58 | link | Tags: , , , , | 4 comments

Thu, 10 Dec 2009

FUDCon Toronto 2009

Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion.

My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on.

Moksha

Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later.

One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker.

I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing:

"I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is https://fedorahosted.org/moksha/ and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well http://meetbot.fedoraproject.org/fudcon-room-3/2009-12-05/fudcon-room-3.2009-12-05-22.07.log.html ... presentation slides found: http://lmacken.fedorapeople.org/moksha-FUDConToronto-2009.odp"

Fedora Community

So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform.

After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen.

During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them).

I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to come. My main concern was a blocker issue that we were hitting with our flot graphs when you quickly bounce between tabs. I ended up "fixing" the bug, so I'll be pushing what we have of the stats branch into production in the near future.

TurboGears2

TurboGears has definitely been our favorite web framework within Fedora's Infrastructure for many years now. TurboGears2, a complete re-invention of itself, has been released recently, and is catching on *very* quickly in the community. Tons of people are working on awesome new apps, and loving every minute of it. I was also able to convert a rails hacker over to it, after he was able to quickly dive into one of the tutorials with ease. See my previous blog post about getting up and running with TG2 in Fedora/EPEL.

python-fedora

One of my main tasks during the hackfests was to pull the authentication layer in Fedora Community that authenticates against the Fedora Account System, and port it over to python-fedora, so we can use it in any TurboGears2 application. I committed the initial port to python-fedora-devel, and have started working on integrating it into a default TG2 quickstart and document the process. There are still a couple of minor things I want to fix/clean up before releasing it, so expect a blog about it soon.

Bodhi

It seems like yesterday that I was an intern at Red Hat working on an internal updates system for Fedora Core. Coming up on 5 years later, and I am now working on my 3rd implementation of an updates system, Bodhi v2.0. What's wrong with the current Bodhi you ask? Well, if you talk to any user of it, you'll probably get a pretty long list. Bodhi is the first TurboGears application written & deployed in Fedora Infrastructure, and uses the vanilla components (SQLObject, kid, CherryPy2). The TG1 stack has been holding up quite nicely over the years, and is still supported upstream, but bodhi's current implemention and design does not make it easy to grow.

Bodhi v2.0 will be implemented in TurboGears2, using SQLAlchemy for an ORM, Mako for templates, and ToscaWidgets2 for re-usable widgets. It will be hook-based and plugin-driven, and will be completely distribution agnostic. Another important goal will be AMQP message-bus integration, which will allow other services or users to react to various events inside of the system as they happen.

So far I've ported the old DB model from SQLObject to SQLAlchemy, and have begun porting the old unit tests, and writing new ones. Come the new year, I'll be giving this much more of my focus.

During the hackfests I got a chance to talk to Dennis Gilmore about various improvements that we need to make with regard to the update push process. It was also great to talk to many different users of bodhi, who expressed various concerns, some of which I've already fixed. I also got a chance to talk to Xavier Lamien about deploying Bodhi for rpmfusion. On the bus ride home I helped explain to Mel how Bodhi & Koji fit into the big picture of things.

During the BarCamp sessions I also attended a session about the Update Experience, where we discussed many important issues surrounding updates.

liveusb-creator

So I got a chance to finally meet Sebastian Dziallas, of Sugar on a Stick fame, and was able to fix a few liveusb-creator issues on his laptop. I ended up pushing out a new release a couple of days ago that contains some of those fixes, along with a new version of Sugar on a Stick.

The liveusb-creator has been catching a lot of press recently (see the front page for a list). Not only did it have a 2 page spread in Linux Format, but it was also featured in this weeks Wired.com article New Sugar on a Stick Brings Much Needed Improvements. Rock.

Python

There was lot of brainstorming done by Dave Malcolm, Colin Walters, Toshio Kuratomi, Bernie Innocenti, I, and many others about various improvements that we could make to the Python interpreter. From speeding up startup time by doing some clever caching to potentially creating a new optimized compiled binary format. We also looked into how WebError/abrt gather tracebacks, and discussed ways of enabling interactive traceback debugging for vanilla processes, without requiring a layer of WSGI middleware.

There was also work done on adding SystemTap probes to Python, which is very exciting. There are many ideas for various probe points, including one that I blogged about previously.

Intel iMac8,1 support

My iMac sucks at Linux. This has been something that has been nagging me for a long time, and I've been slowly trying to chip away at the problems. First, I've been doing work on a Mac port of the liveusb-creator. I also started to work on a kernel patch for getting the EFI framebuffer working, and discussed how to do it with ajax and pjones. The screen doesn't display anything after grub, and since we don't know the base address of the framebuffer, it involves writing code to iterate over memory trying to find some common pixel patterns. I'm still trying to wrap my head around all of it, but I'll probably end up just buying them beer to fix it for me.

Thincrust

Thincrust is a project that I've been excited about for a while, and I actually have some appliances deployed in a production cloud. I was able to run some ideas for various virtual appliances by one of the authors over some beers. Some pre-baked virtual appliances that you can easily throw into a cloud that I would like to see:

dogtail

I'm glad to see that dogtail is still exciting people in the community. It still has a lot of potential to improve not only the way we test graphical software, but we also discussed ways of using it to teach people and automate various desktop tasks. What if you logged in after a fresh install and got the following popup bubble:

Hi, welcome to Fedora, what can I help you do today?

Each task would then allow Fedora to take the wheel and walk the user through various steps. I had this idea a while ago, when dogtail first came out, and I still think it would be totally awesome. Anyway, this was not a focus of the hackfests, but merely a conversation that I had while walking to lunch :)


posted at: 17:49 | link | Tags: , , , , , , , | 9 comments

Mon, 09 Nov 2009

New liveusb-creator release!

So I've gotten some pretty inspiring feedback from various users of the liveusb-creator recently, so I decided to put some cycles into it this weekend and crank out another release.

"As a non-Linux person, Live-USB Creator has improved the quality of my life measurably!" --Dr. Arthur B. Hunkins
Yesterday I released version 3.8.6 of the liveusb-creator. Changes in this release include:
Windows
https://fedorahosted.org/releases/l/i/liveusb-creator/liveusb-creator-3.8.6.zip

Fedora
https://admin.fedoraproject.org/updates/liveusb-creator-3.8.6-1.fc11
https://admin.fedoraproject.org/updates/liveusb-creator-3.8.6-1.fc12

Source
https://fedorahosted.org/releases/l/i/liveusb-creator/liveusb-creator-3.8.6.tar.bz2

Trac
http://liveusb-creator.fedorahosted.org


posted at: 02:39 | link | Tags: , , , | 1 comments

Fri, 10 Jul 2009

Bodhi EPEL support!

It's been a long time coming, but the Extra Packages for Enterprise Linux (EPEL) project is finally utilizing the Koji build system and the Bodhi updates system.

So I spent the past week hacking on EPEL support in bodhi. This was not a trivial task, and took more work than expected. Overall, it was a very beneficial experience, as I was able to hack in some higher level abstractions and also remove a lot of Fedora-specific assumptions in the code. Most of the changes were what I would normally call "hacks", mainly because I wanted to do it without changing the database schema. However, this gives me a much clearer picture as to what we need from the Bodhi v2.0 model. Anyway, 28 bodhi upgrades later, and everything seems to be working fine.

The inevitable TurboGears2 rewrite/port of Bodhi is a little further down the road. I've already ported the original model from SQLObject to SQLAlchemy, but the templates, controllers, and widgets still need to be ported. If you're interested in helping make bodhi suck less, then please come talk to me :)

Anyway, developers can now submit their EPEL updates here, or by running `make update` in their EL CVS branches. Admins can read the Bodhi SOP to learn how to push updates.

The workflow is far from perfect, but there has been some recent discussions as to how we want EPEL to be treated differently compared to Fedora updates. If you have suggestions or comments, discussions should take place on epel-devel-list.


posted at: 19:06 | link | Tags: , , | 3 comments

Tue, 27 Jan 2009

bodhi updates push process

Bodhi's push process is something that is usually quite opaque to Fedora package maintainers. Once an update request goes into bodhi, the developer sits back and waits for the update to go to where it needs to go. The ball is then in releng's court, as they must sign the packages, and tell bodhi to begin the push. From there, bodhi does it's thing for a while, and then updates magically end up on our users machines. Yay!

Pushing updates used to take the better part of a day, mostly due to dumb code and lots of filesystem churn over NFS. Thankfully, a lot of the code is now much smarter, and people like jkeating and mmcgrath have been helping to address the NFS & infrastructure bottlenecks.

Hopefully I can help shed some light on one of the dark corners of bodhi known as The Masher. Here are some statistics of the last updates push that happened earlier today.

Initial push request from releng
Check koji tag / bodhi status consistency38s
Move all of the build tags in Koji 9m32s
Update the comps CVS module 11s
Mash f9-updates-testing 4m16s
Mash f9-updates 1h3m8s
Mash f10-updates-testing 12m43s
Mash f10-updates 37m51s
Set update ids, state modifications, updates-testing digest generation 1m57s
Generate updateinfo.xml 5m55s
Repo sanity checks & symlinking to go live 1m4s
Cache latest repodata, and remove old 1m14s
Wait for updates to hit the master mirror 1h1s
Send update notices, update/close bugs, notify developers/commenters 11m11s
Total3h49m42s


So we've obviously made some great improvements here, and once the signing server is deployed, you can probably expect a much more frequent/consistent flow of updates. However, I definitely think there is still a lot of low-hanging fruit in this process, and many steps can probably be done in parallel. We're going to be adding DeltaRPM generation into the mix in the near future, so I'll give an update a bit later with some details as to how that effects the process.

Anyway... if you know Python, and enjoy optimizing code -- come talk to me :)


posted at: 05:15 | link | Tags: , , , | 0 comments

Wed, 31 Dec 2008

liveusb-creator 3.0

I'm pleased to announce version 3.0 of the liveusb-creator. Aside from the usual batch of bug fixes and code improvements, this release also contains a variety of enhancements:

For Fedora 9 and 10 users, you can currently find the liveusb-creator-3.0 in the updates-testing repository. Feedback is appreciated!

There are still many more great features in the pipeline, so stay tuned!


posted at: 23:19 | link | Tags: , , | 2 comments

Taming the fox (part 2)

So, in my previous post I wrote a script that uses xdotool to put firefox to sleep when it does not have focus. The implementation was pretty straight-forward, but not optimal. Thanks to some help from Jordan Sissel, I threw together a Python script that accomplishes this goal much more effectively, using the python-xlib module instead of xdotool.

http://lmacken.fedorapeople.org/tamefox.py
import os
from signal import SIGSTOP, SIGCONT
from Xlib import X, display, Xatom

def watch(properties):
    """ A generator that yields events for a list of X properties """
    dpy = display.Display()
    screens = dpy.screen_count()
    atoms = {}
    wm_pid = dpy.get_atom('_NET_WM_PID')

    for property in properties:
        atomid = dpy.get_atom(property, only_if_exists=True)
        if atomid != X.NONE:
            atoms[atomid] = property

    for num in range(screens):
        screen = dpy.screen(num)
        screen.root.change_attributes(event_mask=X.PropertyChangeMask)

    while True:
        ev = dpy.next_event()
        if ev.type == X.PropertyNotify:
            if ev.atom in atoms:
                data = ev.window.get_full_property(ev.atom, 0)
                id = int(data.value.tolist()[0])
                window = dpy.create_resource_object('window', id)
                if window.id == 0: continue
                pid = int(window.get_full_property(wm_pid, 0).value.tolist()[0])
                title = window.get_full_property(Xatom.WM_NAME, 0).value
                yield atoms[ev.atom], title, pid, data

def tamefox():
    """ Puts firefox to sleep when it loses focus """
    alive = True
    ff_pid = None
    for property, title, pid, event in watch(['_NET_ACTIVE_WINDOW']):
        if title.endswith('Firefox') or title.endswith('Vimperator'):
            ff_pid = pid
            if not alive:
                print 'Waking up firefox'
                os.kill(ff_pid, SIGCONT)
                alive = True
        elif ff_pid and alive and not title.startswith('Opening') and \
                title not in ('Authentication Required', 'Confirm', 'Alert'):
            print 'Putting firefox to sleep'
            os.kill(ff_pid, SIGSTOP)
            alive = False

if __name__ == '__main__':
    tamefox()


posted at: 07:03 | link | Tags: , | 8 comments

Taming the fox

As with most people these days, I have firefox running at all times. I also usually have about 50-100 tabs open (how I manage that insanity should probably be left for it's own blog post). When I'm not actively using it, firefox "idles" about as well as an ADHD kid after a case of Red Bull, and ends up waking my kernel up hundreds of times a second. When I'm hacking in vim, do I really want or need javascript/flash/animations/etc to be running? Probably not...

So tonight I threw together a little script to "solve" this "problem".

#!/bin/bash
# A tool for putting firefox to sleep when it does not have focus.
alive=1
while true; do
    if xwininfo -id $(xdotool getactivewindow) | egrep -q '(Firefox|Vimperator)' ; then
        if [ ! $alive ]; then
            kill -CONT `pidof firefox`
            alive=1
        fi
    else
        if [ $alive ]; then
            kill -STOP `pidof firefox`
            alive=
        fi
    fi
    sleep 1
done

Obviously the most efficient way to do this would be to hook into X's focus events, but I'm lazy...
*UPDATE*: I got un-lazy and implemented a better version using the python-xlib module. See Taming the fox (part 2).

This script uses xdotool, which I just packaged and pushed into review for Fedora. (update: Oops, looks like xdotool is already in Fedora :)


posted at: 04:23 | link | Tags: , | 10 comments

Sun, 14 Dec 2008

>>> from fedora.client import Wiki

I created a simple Python API for interacting with Fedora's MediaWiki a while back, in an attempt to gather various metrics. I just went ahead and committed it to the python-fedora modules. Here is how to use it:

>>> from fedora.client import Wiki
>>> wiki = Wiki()
>>> wiki.print_recent_changes()
From 2008-12-07 20:59:01.187363 to 2008-12-14 20:59:01.187363
500 wiki changes in the past week

== Most active wiki users ==
 Bbbush............................................ 230
 Konradm........................................... 25
 Duffy............................................. 22
 Jreznik........................................... 21
 Ianweller......................................... 14
 Jjmcd............................................. 14
 Geroldka.......................................... 10
 Gdk............................................... 9
 Anouar............................................ 7
 Gomix............................................. 6

== Most edited pages ==
 Features/KDE42.................................... 21
 SIGs/SciTech/SAGE................................. 15
 FUDCon/FUDConF11.................................. 14
 Special:Log/upload................................ 13
 How to be a release notes beat writer............. 12
 Special:Log/move.................................. 11
 Design/SETroubleshootUsabilityImprovements........ 10
 PackageMaintainers/FEver.......................... 9
 User:Gomix........................................ 6
 Zh/主要配置文件..................................... 5

>>> for event in wiki.send_request('api.php', req_params={
...         'action': 'query',
...         'list': 'logevents',
...         'format': 'json',
...         })['query']['logevents']:
...     print '%-10s %-15s %s' % (event['action'], event['user'], event['title'])
...
patrol     Ianweller       User:Ianweller/How to create a contributor business card
move       Nippur          REvanderLuit
patrol     Ianweller       Project Leader
move       Ianweller       FPL
upload     Anouar          Image:AnouarAbtoy.JPG
move       Liangsuilong    ZH/Docs/FetionOnFedora
move       Liangsuilong    FetionOnFedora
patrol     Ianweller       User:Ianweller

It uses the fedora.client.BaseClient, which is a class that simplifies interacting with arbitrary web services. Toshio and I created it a while back as a the core client for talking with our various TurboGears-based Fedora Services (bodhi, pkgdb, fas, etc.), but it has now seemed to morph into a much more flexible client for talking JSON with web applications.

from datetime import datetime, timedelta
from collections import defaultdict
from fedora.client import BaseClient

class Wiki(BaseClient):

    def __init__(self, base_url='http://fedoraproject.org/w/', *args, **kwargs):
        super(Wiki, self).__init__(base_url, *args, **kwargs)

    def get_recent_changes(self, now, then, limit=500):
        """ Get recent wiki changes from `now` until `then` """
        data = self.send_request('api.php', req_params={
                'list'    : 'recentchanges',
                'action'  : 'query',
                'format'  : 'json',
                'rcprop'  : 'user|title',
                'rcend'   : then.isoformat().split('.')[0] + 'Z',
                'rclimit' : limit,
                })
        if 'error' in data:
            raise Exception(data['error']['info'])
        return data['query']['recentchanges']

    def print_recent_changes(self, days=7, show=10):
        now = datetime.utcnow()
        then = now - timedelta(days=days)
        print "From %s to %s" % (then, now)
        changes = self.get_recent_changes(now=now, then=then)
        num_changes = len(changes)
        print "%d wiki changes in the past week" % num_changes

        users = defaultdict(list) # {username: [change,]}
        pages = defaultdict(int)  # {pagename: # of edits}

        for change in changes:
            users[change['user']].append(change['title'])
            pages[change['title']] += 1

        print '\n== Most active wiki users =='
        for user, changes in sorted(users.items(),
                                    cmp=lambda x, y: cmp(len(x[1]), len(y[1])),
                                    reverse=True)[:show]:
            print ' %-50s %d' % (('%s' % user).ljust(50, '.'), len(changes))

        print '\n== Most edited pages =='
        for page, num in sorted(pages.items(),
                                cmp=lambda x, y: cmp(x[1], y[1]),
                                reverse=True)[:show]:
            print ' %-50s %d' % (('%s' % page).ljust(50, '.'), num)

I added a Wiki.login method to the latest version, but it isn't quite working yet. This is due to some minor limitations in the ProxyClient, so we currently cannot handle authenticated requests. However, this shouldn't be very difficult to implement. The reason for this is that we need to be able to run authenticated queries as a 'bot' account in order to mitigate the 500 entry API return limit.

This module makes it easy to talk to MediaWiki's API, so if you do anything cool with it feel free to send patches here. It's currently not being shipped in a python-fedora release, so you'll have to grab the code from Bazaar:

bzr branch bzr://bzr.fedorahosted.org/bzr/python-fedora/python-fedora-devel


posted at: 23:12 | link | Tags: , , , | 13 comments

Sat, 13 Dec 2008

Time spent in updates-testing purgatory

Will Woods asked me on IRC earlier today how easy it would be to determine the amount of time Fedora updates spend in testing within bodhi. It turned out to be fairly easy to calculate, so I thought I would share the code and results.

from datetime import timedelta
from bodhi.model import PackageUpdate

deltas = []
occurrences = {}
accumulative = timedelta()

for update in PackageUpdate.select():
    for comment in update.comments:
        if comment.text == 'This update has been pushed to testing':
            for othercomment in update.comments:
                if othercomment.text == 'This update has been pushed to stable':
                    delta = othercomment.timestamp - comment.timestamp
                    deltas.append(delta)
                    occurrences[delta.days] = occurrences.setdefault(delta.days, 0) + 1
                    accumulative += deltas[-1]
                    break
            break

deltas.sort()
all = PackageUpdate.select().count()
percentage = int(float(len(deltas)) / float(all) * 100)
mode = sorted(occurrences.items(), cmp=lambda x, y: cmp(x[1], y[1]))[-1][0]

print "%d out of %d updates went through testing (%d%%)" % (len(deltas), all, percentage)
print "mean = %d days" % (accumulative.days / len(deltas))
print "median = %d days" % deltas[len(deltas) / 2].days
print "mode = %d days" % mode


4878 out of 10829 updates went through testing (45%)
mean = 17 days
median = 11 days
mode = 6 days

So, it seems that the majority of updates leave updates-testing in less than a week. This is interesting when taking into consideration the testing workflow mechanisms that bodhi employs. An update can go from testing to stable in two ways: 1) The update's karma can reach an optional stable threshold, and automatically get pushed to the stable repository based on positive community feedback. 2) The developer can request that the update be marked as stable. After an update sits in testing for two weeks, bodhi will send the developer nagmail, which seems to help mitigate stale updates. When initially deploying bodhi, I thought that we would get bogged down with a ton of stale testing updates and would have to implement a timeout to have them automatically get marked as stable. This is still a viable option (which would require FESCo rubberstamping), but I'm quite surprised to see how effective this community-driven workflow is already. Now we just need to encourage more people to use it :)

Due to the limitations of the current model I couldn't figure out an easy way to determine which updates were marked as stable by positive community feedback. This issue will be assessed with the long-awaited SQLAlchemy port that I will hopefully finish up at some point early next year.


posted at: 08:13 | link | Tags: , , , | 1 comments

Tue, 28 Oct 2008

TurboGears2 on Fedora

So, I was finally able to force myself to take a break from hacking on a few different TG2 applications to actually sit down and package everything up, submit it all for review, and make sure everything Just Works on Fedora.

You can start playing with TurboGears2 by throwing my tg2.repo file in /etc/yum.repos.d, and then by installing the TurboGears2 package. It should work fine alongside of any existing TurboGears 1.x installs, however it will pull in SQLAlchemy 0.5, which may or may not cause problems with older models. Once installed, checkout the fantastic TurboGears2 documentation for details on diving into the framework.

I already have two TurboGears2 applications currently in production, and have had nothing but amazing experiences with the new stack. I still love working on my older TG1.0 apps (such as bodhi), which will still be supported for a long time, but with the ridiculous amounts of power that WSGI/Pylons/Paste/SQLAlchemy/ToscaWidgets/Mako gives you, I'm extremely tempted to start porting :)

Aside from testing the packages, you can also help by reviewing various parts of the stack so we can get them into Fedora as soon as possible. The TG2 dependency tree of pending reviews can be found here.


posted at: 06:18 | link | Tags: , , | 1 comments

Wed, 10 Sep 2008

bodhi 0.5!

As many have already noticed, I performed a large bodhi upgrade recently. A few weeks ago, during The Incident, I was forced to perform what was originally going to be a week long bodhi migration and upgrade, overnight. During the past two weeks I've pushed out 24 revisions of bodhi to our infrastructure, fixing various show-stoppers, and helping to make sure that updates got out the door.

One of the most noticable changes is that bodhi is much more responsive. Previously, bodhi was a single python process, running on a single server. This single server was also responsible for composing the updates repositories, and rawhide, among lots of other bodhi-related churn. This lead to much pain and suffering for all.

The bodhi deployment has since changed. All bodhi requests are now load balanced to a bunch of app servers, each running mod_wsgi with multiple bodhi processes, each with multiple threads. All of the hard work is now done on an isolated releng server. This separate bodhi "masher" is now responsible for composing repositories, updating bugs, generating update notices, sending emails, extended metadata generation, and calculating metrics. I also added support for inter-bodhi communication, which allows our bodhi web frontends to kick off push requests to our bodhi-masher instance.

Some of the new features in this release:

Bodhi is far from being feature complete. Some new features in the pipeline:

As always,

Also, if you're currently having issues with the bodhi client, a fixed version will be going out with the next batch of updates. For the impatient, you can pull fixed versions from koji (also, make sure your Makefile.common is up to date):

    koji download-build --arch=noarch python-fedora-0.3.5-1.fc10
    koji download-build --arch=noarch bodhi-0.5.2-1.fc9


posted at: 21:20 | link | Tags: , , | 1 comments

Wed, 16 Jul 2008

Python dictionary optimizations

In my recent journey through the book Beautiful Code, I came across a chapter devoted to Python's dictionary implementation. I found the whole thing quite facinating, due to the sheer simplicity and power of the design. The author mentions various special-case optimizations that the Python developers cater for in the CPython dictionary implementation, which I think are valuable to share.

Key lookups
In CPython, all PyDictObject's are optimized for dictionaries containing only string keys. This seems like a very common use case that is definitely worth catering for. The key lookup function pointer looks like this:

struct PyDictObject {
    PyDictEntry *(*ma_lookup)(PyDictObject *mp, PyObject *key, long hash);
    ...

ma_lookup is initially set to the lookdict_string function (renamed to lookdict_unicode in 3.0), which assumes that both the keys in the dictionary and the key being searched for are standard PyStringObject's. It is then able to make a couple of optimiziations, such as mitigating various error checks, since string-to-string comparison never raise exceptions. There is also no need for rich object comparisons either, which means we avoid calling PyObject_RichCompareBool, and always use _PyString_Eq directly.

This string-optimized key lookup function is utilized until you search for a non-string key. When lookdict_string detects this, it permanently changes the ma_lookup function to a slower, more generic lookdict function. Here is an example of how to trigger this degradation:

>>> d = {'foo': 'bar'}
>>> d.get(1) # Congratulations, your dictionary is now slower...
>>> d.get(u'foo') # Yes, even unicode objects trigger this degradation as well

Jython does not contain this optimization, however, it does have a string-specialized map object, org.python.core.PyStringMap, which is used for the __dict__ underpinning of all class instances and modules. User code that creates a dictionary utilizes a different class, org.python.core.PyDictionary, which is a heavyweight object that uses the java.util.Hashtable along with some extra indirection, allowing it to be subclassed.

Small dictionaries
Python's dictionary makes an effort to never be more than 2/3rds full. Since the default size of dict is 8, this allows you to have 5 active entries in your dict while avoiding an additional malloc. Dictionaries used for keyword arguments are usually within this limit, and thus are fairly efficient (along with the fact that they most likely come from a pool of cached unused dicts). This also can help improve cache locality. For example, the PyDictObject structure uses 124 bytes of space (on x86 w/gcc) and therefore can fit into two 64-byte cache lines.

So, the moral of the story: use dictionaries with string-only keys, and only look for string keys within them. If you can keep them small enough to avoid the extra malloc (<= 5), bonus. As expected, things get better in Python 3.0, as unicode keys will no longer slow your dictionary down.


posted at: 15:39 | link | Tags: , | 4 comments

Tue, 27 May 2008

New openbox hotness in Fedora

I've been a huge fan of the *box window managers for a long time -- openbox being my favorite, due to it's extremely minimal nature. Thanks to the help of Fedora openbox maintainer Miroslav Lichvar, the dynamic pipe menu that I wrote a while back is now integrated into Fedora's openbox package. This means that you no longer need to hack XML files to make your menu useful. Here is what it looks like "out of the box":


posted at: 19:57 | link | Tags: , , | 1 comments

Sat, 17 May 2008

liveusb-creator v2.4

Last night I released v2.4 of the liveusb-creator, which fixes a bunch of bugs and is much more robust.

Someone wrote a pretty cool article about the liveusb-creator on Lifehacker the other day, which made it to the front page of digg last night. Digg it up!


posted at: 15:10 | link | Tags: , , | 7 comments

Fedora LiveUSB Creator v2.0!

The liveusb-creator is a cross-platform tool for easily installing live operating systems on to USB flash drives. Today I released version 2.0, which brings you a brand new graphical interface and a bunch of new features, including:

Download: liveusb-creator-2.5.zip (8.8mb)

This release is for Windows-only at the moment. Linux support is nearing completion, and will exist in later versions. In the mean time, there are already tools available for creating persistent LiveUSB keys with Fedora.

If you would like to help contribute to the liveusb-creator, see the Developers Guide for more information. If you encounter problems with the tool, please file bug reports here.

For those interested in trying out this program, but don't have a USB stick, you can buy one here.


posted at: 15:10 | link | Tags: , , | 13 comments

Mon, 24 Mar 2008

PyCon 2008

I was in Chicago last week for PyCon 2008. It was my first time in the windy city, and I must say that I was thoroughly impressed. As expected in any city, we got a chance to see a lady get her purse snattched, and a mentally unstable gentleman on the train yelling profanities at god. Anyway, the conference itself was extremely well done, and tons of awesome innovation happened at the sprints afterwords.

Day 1: Tutorials
8+ hours of TurboGears/Pylons/WSGI tutorials. Awesome. I'm really excited with what is in the works for TurboGears2. By wielding Pylons, the TG2 team was able to completely re-write their framework with minimal amounts of code, while at the same time, gaining a *ton* of new features and some amazing middleware. Mark Ramm and Ben Bangert took turns walking us through the deep internals of their frameworks, while also giving some examples how to use them.

Sessions
During the 3-day conference portion of PyCon, there was a vast plethora of incredibly interesting sessions and conversations. You can find a schedule of the talks and some slides here. Everything was video taped as well, so the sessions should be making their way on to YouTube hopefully at some point soon.

Here are some things that caught my attention while I was there.

WSGI
Defined by Phillip J. Eby in PEP-333, the Web Server Gateway Interface is a simple interface between web servers, applications, and frameworks. Or, as explained by Ian Bicking: WSGI is a series of Tubes. The basic idea is that it lets you connect a bunch of different applications together into a functioning whole. Since TurboGears2 is based on Pylons, it will be a full blown WSGI application out the box, loaded with lots of useful middleware (WebError, Routes, Sessions, Caching, etc), and will allow you to use any WSGI server that you wish (Paste, CherryPy, orbited, mod_wsgi, etc). An example of a basic Hello World WSGI application:

def wsgi_app(environ, start_response):
    start_response('200 OK', [('content-type', 'text/html')])
    return ['Hello world!']

So, what is WSGI middleware? Well, it's essentially the WSGI equivalent of a python decorator, but instead of wrapping one function in another, you're wrapping one web-app in another. You can see a list of some existing WSGI middleware here.

virtualenv
With so many new shiny python programs to play with, I really tried to resist the urge to easy_install everything into my global Python site-packages so I could tinker with things. This is generally a Bad Thing in a distribution, as easy_install not only installs things behind your package managers back, but it also lacks the ability to uninstall anything with it, unless you want to take Zed's easy_fucking_uninstall approach ;) During the TurboGears tutorial, I was introduced to a tool call virtualenv, which will setup a virtual python environment in which you can easy_install as many eggs as you want without worrying about butchering your site-packages.

$ easy_install virtualenv
$ virtualenv --no-site-packages foo
$ cd foo; source bin/activate
$ easy_install <shiny python programs>

nose
I've been in love with nose since day one, but realized that I haven't been utilizing it to it's fullest abilities. I blogged in the past about nose's profiler plugin. Come to find out, nose offers a lot more plugins that can seriously help make your life easier:

$ nosetests --pdb --pdb-failures
.............................................................> /home/lmacken/tg1.1/turbogears/turbogears/identity/tests/test_visit.py(92)test_cookie_permanent()
-> assert abs(should_expire - expires) < 3
(Pdb) locals()
{'morsel': <Morsel: tg-visit='452c94de3900fc2adff2cd6b0b0f04c4533e3e9e'>, 'self': <turbogears.identity.tests.test_visit.TestVisit testMethod=test_cookie_permanent>, 'expires': 1206228604.0, 'should_expire': 1206232205.0, 'permanent': False}
(Pdb)

You can also measure code coverage during your unit test execution using the '--with-coverage' option, which utilizes coverage.py.

SQLAlchemy
Also known as "the greatest object-relational-mapper created for any language. ever.", 0.4 has seen vast improvements since 0.3. Among them, a new declarative API is now available that essentially lets you define your class, Table and mapper constructs "at once" under a single class declaration (giving you a similar ActiveMapper feel like SQLObject or Elixir).

from sqlalchemy.ext.declarative import declarative_base

engine = create_engine('sqlite://')
Base = declarative_base(engine)

class SomeClass(Base):
    __tablename__ = 'some_table'
    id = Column('id', Integer, primary_key=True)
    name = Column('name', String(50))

Unicode, demystified.
By far, the most frustrating problems I've ever encountered in Python have been unicode related. I was fortunate enough to catch Kumar McMillan's presentation, "Unicode in Python, Completely Demystified". This presentation helped enlighten many on the concept of unicode, clear up many misconceptions, and explain how to handle it properly in Python. Check out his slides for more details, but the general idea here is to follow these three rules:

His solution to decoding to unicode turns out to be quite elegant compared to some nasty try/except UnicodeDecodeError blocks that I have written in the past ;)
def to_unicode_or_bust(obj, encoding='utf-8'):
    if isinstance(obj, basestring):
        if not isinstance(obj, unicode):
            obj = unicode(obj, encoding)
    return obj

Later that night I went and shined some light on some dark corners of certain projects that I've been working on to try and handle unicode the Right Way.

Grassyknoll
After the code sprints, I got a chance to see these guys show off their hard work. grassyknoll is a search engine written in Python. With the ability to handle multiple backends, frontends, and wire formats, grassyknoll has a ton of potential to revolutionize the open source search engine. There has been recent talk in Fedora land about what kind of search engine to use, and I think grassyknoll is definitley a viable option.

Packaging BOF
Toshio, Spot, and I attended a Packaging BOF where we discussed our experiences with distutils and setuptools with a bunch of people from various companies and distros. This then sparked discussions on python-dev and the distutils-sig mailing lists. You can also find the details of the BOF session on the Python wiki. There is definitely a lot of energy behind this, so hopefully we'll see some good changes in setuptools in the near future that will make our lives as distro packagers much easier :)

Orbited
Orbited is an HTTP daemon that is optimized for long-lasting comet connections. This allows you to write real-time web applications with ease. For example, embeding an irc channel anywhere:

You can also use orbited as a WSGI server! Toshio did some brief benchmarking of of CherryPy{2,3}, Paste, and Orbited WSGI servers, and orbited seemed to be the clear winner in all scenerios. There is a good chance that we will be using orbited to handle our comet widgets within MyFedora :)

Code Sprints
I stayed the entire time for the code sprints, and mainly focused on TurboGears hacking. This is what I ended up working on:

Want to read more blog posts about PyCon 2008? You can find links to lots of PyCon related posts here and on Planet Python.


posted at: 22:05 | link | Tags: , | 1 comments

Mon, 11 Feb 2008

Creating live Fedora USB sticks, in Windows!

Last weekend I sat down and developed the liveusb-creator, a tool for creating live Fedora USB sticks from Windows. It will automatically detect all removable drives, find your ISO, extract it to your USB key, modify the syslinux.cfg, and install the bootloader. Technical details aside for a moment, the end-user workflow turns out to be something like this:

At the moment it is a simple a console application that asks the user if it has any questions (by default it won't), and then gets the job done. So shortly after announcing this tool, I started throwing together a graphical interface using PyGTK. While I was doing this, Kushal Das was on the other side of the planet working on a PyQT version :) It turns out his code is much cleaner than my gtk implementation, so I went ahead and committed it. Furthermore, I'm pretty excited to get a chance to play with The Other widget toolkit for once ;)

So, detecting removable devices and such is *trivial* in Windows using the powerfully-undocumented win32api Python module (after playing a few rounds of "match the return code to the enum", of course). Ideally, I want this tool to work in both Linux and Windows, so I redesigned the code quite a bit, broke it out into various modules, and recently started working on the Linux side of things. At this point in time, there is now code that detects removable drives using dbus and HAL. I still have a bunch of sanity checking and other bits to write, but progress is definitely being made. In the mean time for Linux users, see the USBHowTo for creating a live USB stick using the livecd-iso-to-disk tool.

If you're interested in helping out with the liveusb-creator, you can get ahold of the source code using git:

git clone git://git.fedorahosted.org/git/liveusb-creator.git
If you encounter any problems, please create a new ticket at the liveusb-creator trac.


posted at: 14:38 | link | Tags: , , , | 132 comments

Tue, 05 Feb 2008

F8 -> F9 Alpha Live Diff

Fedora 9 Alpha is scheduled to be released today! Not only did I spin the live bits for this alpha, I also generated some statistics as to what changed in this release since Fedora 8. Here are the livecd diffs for all of our spins.

Below are the details as to what changed in our default GNOME desktop spin since Fedora 8.
--- F8-Live-i686-20080204.0.iso (694M)
+++ F9-Alpha-i686-20080129.0.iso (698M)
+ new package libgdiplus-devel: 8584
+ new package xorg-x11-server-common: 38863
+ new package PolicyKit-gnome-libs: 40188
+ new package kerneloops: 52570
+ new package swfdec-gtk: 55786
+ new package gnome-panel-libs: 56936
+ new package swfdec-mozilla: 75911
+ new package libconfig: 120055
+ new package obex-data-server: 136538
+ new package at-spi-python: 170868
+ new package ncurses-base: 176949
+ new package pixman: 209556
+ new package scim-python: 247730
+ new package libcurl: 258148
+ new package libggz: 289477
+ new package hfsutils: 362228
+ new package libmtp: 398952
+ new package xorg-x11-drv-openchrome: 415754
+ new package ggz-client-libs: 434830
+ new package samyak-fonts: 457144
+ new package perl-Date-Manip: 458629
+ new package libtasn1: 466849
+ new package python-crypto: 571535
+ new package elilo: 613010
+ new package gfs2-utils: 650707
+ new package ncurses-libs: 668620
+ new package swfdec: 958169
+ new package reiserfs-utils: 1022402
+ new package iscsi-initiator-utils: 1138529
+ new package jfsutils: 1138726
+ new package gvfs: 1700127
+ new package totem-pl-parser: 2627745
+ new package xfsprogs: 3408051
+ new package VLGothic-fonts: 3831447
+ new package VLGothic-fonts-proportional: 3831790
+ new package gnome-settings-daemon: 6218660
+ new package mesa-libOSMesa: 7248256
+ new package scim-python-chinese: 7621164
+ new package libgweather: 14592282
+ new package dejavu-fonts: 15593008
+ new package vim-common: 16294034
+ new package xulrunner: 24481155
+ crontabs grew 144 bytes (6.83%) (2107->2251)
+ libopenraw-gnome grew 348 bytes (7.94%) (4384->4732)
+ xorg-x11-drv-fbdev grew 380 bytes (1.84%) (20597->20977)
+ irqbalance grew 400 bytes (1.85%) (21595->21995)
+ m17n-contrib grew 469 bytes (1.28%) (36757->37226)
+ pam_ccreds grew 497 bytes (1.49%) (33428->33925)
+ smolt-firstboot grew 655 bytes (6.01%) (10893->11548)
+ pcsc-lite-libs grew 848 bytes (2.44%) (34696->35544)
+ dbus-x11 grew 884 bytes (3.63%) (24353->25237)
+ numactl grew 896 bytes (1.00%) (89239->90135)
+ gnome-bluetooth-libs grew 1278 bytes (1.02%) (124866->126144)
+ xorg-x11-drv-evdev grew 1445 bytes (4.05%) (35642->37087)
+ m17n-db-hindi grew 1717 bytes (21.74%) (7899->9616)
+ sysreport grew 1783 bytes (5.30%) (33620->35403)
+ libpciaccess grew 1796 bytes (7.21%) (24901->26697)
+ sg3_utils-libs grew 2156 bytes (1.97%) (109392->111548)
+ pciutils grew 2464 bytes (1.36%) (180975->183439)
+ setroubleshoot grew 2541 bytes (1.11%) (229578->232119)
+ gnome-keyring-pam grew 2556 bytes (8.89%) (28760->31316)
+ libcap grew 2618 bytes (5.79%) (45230->47848)
+ apr grew 2823 bytes (1.04%) (271801->274624)
+ bc grew 2861 bytes (1.50%) (190964->193825)
+ libsepol grew 2992 bytes (1.33%) (224692->227684)
+ lohit-fonts-telugu grew 3100 bytes (1.78%) (174487->177587)
+ e2fsprogs-libs grew 3332 bytes (1.33%) (250016->253348)
+ device-mapper-libs grew 3680 bytes (4.25%) (86516->90196)
+ glx-utils grew 3704 bytes (10.98%) (33736->37440)
+ scim-chewing grew 4072 bytes (3.22%) (126383->130455)
+ dbus-libs grew 4100 bytes (1.63%) (251944->256044)
+ nash grew 4128 bytes (1.74%) (237698->241826)
+ libjpeg grew 4420 bytes (1.61%) (275021->279441)
+ authconfig-gtk grew 4808 bytes (2.75%) (175143->179951)
+ mkinitrd grew 4854 bytes (4.84%) (100334->105188)
+ linuxwacom grew 5518 bytes (1.10%) (502293->507811)
+ desktop-file-utils grew 5523 bytes (4.50%) (122601->128124)
+ gnome-python2-gnomeprint grew 5547 bytes (1.27%) (437641->443188)
+ bluez-utils-alsa grew 5856 bytes (13.67%) (42824->48680)
+ m17n-contrib-telugu grew 6114 bytes (28.08%) (21776->27890)
+ rsyslog grew 6922 bytes (1.45%) (477587->484509)
+ ustr grew 7531 bytes (3.12%) (241610->249141)
+ rhpxl grew 7783 bytes (2.36%) (329907->337690)
+ xorg-x11-drv-mga grew 8319 bytes (4.91%) (169473->177792)
+ taglib grew 8368 bytes (1.71%) (489415->497783)
+ gtk-nodoka-engine grew 8948 bytes (9.32%) (96057->105005)
+ nscd grew 9484 bytes (6.73%) (140911->150395)
+ exempi grew 9692 bytes (1.39%) (698782->708474)
+ gnome-menus grew 9841 bytes (1.57%) (626493->636334)
+ dbus-glib grew 9970 bytes (2.10%) (473790->483760)
+ libdhcp6client grew 10524 bytes (6.30%) (166956->177480)
+ openldap grew 10658 bytes (1.76%) (604986->615644)
+ nss_ldap grew 12224 bytes (2.17%) (562402->574626)
+ dmidecode grew 14466 bytes (10.46%) (138266->152732)
+ NetworkManager-vpnc grew 14477 bytes (4.58%) (316033->330510)
+ system-config-rootpassword grew 14962 bytes (16.07%) (93118->108080)
+ gstreamer-python grew 15266 bytes (1.64%) (933175->948441)
+ rarian grew 15824 bytes (4.99%) (316947->332771)
+ at-spi grew 16072 bytes (2.38%) (674624->690696)
+ isomd5sum grew 17146 bytes (36.61%) (46840->63986)
+ usbutils grew 17192 bytes (19.31%) (89044->106236)
+ acl grew 17875 bytes (11.97%) (149393->167268)
+ hicolor-icon-theme grew 17992 bytes (79.30%) (22688->40680)
+ gnome-python2-desktop grew 18187 bytes (7.44%) (244527->262714)
+ libdhcp grew 19318 bytes (14.23%) (135727->155045)
+ which grew 20480 bytes (65.05%) (31485->51965)
+ NetworkManager-gnome grew 20604 bytes (2.90%) (710665->731269)
+ pam_krb5 grew 20943 bytes (8.06%) (259736->280679)
+ system-config-language grew 21674 bytes (8.55%) (253576->275250)
+ libxcb grew 22328 bytes (5.40%) (413804->436132)
+ bluez-utils grew 22572 bytes (1.76%) (1280277->1302849)
+ pygtksourceview grew 23100 bytes (36.06%) (64064->87164)
+ libgpg-error grew 23525 bytes (12.14%) (193728->217253)
+ glibmm24 grew 24411 bytes (5.25%) (465396->489807)
+ fribidi grew 24912 bytes (17.19%) (144894->169806)
+ gmime grew 24916 bytes (4.24%) (587824->612740)
+ libuser grew 25215 bytes (1.56%) (1616562->1641777)
+ httpd grew 28595 bytes (1.12%) (2551734->2580329)
+ m17n-lib grew 30893 bytes (10.14%) (304750->335643)
+ rhpl grew 31037 bytes (3.99%) (778235->809272)
+ libdhcp4client grew 32772 bytes (6.57%) (499144->531916)
+ bind-utils grew 33408 bytes (10.87%) (307362->340770)
+ NetworkManager grew 33657 bytes (1.42%) (2377366->2411023)
+ dbus-python grew 36266 bytes (5.11%) (710089->746355)
+ gnome-mag grew 36431 bytes (7.21%) (504936->541367)
+ libXpm grew 37746 bytes (52.09%) (72467->110213)
+ libgnomekbd grew 38042 bytes (6.68%) (569521->607563)
+ pm-utils grew 39200 bytes (117.36%) (33402->72602)
+ nautilus-sendto grew 42489 bytes (16.21%) (262118->304607)
+ dhclient grew 42699 bytes (8.59%) (497288->539987)
+ gtksourceview2 grew 42895 bytes (2.00%) (2148753->2191648)
+ krb5-libs grew 45052 bytes (2.96%) (1522532->1567584)
+ system-config-printer grew 47675 bytes (5.03%) (948043->995718)
+ gnutls grew 58282 bytes (5.99%) (972804->1031086)
+ bluez-gnome grew 60576 bytes (22.56%) (268531->329107)
+ mono-data grew 61605 bytes (1.21%) (5087435->5149040)
+ libwnck grew 62234 bytes (5.42%) (1148126->1210360)
+ gtk2-engines grew 63679 bytes (6.09%) (1045391->1109070)
+ system-config-users grew 64047 bytes (4.40%) (1455495->1519542)
+ gnokii grew 68723 bytes (4.36%) (1575916->1644639)
+ rsync grew 73058 bytes (18.04%) (404896->477954)
+ hal-info grew 75029 bytes (20.94%) (358305->433334)
+ mesa-libGLU grew 77812 bytes (17.12%) (454428->532240)
+ mdadm grew 83417 bytes (4.79%) (1743098->1826515)
+ shared-mime-info grew 85852 bytes (9.51%) (902332->988184)
+ compiz-gnome grew 87904 bytes (7.16%) (1227682->1315586)
+ PolicyKit-gnome grew 89123 bytes (126.49%) (70457->159580)
+ GConf2 grew 89585 bytes (1.68%) (5342705->5432290)
+ dhcpv6-client grew 94965 bytes (54.70%) (173599->268564)
+ system-config-firewall grew 103528 bytes (4.50%) (2300495->2404023)
+ ntfs-3g grew 107185 bytes (36.31%) (295187->402372)
+ f-spot grew 110065 bytes (1.44%) (7621883->7731948)
+ PolicyKit grew 121200 bytes (71.93%) (168495->289695)
+ gnupg grew 126829 bytes (2.62%) (4841029->4967858)
+ libgcrypt grew 132376 bytes (38.24%) (346204->478580)
+ libopenraw grew 136838 bytes (101.68%) (134583->271421)
+ pykickstart grew 141694 bytes (17.92%) (790784->932478)
+ gnome-python2-gnomevfs grew 142165 bytes (87.24%) (162958->305123)
+ shadow-utils grew 144973 bytes (5.29%) (2739389->2884362)
+ gnome-volume-manager grew 158480 bytes (7.38%) (2146417->2304897)
+ vbetool grew 162208 bytes (139.43%) (116340->278548)
+ openssl grew 166448 bytes (4.81%) (3459831->3626279)
+ libselinux-python grew 171323 bytes (118.46%) (144622->315945)
+ libsilc grew 180620 bytes (17.41%) (1037560->1218180)
+ sound-juicer grew 182617 bytes (5.86%) (3114050->3296667)
+ gnome-system-monitor grew 186353 bytes (3.55%) (5244840->5431193)
+ gdb grew 193437 bytes (3.11%) (6228176->6421613)
+ selinux-policy-devel grew 206105 bytes (3.72%) (5545358->5751463)
+ evolution-data-server grew 208724 bytes (1.89%) (11029422->11238146)
+ PyOpenGL grew 213779 bytes (4.86%) (4398157->4611936)
+ tomboy grew 218900 bytes (3.63%) (6022535->6241435)
+ parted grew 223507 bytes (15.16%) (1474368->1697875)
+ orca grew 231344 bytes (4.05%) (5718621->5949965)
+ util-linux-ng grew 245524 bytes (5.17%) (4749959->4995483)
+ selinux-policy grew 268202 bytes (3.47%) (7731786->7999988)
+ iso-codes grew 269192 bytes (4.80%) (5605136->5874328)
+ system-config-date grew 282500 bytes (10.09%) (2798572->3081072)
+ xorg-x11-drv-ati grew 285328 bytes (35.75%) (798151->1083479)
+ eog grew 292326 bytes (7.82%) (3740424->4032750)
+ dbus grew 299134 bytes (58.75%) (509123->808257)
+ totem grew 321458 bytes (5.87%) (5476956->5798414)
+ gnome-keyring grew 333541 bytes (32.87%) (1014819->1348360)
+ glibc grew 347221 bytes (2.59%) (13402107->13749328)
+ sqlite grew 358672 bytes (76.12%) (471170->829842)
+ setroubleshoot-server grew 363273 bytes (22.18%) (1637732->2001005)
+ bind-libs grew 389872 bytes (17.27%) (2258064->2647936)
+ gcalctool grew 496578 bytes (10.21%) (4862745->5359323)
+ gnome-panel grew 509960 bytes (4.35%) (11714901->12224861)
+ ghostscript grew 534784 bytes (1.87%) (28646835->29181619)
+ rhythmbox grew 678314 bytes (6.41%) (10582223->11260537)
+ mono-core grew 686301 bytes (2.01%) (34154946->34841247)
+ nautilus grew 693197 bytes (5.04%) (13751211->14444408)
+ mono-winforms grew 729754 bytes (7.47%) (9765822->10495576)
+ totem-mozplugin grew 770229 bytes (136.17%) (565632->1335861)
+ mono-web grew 797665 bytes (9.85%) (8097242->8894907)
+ glib2 grew 849381 bytes (29.07%) (2922173->3771554)
+ gnome-power-manager grew 934030 bytes (8.33%) (11214535->12148565)
+ nss grew 937664 bytes (44.33%) (2114975->3052639)
+ libsmbclient grew 1191360 bytes (50.53%) (2357736->3549096)
+ gnome-games grew 1373569 bytes (4.65%) (29510057->30883626)
+ kernel grew 2170562 bytes (4.60%) (47161413->49331975)
+ anaconda grew 2566351 bytes (17.76%) (14448198->17014549)
- krb5-auth-dialog shrunk 1 bytes (53674->53673)
- libtirpc shrunk 1 bytes (150301->150300)
- gmime-sharp shrunk 6 bytes (197336->197330)
- gzip shrunk 6 bytes (219689->219683)
- gedit shrunk 8 bytes (13487572->13487564)
- readline shrunk 8 bytes (350214->350206)
- system-config-network shrunk 8 bytes (1905298->1905290)
- perl shrunk 14 bytes (31645884->31645870)
- ntsysv shrunk 16 bytes (22156->22140)
- xdg-utils shrunk 18 bytes (176553->176535)
- python-pyblock shrunk 22 bytes (175969->175947)
- pavucontrol shrunk 23 bytes (169857->169834)
- logrotate shrunk 32 bytes (77454->77422)
- libXfont shrunk 32 bytes (456948->456916)
- gnome-python2-canvas shrunk 32 bytes (48902->48870)
- xorg-x11-drv-vmmouse shrunk 32 bytes (16364->16332)
- mono-data-sqlite shrunk 33 bytes (457296->457263)
- gnome-pilot shrunk 36 bytes (1930958->1930922)
- libflashsupport shrunk 40 bytes (11044->11004)
- nautilus-extensions shrunk 48 bytes (31308->31260)
- pulseaudio-utils shrunk 56 bytes (234499->234443)
- libselinux shrunk 61 bytes (148311->148250)
- gimp shrunk 64 bytes (38423455->38423391)
- libXrender shrunk 64 bytes (47254->47190)
- paps shrunk 64 bytes (51462->51398)
- libxkbfile shrunk 96 bytes (146358->146262)
- librsvg2 shrunk 120 bytes (337750->337630)
- pulseaudio-libs shrunk 128 bytes (343843->343715)
- isdn4k-utils shrunk 144 bytes (9789025->9788881)
- less shrunk 171 bytes (176124->175953)
- pulseaudio shrunk 192 bytes (926686->926494)
- xorg-x11-drv-keyboard shrunk 256 bytes (26608->26352)
- anacron shrunk 274 bytes (56515->56241)
- libgtop2 shrunk 320 bytes (341012->340692)
- xorg-x11-drv-cirrus shrunk 355 bytes (77986->77631)
- gtkspell shrunk 400 bytes (56779->56379)
- pulseaudio-core-libs shrunk 416 bytes (439696->439280)
- liberation-fonts shrunk 444 bytes (1865074->1864630)
- libpcap shrunk 485 bytes (261897->261412)
- nspluginwrapper shrunk 509 bytes (311525->311016)
- acpid shrunk 542 bytes (61235->60693)
- udev shrunk 601 bytes (654430->653829)
- setroubleshoot-plugins shrunk 689 bytes (2422617->2421928)
- python-numeric shrunk 1040 bytes (1722779->1721739)
- file-libs shrunk 1181 bytes (1669772->1668591)
- xorg-x11-drv-vesa shrunk 1208 bytes (26099->24891)
- system-config-keyboard shrunk 1259 bytes (182829->181570)
- fedora-release shrunk 1450 bytes (46680->45230)
- gnome-spell shrunk 1568 bytes (377889->376321)
- gparted shrunk 1600 bytes (1569813->1568213)
- nspr shrunk 1856 bytes (248628->246772)
- dvd+rw-tools shrunk 1860 bytes (283930->282070)
- libdrm shrunk 1878 bytes (39456->37578)
- at shrunk 1892 bytes (83059->81167)
- synaptics shrunk 2072 bytes (113498->111426)
- m17n-contrib-hindi shrunk 2162 bytes (16757->14595)
- pango shrunk 3747 bytes (862572->858825)
- system-config-services shrunk 3768 bytes (561578->557810)
- libcdio shrunk 3961 bytes (555394->551433)
- audit-libs shrunk 4096 bytes (130447->126351)
- atmel-firmware shrunk 4458 bytes (732612->728154)
- fontconfig shrunk 4805 bytes (374336->369531)
- checkpolicy shrunk 6752 bytes (510825->504073)
- ppp shrunk 8123 bytes (841674->833551)
- ntp shrunk 8426 bytes (2652615->2644189)
- procps shrunk 11625 bytes (374451->362826)
- iproute shrunk 11660 bytes (2152986->2141326)
- gnome-bluetooth shrunk 12179 bytes (545675->533496)
- smolt shrunk 13811 bytes (635219->621408)
- b43-fwcutter shrunk 17375 bytes (37123->19748)
- im-chooser shrunk 19198 bytes (208789->189591)
- evolution shrunk 30364 bytes (38160208->38129844)
- ntfsprogs shrunk 33559 bytes (1148797->1115238)
- zip shrunk 33692 bytes (302492->268800)
- vixie-cron shrunk 46771 bytes (673502->626731)
- compiz shrunk 57822 bytes (1846352->1788530)
- cups shrunk 73290 bytes (10355510->10282220)
- system-config-printer-libs shrunk 88138 bytes (2438676->2350538)
- eel2 shrunk 90562 bytes (848996->758434)
- yelp shrunk 99732 bytes (2559882->2460150)
- xorg-x11-drv-i810 shrunk 249946 bytes (655965->406019)
- gtk2 shrunk 308952 bytes (20696815->20387863)
- fedora-release-notes shrunk 329470 bytes (12271152->11941682)
- firstboot shrunk 356968 bytes (783521->426553)
- cairo shrunk 423604 bytes (1623706->1200102)
- control-center shrunk 492635 bytes (8945618->8452983)
- gnome-media shrunk 782217 bytes (4946507->4164290)
- e2fsprogs shrunk 904410 bytes (2366733->1462323)
- device-mapper shrunk 917957 bytes (1027430->109473)
- gnome-themes shrunk 1007647 bytes (4994173->3986526)
- xkeyboard-config shrunk 1188506 bytes (3043679->1855173)
- lvm2 shrunk 1386888 bytes (2186994->800106)
- xorg-x11-server-Xorg shrunk 1410978 bytes (7431876->6020898)
- hwdata shrunk 1768168 bytes (2833960->1065792)
- python shrunk 1870258 bytes (18499208->16628950)
- xorg-x11-fonts-Type1 shrunk 1919814 bytes (2803806->883992)
- ncurses shrunk 2311863 bytes (2562051->250188)
- selinux-policy-targeted shrunk 3623635 bytes (27306881->23683246)
- gdm shrunk 6450348 bytes (14040436->7590088)
- gnome-applets shrunk 8005406 bytes (26478262->18472856)
- firefox shrunk 39666395 bytes (42009290->2342895)
- removed package fonts-gujarati: 0
- removed package fonts-korean: 0
- removed package scim-lang-kannada: 0
- removed package fonts-arabic: 0
- removed package fonts-punjabi: 0
- removed package fonts-oriya: 0
- removed package fonts-chinese: 0
- removed package scim-lang-tibetan: 0
- removed package fonts-kannada: 0
- removed package fonts-bengali: 0
- removed package fonts-hebrew: 0
- removed package fonts-hindi: 0
- removed package scim-lang-assamese: 0
- removed package fonts-sinhala: 0
- removed package scim-lang-sinhalese: 0
- removed package fonts-tamil: 0
- removed package fonts-telugu: 0
- removed package fonts-malayalam: 0
- removed package m17n-contrib-urdu: 3608
- removed package m17n-db-assamese: 6079
- removed package m17n-db-kannada: 7749
- removed package m17n-contrib-kannada: 8717
- removed package m17n-contrib-sinhala: 11327
- removed package m17n-contrib-assamese: 12581
- removed package m17n-db-sinhala: 14228
- removed package m17n-db-tibetan: 15214
- removed package xorg-x11-drv-ark: 18888
- removed package xorg-x11-drv-tseng: 52907
- removed package xorg-x11-drv-s3: 58401
- removed package scim-sinhala: 66881
- removed package totem-plparser: 70428
- removed package libbeagle: 94156
- removed package xorg-x11-drv-avivo: 107204
- removed package xorg-x11-drv-chips: 154533
- removed package lohit-fonts-kannada: 210687
- removed package fuse: 216231
- removed package beecrypt: 242015
- removed package lklug-fonts: 333507
- removed package xorg-x11-drv-via: 363192
- removed package xorg-x11-fonts-ethiopic: 437981
- removed package SDL: 495206
- removed package curl: 514238
- removed package firstboot-tui: 653472
- removed package xorg-x11-fonts-truetype: 909077
- removed package db4o: 1414265
- removed package jomolhari-fonts: 2293163
- removed package pwlib: 2423701
- removed package evince: 3452782
- removed package aspell-en: 3567971
- removed package tibetan-machine-uni-fonts: 4529886
- removed package dejavu-lgc-fonts: 6293390
- removed package opal: 10988583
- removed package ekiga: 13323689
old has 896 packages
new has 885 packages


posted at: 13:23 | link | Tags: , | 0 comments

Wed, 16 Jan 2008

FUDCon

So FUDCon happened over the weekend in Raleigh, North Carolina. It was a great chance to meet a bunch of new people, catch up with some old friends, and kick around in the south for a bit. I was amazed to see such a huge turnout for the first day of the hackfest. It was nice to see a ton of new contributors looking to dive in head first into projects. My goal for the day was to hack on the MyFedora framework, and solidify our architecture and base widget classes, making it easy to create and display your own widgets. It's probably safe to say that we exceeded those expectations.

I sat down with J5, Toshio, and Douglas Warner, fired up a Gobby instance, and started hacking. Thanks to the wonders of distributed source control (git!), TurboGears, and Gobby, we were all able to simultaneously run, commit, and hack on the code. The result of our days work turns out to be a pretty solid architecture for writing, configuring, and displaying reusable Python widgets (based on ToscaWidgets) that can pull from various data sources. For example, writing a widget to display the latest entries in an RSS feed couldn't really be much easier:

class FedoraPeopleWidget(RSSWidget):
    url = 'http://planet.fedoraproject.org/rss20.xml'
    title = 'Fedora People'

The next day during the MyFedora session we got a chance to show off some of the work we did, and get some more ideas from various types of contributors. This project has the potential to make a lot of peoples lives easier, so if you're interested in helping out, grab the code and dive in: $ git clone git://git.fedorahosted.org/git/myfedora.git

Toshio and I gave a session on TurboGears, which seemed to go pretty well. Lots of good discussion and code examples. You can checkout the slides for my presentation here: http://tg.lewk.org.

I was going to be giving a session on bodhi, which we eventually merged with the TurboGears talk. However, the TG session went a lot longer than expected, and bodhi never emerged. So, for those who were interested, you can find my bodhi slides here, and some transcripts from our last virtual fudcon.

The PackageKit session went well too. People definitely were interested, and also had some interesting ideas.

Saturday night was FUDPub, where we had the back room of the Flying Saucer all to ourselves. People kept feeding me drinks, and I didn't complain. Good times :)

Sunday was the second day of the hackfests. I decided to context-switch a bit and get my func on. I wrote a patch that adds a "mem" method to the ProcessModule that returns per-program memory usage from your minion in the format of [[Private, Shared, Total RAM used, Program], ...]. This allows you to do something like,

[lmacken@crow ~]$ sudo func "*" call process mem
on https://tomservo:51234 running process mem ()
[['16.8 MiB', '6.5 MiB', '23.4 MiB', 'Xorg'],
 ['21.7 MiB', '8.3 MiB', '30.1 MiB', 'tomboy'],
 ['33.6 MiB', '2.3 MiB', '35.9 MiB', 'ssh (5)'],
 ['23.2 MiB', '14.3 MiB', '37.5 MiB', 'deskbar-applet'],
 ['139.9 MiB', '9.9 MiB', '149.8 MiB', 'firefox-bin']]

I also discussed a potential TurboGears FuncWeb implementation with Michael DeHaan. I got a chance to create create a skeleton project, and jot some ideas down. Just as I was about to dive in, I got a phone call notifying me of my flight cancellation. I then had to immediately sketch off to catch a 2:20pm flight and head back to Boston.

Last night I got a little bit A.D.D. and re-wrote some chunks of the func minion module_loader/server to make writing func modules a lot easier.

So, the moral of the story is: FUDCon rocks. Feeding large quantities of geeks caffeine, beer, and barbeque can result in amazing things.

Of course there are no ups without downs, so I was stuck dealing with a nasty cold most of the time there, and my laptop power adapter melted as well. Thankfully, both of those issue have since been resolved :)


posted at: 17:21 | link | Tags: , , , | 0 comments

Wed, 19 Dec 2007

TurboFlot 0.0.1

In an effort to clean up bodhi's metrics code a bit, I wrote a TurboFlot plugin that allows you to wield the jQuery plugin flot inside of TurboGears applications. The code is quite trivial -- it's essentially just a TurboGears JSON proxy to the jQuery flot plugin. Breaking this code out into it's own widget makes it really easy to generate shiny graphs in a Pythonic fashon, without having to write a line of javascript.

Check out the README to see the code for the example above.

To use TurboFlot in your own application, you just pass your data and graph options to the widget, and then throw it up to your template. Read the flot API documentation for details on all of the arguments. Here is a simple usage example:

flot = TurboFlot([
    {
        'data'  : [[0, 3], [4, 8], [8, 5], [9, 13]],
        'lines' : { 'show' : True, 'fill' : True }
    }],
    {
        'grid'  : { 'backgroundColor' : '#fffaff' },
        'yaxis' : { 'max' : '850' }
    }
)
Then, to display the widget in your template, you simply use:
${flot.display()}

The code for the widget itself is pretty simple. It just takes your data and graph options, encodes them as JSON and tosses them at flot.
class TurboFlot(Widget):
    """
        A TurboGears Flot Widget.
    """
    template = """
      <div xmlns:py="http://purl.org/kid/ns#" id="turboflot"
           style="width:${width};height:${height};">
        <script>
          $.plot($("#turboflot"), ${data}, ${options});
        </script>
      </div>
    """
    params = ["data", "options", "height", "width"]
    javascript = [JSLink('turboflot', 'excanvas.js'),
                  JSLink("turboflot", "jquery.js"),
                  JSLink("turboflot", "jquery.flot.js")]

    def __init__(self, data, options={}, height="300px", width="600px"):
        self.data = simplejson.dumps(data)
        self.options = simplejson.dumps(options)
        self.height = height
        self.width = width

You can download the latest releases from the Python Package Index:

http://pypi.python.org/pypi/TurboFlot
Or you can grab my latest development tree out of mercurial:
http://hg.lewk.org/TurboFlot
As always, patches are welcome :)


posted at: 20:21 | link | Tags: , , , | 4 comments

Sun, 09 Dec 2007

Fedora update metrics

Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.

I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.

When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.


posted at: 01:05 | link | Tags: , , , , , | 2 comments

Tue, 13 Nov 2007

bodhi command-line client

The bodhi-client package should be making its way to an updates-testing repository near you! Not only does this command-line tool give developers easier access to bodhi, but also provides some new features to help people get more involved with testing updates and providing useful feedback.

I wrote up some documentation on various usage examples of the tool, which can be found on the bodhi wiki. I also submitted a Makefile.common patch that, once applied, will allow you to run `make update` from your package branch. This will drop you into a new update template, and will then submit your update straight to bodhi.

Some noteworthy features in the bodhi-client, aside from the normal bodhi functionality:

I also upgraded our production bodhi instance yesterday, which pulled in a ton of bugfixes and some new features, such as:

As always, patches/questions/criticisms/comments are welcome. You can file tickets in the usual place. Happy hacking!


posted at: 15:42 | link | Tags: , , | 2 comments

Mon, 01 Oct 2007

Use your Nose!

Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.

So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.

For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update.

import urllib, cherrypy
from turbogears import update_config, database, testutil, url

update_config(configfile='dev.cfg', modulename='bodhi.config')
database.set_db_uri("sqlite:///:memory:")

class TestControllers(testutil.DBTest):

    def test_unauthenticated_update(self):
        params = {
                'builds'  : 'TurboGears-1.0.2.2-2.fc7',
                'release' : 'Fedora 7',
                'type'    : 'enhancement',
                'bugs'    : '1234 5678',
                'cves'    : 'CVE-2020-0001',
                'notes'   : 'foobar'
        }
        path = url('/save?' + urllib.urlencode(params))
        testutil.createRequest(path, method='POST')
        assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'.
[lmacken@tomservo bodhi]$ nosetests
.................................
----------------------------------------------------------------------
Ran 33 tests in 16.798s

OK
Now, for the fun part. Nose comes equipped with a profiling plugin that will profile your test cases using Python's hotshot module. So, I went ahead and added a 'profile' target to bodhi's Makefile:
profile:
    nosetests --with-profile --profile-stats-file=nose.prof
    python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
Now, typing 'make profile' will execute and profile all of our unit tests, and spit out the top 20 method calls -- ordered by internal time and call count.
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile --profile-stats-file=nose.prof
.................................
----------------------------------------------------------------------
Ran 33 tests in 42.878s

OK
python -c "import hotshot.stats ; stats = hotshot.stats.load('nose.prof') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
         800986 function calls (702850 primitive calls) in 42.878 CPU seconds

   Ordered by: internal time, call count
   List reduced from 3815 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       14   13.675    0.977   13.675    0.977 /usr/lib/python2.5/socket.py:71(ssl)
       31   10.683    0.345   10.683    0.345 /usr/lib/python2.5/httplib.py:994(_read)
2478/2429    9.297    0.004    9.677    0.004 :1()
        1    0.604    0.604    0.604    0.604 /usr/lib/python2.5/commands.py:50(getstatusoutput)
     2999    0.536    0.000    0.539    0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/sqliteconnection.py:177(_executeRetry)
   105899    0.448    0.000    0.773    0.000 Modules/pyexpat.c:871(Default)
       60    0.327    0.005    1.102    0.018 /usr/lib/python2.5/site-packages/kid/parser.py:343(_buildForeign)
   105899    0.325    0.000    0.325    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:452(_default)
     3396    0.280    0.000    0.420    0.000 /usr/lib/python2.5/site-packages/cherrypy/config.py:107(get)
     2965    0.263    0.000    0.263    0.000 /usr/lib/python2.5/logging/__init__.py:364(formatTime)
44964/6587    0.238    0.000    0.252    0.000 /usr/lib/python2.5/site-packages/kid/parser.py:156(_pull)
       60    0.116    0.002    0.116    0.002 /usr/lib/python2.5/site-packages/kid/compiler.py:38(py_compile)
     8127    0.114    0.000    0.114    0.000 /usr/lib/python2.5/site-packages/cherrypy/_cputil.py:311(lower_to_camel)
     8982    0.110    0.000    0.137    0.000 /usr/lib/python2.5/site-packages/sqlobject/dbconnection.py:902(__getattr__)
13740/4044    0.108    0.000    2.176    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:209(_coalesce)
24353/4026    0.107    0.000    2.143    0.001 /usr/lib/python2.5/site-packages/kid/parser.py:174(_track)
     3170    0.093    0.000    0.398    0.000 /usr/lib/python2.5/logging/__init__.py:405(format)
        1    0.082    0.082    0.082    0.082 /usr/lib/python2.5/site-packages/rpm/__init__.py:5()
     4777    0.081    0.000    1.320    0.000 /usr/lib/python2.5/site-packages/kid/serialization.py:564(generate)
  759/176    0.074    0.000    0.210    0.001 /usr/lib/python2.5/sre_parse.py:385(_parse)


posted at: 14:40 | link | Tags: , , , , | 8 comments

Sat, 15 Sep 2007

bodhi 0.2.0

I'm pleased to announce that bodhi 0.2.0 has been released and deployed. This release has fixed a ton of issues and introduces many new features, such as:

This release introduces many database changes from the previous version, so it will be much easier to jump back into the release-early-release often cycle.

Soon to come:

I would also like to thank Till Maas and Tim Lauridsen for taking the time to help out and do some great work.

There is still much to be done with bodhi, so if you're interested in helping out, you can setup a local bodhi development playground with just a few commands and dive in.

As always, please file any bugs or enhancement requests here.


posted at: 15:44 | link | Tags: , , | 0 comments

Sat, 19 May 2007

Security LiveCD

So last week I created an initial version of a potential Fedora Security LiveCD spin. The goal is to provide a fully functional livecd based on Fedora for use in security auditing, penetration testing, and forensics. I created it as a bonus project for my Security Auditing class (instead of following the 5-pages of instructions on how to create a Gentoo livecd that she handed out (mad props to davidz for creating an amazing LiveCD tool)), but it has the potential to be extremely useful and also help increase the number and quality of Fedora's security tools. I threw in all of the tools I could find that already exist in Fedora, but I'm sure I'm missing a bunch, so feel free to send patches or suggestions. I also added a Wishlist of packages that I would eventually like to see make their way in Fedora, after the core->extras merge reviews are done.

I would eventually like to see Fedora offer a LiveCD that puts all of the existing linux security livecds to shame. We have quite a ways to go, but this is a start. I'm taking a computer forensics class next quarter, so I will be expanding it to fit the needs of our class as well.


posted at: 19:15 | link | Tags: , , , , | 0 comments

Dealing with multiple email addresses in mutt

I recently watched Bram Moolenaar's "7 Habits for Effective Text Editing 2.0" Google Tech Talk presentation, which inspired me to dust off the old vimrc and poke around for a bit. I tend to live most of my life inside of vim, which is actually a pretty kickass place to live (I even started working on a life.vim script to replace my [excessive] tomboy usage, and add a bit of GTD sauce).

I use mutt for my email accounts, and recently made some modifications to my vimrc to help automate a task that utilizes far too many of my keystrokes each day: changing the From field of outgoing messages.

let g:addresses = ['lewk csh rit edu', 'lmacken redhat com', 'lmacken fedoraproject org']
function NextEmailAddress()
    if !exists("g:email_idx")
        let g:email_idx = 0
    endif
    s@^\(From:.*<\).*>@\=submatch(1) . g:addresses[g:email_idx % len(g:addresses)] . ">"@
    let g:email_idx += 1
endfunction
map <silent> @ :call NextEmailAddress()<CR>

This lets you cycle through your email addresses by pressing '@' when composing a new mail. From here, my messages go through esmtp, which ships them off to the proper SMTP server based on the identity of the mail.

I'd be curious to hear of any other tweaks people have made to their vimrc's to help improve their mutt usage, or of any life-changing hack that saves you tons of keystrokes every day.


posted at: 19:15 | link | Tags: , , , | 1 comments

FUDCon 2007

So FUDCon went down last weekend, and was an absolute blast. My plane landed and I went straight to BU and arrived right while people were voting on sessions. It was great to finally meet and chat with a ton of Fedora hackers that I've been collaborating with for quite some time now.

I caught Bryan Clark's Mugshot session, where he talked about some future ideas and goals for mugshot. A potentially badass idea that he mentioned entails monitoring application usage patterns, which could help everyone get a good idea as to what programs are used and how much. Both great software, and heavily used shitty software, will shine. This will help the community get a jist of what needs work and what should be focused on / improved. But it doesn't stop there, another idea mentioned was to have mugshot maintain basic application settings (email, rss, aim, etc) and allow users to be able to load them from anywhere.

After FUDCon we headed to the FUDPub where everyone indulged as much free beer/food that our stomachs could[n't] handle. I got a chance to chat with many people, one being a guy from Pepper, who showed off his Pepper Pad a bit. I talked with him about how he can get Pepper more involved with Fedora (we have overlaps in terms of infrastructure and such, and can greatly benefit from collaborating (especially on their end, since they are stuck maintaining FC4 which is what their platform is based on)), and get their code out into the public.

Saturday night involved our hotel room filled with 10+ fedora hackers, an OLPC, the buddha machine, and a handle of Tanqueray. Good times ensued (without any ambulance involvement).

During the two days of hacking that followed FUDCon, I definitely didn't write as much code as I had hoped, but I got a chance to have some great conversations with a ton of people.

Much discussion was had about making Fedora a BugBuddy that can help make it trivial to submit non-useless bug reports:

  • anonymous bug creation against components
  • Provide a smolt uuid for complete hardware list of users machine
  • Ability to acquire proper debuginfo packages and reproduce useful traceback
  • Hash traceback (after cleaning it up a bit) and check it against a Traceback DB to see if there are any bugs already opened for that issue (suggested by Peter Jones during the Fedora QA talk).
Ideally, our software should be able to correctly file bugs against itself.

With the emergence of our shiny new hardware profiler Smolt, Fedora users can now anonymously submit hardware information. Not only is acquiring metrics a good thing, but this could potentially give people the ability to report issues by simply simply referencing a unique id. This requires some knowledge in Bugzilla, but if we had our own Bugzilla playground we would be able to experiment with features such as the traceback database, and smolt integration. Our fearless QA leader, Will Woods, began setting up a bugzilla instance for experimenting with the crazy ideas during the hackfest.

What seems like a continued effort to make fedora more of a 'Buddy', we also had some discussions about a CodecBuddy that will essentially detect when you first attempt to play a file that requires an evil proprietary codec. It will then educate the user on the implications of proprietary codecs, provide details on alternative open formats, and let users click on a link to legally acquire this codec, with of course the ability to say "hell no" to sustain your hardcore 100% open source mentality :)

Some other cools stuff that I observed go down was a graphical yum repository configuration tool called repoman that Chris Lumens was working on during the hackfest. The yum guys also got a bunch of stuff done, most importantly adding support to createrepo for inserting sqlite db files into repositories.

Many other technical items were discussed, but we also got a chance to step away from the bits for a little while and really talk about the current state of "collaboration" on the internet, and what we'd like to see Fedora do to push the bar in terms of innovation of social collaboration; but that is for another blog post :)


posted at: 19:15 | link | Tags: , | 0 comments

Fedora Infrastructure Project notifications and IRC hackery

I started a Fedora Infrastructure mugshot group the other day, and hooked it up to the Timelines of some of our Infrastructure projects that are currently under development: smolt, bodhi, pungi, koji, mirrormanager, presto, and beaker. This will keep members of the group notified of code commits, ticket activity, as well as wiki and milestone changes.

Recently, Bugzilla hacker Max Kanat-Alexander setup a new Supybot IRC bot for us in #fedorabot that spits out Bugzilla activity in real time. I quickly threw together a Fedora plugin that will allow us to implement some useful functions to help out users and developers. At the moment, I only created a whoowns command.

<lmacken> buggbot: whoowns nethack
<buggbot> lmacken: nethack is owned by lmacken@redhat.com

With Fedora 7 and a bunch of shiny new infrastructure right around the corner, I'm sure we'll be able to do some fun things with this. If you're interested in hacking on this plugin, you can clone my mercurial repository by doing the following:

$ hg clone http://hg.lewk.org/fedora-supybot


posted at: 19:15 | link | Tags: , , | 0 comments

Mon, 07 May 2007

bloatware

I'm usually not one for unproductive ranting, but I noticed this right after a fresh rawhide bootup...

[lmacken@tomservo ~]$ sudo ./ps_mem.py
 Private  +   Shared  =  RAM used       Program 
[...]
 17.7 MiB +   5.3 MiB =  23.0 MiB       setroubleshootd
 13.4 MiB +  11.2 MiB =  24.6 MiB       /usr/bin/sealer
 51.9 MiB +   2.9 MiB =  54.8 MiB       yum-updatesd

I'm sorry, but most of my machines don't have this kind of memory to spare. Is Python to blame? As far as interpreter overhead, maybe a little bit, or could it be the sheer simplicity and power of the language which is allowing it to be abused?

I'd be interested if anyone has any tips/tricks for slimming down Python daemons.
I'd be thankful if someone did something about it.


posted at: 23:25 | link | Tags: , | 4 comments

Wed, 02 May 2007

Creating a Fedora Security Live USB key

Here is how to easily create a security-distribution based on what will eventually be Fedora 7. This requires that you be running FC7Test* or rawhide, as the livecd-tools are not currently available for FC6.

Prepare the USB key
You may not need to do this for some USB sticks, but I had to remove all partitions on my Cruzer Micro and format the whole thing as vfat to get it to boot. Make sure to change /dev/sdd to your USB device.

# mkfs.vfat -I /dev/sdd
Spin the livecd
# yum install livecd-tools mercurial
$ hg clone http://hg.lewk.org/security-livecd
$ cd security-livecd
# ./spin-livecd.sh

Copy the ISO to your USB stick

# livecd-iso-to-disk Fedora7-SecurityLiveCD.iso /dev/

Interested in helping make the Security LiveCD better? See the SecurityLiveCD wiki for more information.


posted at: 06:24 | link | Tags: , , , | 43 comments

Thu, 15 Feb 2007

bodhi

So I've been spending the majority of my free time recently getting the new Fedora Updates System ready for Fedora 7, which is quickly approaching. Since the code is going to exist in multiple instances for different projects (Fedora, RH, etc), and is already fairly modular and distro-independent, I decided it would be best to give the code a new name, and a new home: Bodhi emerged. Being hardcore into Zen recently, I feel like the name seems to fit the goals of the project nicely.

A Buddhist term for the wisdom by which one attains enlightenment. Bodhi is the opposite of ignorance, the insight into reality which destroys mental afflictions and brings peace.

Previously, the updates system code was in CVS, and bugs were to be filed in OTRS, which ended up being extremely painful to work with. After switching to our new shiny hosting setup, Bodhi is now using Mercurial for source control, and Trac for managing the project milestones, bugs, and documentation. Overall, I've been extremely impressed with Mercurial and Trac so far.

Here are some screenshots of the current development version. More screenshots can be found on the bodhi wiki.

So what's going to change for Fedora developers? Well, currently any [extras] developer can push out updates to their packages for any release with ease (cvs commit && make tag build). If your updated package fixes 4 bugs and 2 CVE's, the end user who is blindly installing this update has no idea. Not only is there no differentiation between bugfix/enhancement/security updates, but none of them go through any sort of QA whatsoever. This is what bodhi is aiming to change.

I've been designing this system completely independent of any Buildsystem (mainly because the unveiling of brew is still churning slowly), so the process of building packages is going to remain the same (for now). Once your packages is built, you will enter it into bodhi via a web form, or [eventually] a command-line tool (Ideally, I'd like to see the process of preparing/testing/releasing a package to be a single point of interaction; either completely from command-line or in web interface). From here, your update will undergo various checks (and will eventually dispatch tests to the beaker test lab). If all goes well, your update will get signed and pushed, all referenced Bugzillas will be updated/closed, and an update notification mail will be sent to the appropriate list.

updates-testing is currently our "testing" repository that developers can choose to push their updates out to before going to 'final'. This is not very appealing for devs, as it requires them to 'Move to final' after a certain amount of time when they feel it is necessary, and the system will nag them if they don't. The updates-testing repo is not widely used, and gives testers no incentive or way to give feedback, other than filing a bug. One of the goals of bodhi is to require all updates to go through the updates-testing process, and provide a simple interface for developers/testers/users to provide positive/negative feedback regarding an update, and also make it trivial to submit bugs about them as well (we threw around some ideas about making a reusable QA feedback widget for this at a recent Fedora QA meeting). After a certain number of positive responses (or after a given length of time with no negative responses), an update will then able to make it's way to the stable updates repository.

Hackery
I've written a ton of code so far, but there is still much work to be done to accomplish all of the goals mentioned above. I layed out a few milestones for bodhi, 1.0 being the minimal functionality needed for the release of Fedora 7. I also added a 1.1 milestone for features that aren't crucial for minimal functionality, but are definitely important, and also a wishlist milestone for features that might be nice to have someday (metrics, rss feeds, etc).

With FUDCon and the Fedora Infrastructure HackFest coming up, I'm encouraging anyone interested to dive in help out in any way they want. If you can't make it out to Boston, just hop on IRC. If you would like to see a feature in bodhi, feel free to add a ticket to any of the milestones.

This system holds much potential for revolutionizing the way Fedora releases evolve, and how developers and testers interact with it. Since this is going to be utilized by all package maintainers, it's important that the system is molded to fit the needs of the developers and testers; so if you have any suggestions/improvements/fixes, don't hesitate: Contribute.


posted at: 03:12 | link | Tags: , , | 13 comments

Wed, 13 Sep 2006

python list generation optimization

Menno Smits recently made some cleanups to yum's RPMDBPackageSack class. In a mail to the yum-devel list, Menno points out that python's list comprehension turns out to be 20% faster than using a for loop while appending items to a list.

A simple benchmark script shows the speed increase claim to be true. Menno's script test two different methods, the first being the standard for/list.append() loop.

l1 = []
for x in xrange(LOOPS):
    if x % 2 == 0:
        l1.append(x)

The second being list comprehension:

l2 = [x for x in xrange(LOOPS) if x % 2 == 0]

Both methods utilize python's xrange, which is a generator object written in pure C, and is faster than the plain old range. Out of curiosity, I went ahead and modified this script to test a few more methods of generating the same list as above.

Using a for loop, and an append function reference:

l3 = []
append = l3.append
for x in xrange(LOOPS):
    if x % 2 == 0:
        append(x)

Using the filter/lambda approach:

l4 = filter(lambda x: x % 2 == 0, xrange(LOOPS))

Using filter and a function reference to the lambda:

x = lambda x: x % 2 == 0
l5 = filter(x, xrange(LOOPS))

The results were actually quite surprising. Using a modified version of Menno's script, I came up with these results after 10 iterations. The percentages are compared to the standard for loop/list.append() method.

for loop/append function reference: 23% faster list comprehension: 36% faster filter/lambda: 40% faster filter/reference to lambda: 57% faster

As it turns out, using the filter/lambda approach seems to shave off ~57% of the usual overhead from the for/list.append() approach.

Although extremely powerful, filter/map/lambdas tend to compact code, thus making it more difficult to read/maintain. This reason alone may cause people to completely stay away from them all together.

Avoiding powerful language features because of 'readability' can be a mistake in my opinion. Any competent programmer should have the ability to make their code understandable to anyone -- yes, it may actually require writing comments. Even the most complex regular expressions can be written so they can be understood easily (see Five Habits for Successful Regular Expressions).

I still tend to agree with Donald Knuth's statement "Premature optimization is the root of all evil". Jumping right into a complex algorithm with maps/filters/lambdas/etc may not be the best idea, and will most likely make it a nightmare to debug down the road.


posted at: 19:11 | link | Tags: , | 7 comments

Tue, 15 Aug 2006

Teaching an old pup some new tricks

Thanks to the previously mentioned core update metadata enhancements that I've been working on lately, we are now able to do some pretty neat stuff with our package updating tools. So last week I cranked out a bunch of code for Fedora's package updater (pup) and it's notification applet (puplet) to utilize this new enhanced metadata and actually provide the user with some useful information.

In order to be able to attach the notification to a status icon, I had to patch libnotify's python bindings to utilize libnotify's notify_notification_attach_to_status_icon function (notify-python-0.1.0-attach_to_status_icon.patch). While poking around with the notify-python code, I tripped over a bug in libnotify that needed to be patched as well (libnotify-0.4.2-status-icon.patch). I hope to kick off patched versions of libnotify and notify-python for rawhide at some point in the near future, and apply my puplet patch as well (puplet-bubbly.patch).

For pup, I whipped up a hacked out GtkTextView to display the new update metadata, when it's available (pup-enhanced-metadata.patch) (the look and feel will most likely change). Due to the current core/extras dichomoty, this update metadata is only generated for core updates until we get a common core/extras update system in place (I've been working on porting the current update system for core over to TurboGears, and make it encompass core/extras/legacy updates). Hopefully by FC7 we will have bridged the core/extras gap completely (optimism++).


posted at: 19:10 | link | Tags: , | 0 comments

Mon, 14 Aug 2006

hdaps + atspi hotness

I started playing around with the hdaps kernel module on my Thinkpad T43 the other day and wrote some code to utilize some of the accelerometer data. With the help of Juozas and AT-SPI, I wrote a little bit of python that allows you to control your mouse based on your laptops position.

#!/usr/bin/env python

import sys, time, math, atspi

SYSFS_POSITION_FILE = "/sys/devices/platform/hdaps/position"

def main():
    theta = 0
    rho = 0
    ev = atspi.EventGenerator()

    while True:
        fd = open(SYSFS_POSITION_FILE, 'r')
        v = fd.readline().strip()
        theta = int(v[1:4])
        rho = int(v[5:8])
        rads = ((theta - 545)/300.0) * math.pi
        ax = math.sin(rads)
        rads = ((rho - 475)/300.0) * math.pi
        ay = math.sin(rads)
        ev.relativeMotion(ax * 10, ay * 10)
        fd.close()
        time.sleep( 0.01 )

if __name__ == '__main__':
    main()

Extremely useless, I know -- but only in it's current state. I'm sure some cool desktop enhancements can be thought up using hdaps. What if you wanted to un-clutter your desktop, so you tilt or laptop to the side and all of your windows slide over to the next virtual desktop? Sounds pretty badass to me.


posted at: 18:50 | link | Tags: | 0 comments

fedora update metadata & fastestmirror changes

Fedora update metadata
I just committed my createrepo update metadata acquisition patch to HEAD which adds support for the -U (--update-info-location) flag. This feature is going to give mirrors the ability to pull in update metadata [example: httpd-2.0.54-10.3.xml] for each corresponding package in the repo. This will allow tools such as pup to know exactly why an update was released, and present the user with all of the details. The next step is to setup the infrastructure for the metadata server. More details to come later.

fastestmirror
I gave fastestmirror some love recently and added support for a 'maxhostfileage' option, which lets you specify how many days before it should update the cached mirror speeds file again.

As the GeoIP package was working it's way through the Fedora Extras review process, Warren noticed that this package could be used to do aid in smarter mirror selection. So once the python bindings hit extras, I'm going to start to play around with fastestmirror + GeoIP integration. Once I get some free time I will probably also look into adding multiple mirror-selection algorithms into fastestmirror: GeoIP, bestmirror (repomd.xml validation), and the original socket-fu technique.

I finally found a home for fastestmirror, in the yum-utils cvs repo. So if you are interested in playing around with the latest version, look no further:

http://devel.linux.duke.edu/cgi-bin/viewcvs.cgi/*checkout*/yum-utils/plugins/fastestmirror/fastestmirror.py?content-type=text%2Fplain


posted at: 18:50 | link | Tags: , , | 0 comments

firefox's new <a ping> vuln^H^H^H^Hfeature

As seen recently on slashdot, one of the latest trunk builds of Firefox contains support for the ping attribute on anchor and area tags (spec). Ideally, this feature will allow websites to contain links such as:

<a href="http://foo" ping="http://bar">biz</a>
After looking into it a bit more, I found a way to trigger a user's Firefox to SYN flood any given host upon clicking a link.
<script language="JavaScript">
    document.write('<a href="http://foobar" ping="');
    for (var i = 0; i < 1000; i++)
        document.write('URI ');
    document.write('">Weeee!</a>');
</script>

When clicked, the link will cause the client to kick 1000 SYN packets over to the specified ping URI without hesitation. This feature is present in Firefox > 1.6a1, and is enabled by default. You can disable this in about:config by flipping off the browser.send_pings boolean. I filed a bug upstream about this issue, and supplied a patch to de-dupe the ping URI list (which might actually not be the best solution to this problem (limiting the number of ping URI's, or kill the pings when the page is left, or the stop button is pressed might be better solutions), but it is still under discussion).


posted at: 18:49 | link | Tags: , | 1 comments

irssi notification script

Having finals to deal with next week, I've been trying to avoid things that tend to waste huge amounts of time, such as email and IRC. So I wrote a little irssi script to pop up notifcation bubbles when people talk to you.

*Update (20090916)*

Years later, and I can't really vouch for this code anymore, as I don't use it. I recommend using it at your own risk, as there could potentially be security implications, since it is unmaintained. However, this script was the catalyst for a variety of improved implementations, such as irssi-libnotify, fnotify, irssi-notify, etc.

notify.pl
##
## Put me in ~/.irssi/scripts, and then execute the following in irssi:
##
##       /load perl
##       /script load notify
##

use strict;
use Irssi;
use vars qw($VERSION %IRSSI);

$VERSION = "0.01";
%IRSSI = (
    authors     => 'Luke Macken',
    contact     => 'lewk@csh.rit.edu',
    name        => 'notify.pl',
    description => 'TODO',
    license     => 'GNU General Public License',
    url         => 'http://www.csh.rit.edu/~lewk/code/irssi-notify',
);

sub notify {
    my ($dest, $text, $stripped) = @_;
    my $server = $dest->{server};

    return if (!$server || !($dest->{level} & MSGLEVEL_HILIGHT));

    $stripped =~ s/[^a-zA-Z0-9 .,!?\@:\>]//g;
    system("notify-send -i gtk-dialog-info -t 5000 '$dest->{target}' '$stripped'");
}

Irssi::signal_add('print text', 'notify');


posted at: 06:43 | link | Tags: , | 5 comments

irssi notification script

Having finals to deal with next week, I've been trying to avoid things that tend to waste huge amounts of time, such as email and IRC. So I wrote a little irssi script to pop up notifcation bubbles when people talk to you.

*Update (20090916)*

Years later, and I can't really vouch for this code anymore, as I don't use it. I recommend using it at your own risk, as there could potentially be security implications, since it is unmaintained. However, this script was the catalyst for a variety of improved implementations, such as irssi-libnotify, fnotify, irssi-notify, etc.

notify.pl
##
## Put me in ~/.irssi/scripts, and then execute the following in irssi:
##
##       /load perl
##       /script load notify
##

use strict;
use Irssi;
use vars qw($VERSION %IRSSI);

$VERSION = "0.01";
%IRSSI = (
    authors     => 'Luke Macken',
    contact     => 'lewk@csh.rit.edu',
    name        => 'notify.pl',
    description => 'TODO',
    license     => 'GNU General Public License',
    url         => 'http://www.csh.rit.edu/~lewk/code/irssi-notify',
);

sub notify {
    my ($dest, $text, $stripped) = @_;
    my $server = $dest->{server};

    return if (!$server || !($dest->{level} & MSGLEVEL_HILIGHT));

    $stripped =~ s/[^a-zA-Z0-9 .,!?\@:\>]//g;
    system("notify-send -i gtk-dialog-info -t 5000 '$dest->{target}' '$stripped'");
}

Irssi::signal_add('print text', 'notify');


posted at: 06:43 | link | Tags: , | 0 comments