Thu, 07 Mar 2013

Keeping your finger on the pulse of the Fedora community

For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout fedmsg.com. Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close.

This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop!

Introducing fedmsg-notify

fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME.

For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e fedmsg@lmacken-redhat.com` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus:

For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer.

The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package.

The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB.

Under the hood

The fedmsg-notify-daemon itself is fairly minimal (see daemon.py). At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue. 

class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer):
    topic = 'org.fedoraproject.*'

    def consume(self, msg): …

The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop.

The API for writing custom filters is dead simple (see filters.py). Here is an example of one: 

class MyPackageFilter(Filter):
    """ Matches messages regarding packages that a given user has ACLs on """
    __description__ = 'Packages that these users maintain'
    __user_entry__ = 'Usernames'

    def __init__(self, settings):
        self.usernames = settings.replace(',', ' ').split()
        self.packages = set()
        reactor.callInThread(self._query_pkgdb)

    def _query_pkgdb(self):
        for username in self.usernames:
            log.info("Querying the PackageDB for %s's packages" % username)
            for pkg in PackageDB().user_packages(username)['pkgs']:
                self.packages.add(pkg['name'])

    def match(self, msg, processor):
        packages = processor.packages(msg)
        for package in self.packages:
            if package in packages:
                return True
The fedmsg-notify-config interface (see gui.py), automatically introspects the filters and populates the Advanced tab with the appropriate labels, switches, and text entries.

Consuming fedmsg over DBus

Let's say you want to write an application that listens to fedmsg, but you don't want to deal with spinning up your own connection, or you're not using Python, etc. For these cases, fedmsg-notify supports relaying messages over DBus. This functionality can be enabled by running `gsettings set org.fedoraproject.fedmsg.notify emit-dbus-signals true`. You can then easily listen for the MessageReceived DBus signal, like so: 
import json, dbus

from gi.repository import GObject
from dbus.mainloop.glib import DBusGMainLoop

def consume(topic, body):
    print(topic)
    print(json.loads(body))

DBusGMainLoop(set_as_default=True)
bus = dbus.SessionBus()
bus.add_signal_receiver(consume, signal_name='MessageReceived',
                        dbus_interface='org.fedoraproject.fedmsg.notify',
                        path='/org/fedoraproject/fedmsg/notify')
loop = GObject.MainLoop()
loop.run()

Contributing

If you're interested in helping out with any layer of the fedmsg stack, hop in #fedora-apps, and fork it on GitHub:

Hop on the bus!


posted at: 17:30 | link | Tags: , , , | 4 comments

Sat, 21 Apr 2012

Wielding the ANU Quantum Random Number Generator

Last week Science Daily published an article that caught my attention titled 'Sounds of Silence' Proving a Hit: World's Fastest Random Number Generator. The tl;dr is that researchers at the ANU ARC Centre of Excellence for Quantum Computation and Communication Technology created a blazing fast random number generator based on quantum fluctuations in a vacuum. Thankfully, these awesome scientists are giving their data away for free, and they even provide a JSON API.

In an effort to make it simple to leverage this data, I created a new project: quantumrandom. It provides a qrandom command-line tool, a Python API, and also a /dev/qrandom Linux character device.

Installing

$ virtualenv env
$ source env/bin/activate
$ pip install quantumrandom

Using the command-line tool

$ qrandom --int --min 5 --max 15
7
$ qrandom --binary
���I�%��e(�1��c��Ee�4�������j�Կ��=�^H�c�u
oq��G��Z�^���fK�0_��h��s�b��AE=�rR~���(�^
+a�a̙�IB�,S�!ꀔd�2H~�X�Z����R��.f
...
$ qrandom --hex
1dc59fde43b5045120453186d45653dd455bd8e6fc7d8c591f0018fa9261ab2835eb210e8
e267cf35a54c02ce2a93b3ec448c4c7aa84fdedb61c7b0d87c9e7acf8e9fdadc8d68bcaa5a
...

Creating /dev/qrandom

quantumrandom comes equipped with a multi-threaded character device in userspace. When read from, this device fires up a bunch of threads to fetch data. Not only can you utilize this as a rng, but you can also feed this data back into your system's entropy pool.

In order to build it's dependencies, you'll need the following packages installed: svn gcc-c++ fuse-devel gccxml libattr-devel. On Fedora 17 and newer, you'll also need the kernel-modules-extra package installed for the cuse module. 

pip install ctypeslib hg+https://cusepy.googlecode.com/hg
sudo modprobe cuse
sudo chmod 666 /dev/cuse
qrandom-dev -v
sudo chmod 666 /dev/qrandom
By default it will use 3 threads, which can be changed by passing '-t #' into the qrandom-dev.

Testing the randomness for FIPS 140-2 compliance

$ cat /dev/qrandom | rngtest --blockcount=1000
rngtest: bits received from input: 20000032
rngtest: FIPS 140-2 successes: 1000
rngtest: FIPS 140-2 failures: 0
rngtest: FIPS 140-2(2001-10-10) Monobit: 0
rngtest: FIPS 140-2(2001-10-10) Poker: 0
rngtest: FIPS 140-2(2001-10-10) Runs: 0
rngtest: FIPS 140-2(2001-10-10) Long run: 0
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=17.696; avg=386.711; max=4882812.500)Kibits/s
rngtest: FIPS tests speed: (min=10.949; avg=94.538; max=161.640)Mibits/s
rngtest: Program run time: 50708319 microseconds

Adding entropy to the Linux random number generator

sudo rngd --rng-device=/dev/qrandom --random-device=/dev/random --timeout=5 --foreground

Monitoring your available entropy levels

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

Python API

The quantumrandom Python module contains a low-level get_data function, which is modelled after the ANU Quantum Random Number Generator's JSON API. It returns variable-length lists of either uint16 or hex16 data. 

>>> quantumrandom.get_data(data_type='uint16', array_length=5)
[42796, 32457, 9242, 11316, 21078]
>>> quantumrandom.get_data(data_type='hex16', array_length=5, block_size=2)
['f1d5', '0eb3', '1119', '7cfd', '64ce']

Based on this get_data function, quantumrandom also provides a bunch of higher-level helper functions that make easy to perform a variety of tasks. 

>>> quantumrandom.randint(0, 20)
5
>>> quantumrandom.hex()[:10]
'8272613343'
>>> quantumrandom.binary()[0]
'\xa5'
>>> len(quantumrandom.binary())
10000
>>> quantumrandom.uint16()
numpy.array([24094, 13944, 22109, 22908, 34878, 33797, 47221, 21485, 37930, ...], dtype=numpy.uint16)
>>> quantumrandom.uint16().data[:10]
'\x87\x7fY.\xcc\xab\xea\r\x1c`'

Follow quantumrandom on GitHub: https://github.com/lmacken/quantumrandom


posted at: 16:30 | link | Tags: , , , , | 5 comments

Sat, 07 May 2011

Red Hat OpenShift Express & The Leafy Miracle

Red Hat made a lot of awesome announcements this week at The Red Hat Summit, one of which being OpenShift.

I had the opportunity to play with the internal beta for a little while now, and I must say that as a developer I am extremely impressed with the service. Just being able to git push my code into to the cloud drastically simplifies large-scale software deployment, and makes it so I don't even have to leave my development environment.

I figured out a way to get TurboGears2 and Pyramid running on OpenShift Express, and documented it here and here. After that, I proceeded to write my very first Pyramid application.

[ The Leafy Miracle ]

In memory of the proposed [and rejected] Fedora 16 codename "Beefy Miracle", this little app is called "Leafy Miracle".

leafy-miracle.rhcloud.com


[ Features & Tech ]

[ Running ]
sudo yum -y install python-virtualenv
git clone git://fedorapeople.org/~lmacken/leafymiracle && cd leafymiracle
virtualenv env && source env/bin/activate
python setup.py develop
python leafymiracle/populate.py
paster serve development.ini
[ Code ]

git clone git://fedorapeople.org/~lmacken/leafymiracle

[ Props ]

Mad props go out to RJ Bean, who helped me write this app. He is responsible for writing a ton of amazing Python widgets for various JavaScript visualization libraries. You can see some demos of them here: tw2-demos.threebean.org.


posted at: 20:20 | link | Tags: , , , , , , , , , , | 0 comments

Thu, 24 Mar 2011

git clone all of your Fedora packages

After doing a fresh Fedora 15 install on my laptop last night, I wanted to quickly clone all of the packages that I maintain. Here is a single command that does the job:

python -c "import pyfedpkg; from fedora.client.pkgdb import PackageDB; [pyfedpkg.clone(pkg['name'], '$USER') for pkg in PackageDB().user_packages('$USER')['pkgs']]"


posted at: 16:10 | link | Tags: , , | 3 comments

Sun, 13 Mar 2011

Fedora Photobooth @ SXSW

This is the first year that Fedora will have a booth at SXSW! Sadly, I am not going to be attending since it conflicts with PyCon. However, my code will be running at our booth. Usually the Fedora booth at conferences is comprised of a bunch of flyers, media, swag, and some people to help answer questions and tell the Fedora story. However at SXSW, things are going to be a little different.

Aside from the amazing flyers that Máirín created, there will also be a Fedora Photobooth. Someone (probably Spot or Jared) will be dressed in a full Tux costume, and people can come and get their photo taken with them. Spot came to me the other day and asked if I could write some code to streamline the whole process.

An hour or so later, photobooth.py was born. There are definitely lots of improvements that can be made, but here is what it currently does in its initial incarnation:

In Action
See Mo's blog for photos of this code in action at the Fedora SXSW booth!
* SXSW Expo Day 1 from the show floor
* SXSW Expo Day 2
* A Beefy, Miraculous Day at SXSW (Expo Day 3)


The Code
I threw this in a git repo and tossed it up on GitHub:
github.com/lmacken/photobooth.py 
#!/usr/bin/python
# photobooth.py - version 0.3
# Requires: python-imaging, qrencode, gphoto2, surl
# Author: Luke Macken <lmacken@redhat.com>
# License: GPLv3

import os
import surl
import Image
import subprocess

from uuid import uuid4
from os.path import join, basename, expanduser

# Where to spit out our qrcode, watermarked image, and local html
out = expanduser('~/Desktop/sxsw')

# The watermark to apply to all images
watermark_img = expanduser('~/Desktop/fedora.png')

# This assumes ssh-agent is running so we can do password-less scp
ssh_image_repo = 'fedorapeople.org:~/public_html/sxsw/'

# The public HTTP repository for uploaded images
http_image_repo = 'http://lmacken.fedorapeople.org/sxsw/'

# Size of the qrcode pixels
qrcode_size = 10

# Whether or not to delete the photo after uploading it to the remote server
delete_after_upload = True

# The camera configuration
# Use gphoto2 --list-config and --get-config for more information
gphoto_config = {
    '/main/imgsettings/imagesize': 3, # small
    '/main/imgsettings/imagequality': 0, # normal
    '/main/capturesettings/zoom': 70, # zoom factor
}

# The URL shortener to use
shortener = 'tinyurl.com'

class PhotoBooth(object):

    def initialize(self):
        """ Detect the camera and set the various settings """
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 --auto-detect ' +
                        ' '.join(cfg), shell=True)

    def capture_photo(self):
        """ Capture a photo and download it from the camera """
        filename = join(out, '%s.jpg' % str(uuid4()))
        cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]
        subprocess.call('gphoto2 ' +
                        '--capture-image-and-download ' +
                        '--filename="%s" ' % filename,
                        shell=True)
        return filename

    def process_image(self, filename):
        print "Processing %s..." % filename
        print "Applying watermark..."
        image = self.watermark(filename)
        print "Uploading to remote server..."
        url = self.upload(image)
        print "Generating QRCode..."
        qrcode = self.qrencode(url)
        print "Shortening URL..."
        tiny = self.shorten(url)
        print "Generating HTML..."
        html = self.html_output(url, qrcode, tiny)
        subprocess.call('firefox "%s"' % html, shell=True)
        print "Done!"

    def watermark(self, image):
        """ Apply a watermark to an image """
        mark = Image.open(watermark_img)
        im = Image.open(image)
        if im.mode != 'RGBA':
            im = im.convert('RGBA')
        layer = Image.new('RGBA', im.size, (0,0,0,0))
        position = (im.size[0] - mark.size[0], im.size[1] - mark.size[1])
        layer.paste(mark, position)
        outfile = join(out, basename(image))
        Image.composite(layer, im, layer).save(outfile)
        return outfile

    def upload(self, image):
        """ Upload this image to a remote server """
        subprocess.call('scp "%s" %s' % (image, ssh_image_repo), shell=True)
        if delete_after_upload:
            os.unlink(image)
        return http_image_repo + basename(image)

    def qrencode(self, url):
        """ Generate a QRCode for a given URL """
        qrcode = join(out, 'qrcode.png')
        subprocess.call('qrencode -s %d -o "%s" %s' % (
            qrcode_size, qrcode, url), shell=True)
        return qrcode

    def shorten(self, url):
        """ Generate a shortened URL """
        return surl.services.supportedServices()[shortener].get({}, url)

    def html_output(self, image, qrcode, tinyurl):
        """ Output HTML with the image, qrcode, and tinyurl """
        html = """
            <html>
              <center>
                <table>
                  <tr>
                    <td colspan="2">
                        <b><a href="%(tinyurl)s">%(tinyurl)s</a></b>
                    </td>
                  </tr>
                  <tr>
                    <td><img src="%(image)s" border="0"/></td>
                    <td><img src="%(qrcode)s" border="0"/></td>
                  </tr>
                </table>
              </center>
          </html>
        """ % {'image': image, 'qrcode': qrcode, 'tinyurl': tinyurl}
        outfile = join(out, basename(image) + '.html')
        output = file(outfile, 'w')
        output.write(html)
        output.close()
        return outfile

if __name__ == "__main__":
    photobooth = PhotoBooth()
    try:
        photobooth.initialize()
        while True:
            raw_input("Press enter to capture photo.")
            filename = photobooth.capture_photo()
            photobooth.process_image(filename)
    except KeyboardInterrupt:
        print "\nExiting..."


posted at: 02:57 | link | Tags: , , , | 20 comments

Wed, 16 Feb 2011

FUDCon 2011 Tempe

I meant to get this summary out the door weeks ago, however, I didn't want to distract from my post-FUDCon productivity :)

Another FUDCon has come and gone. This time it was in gorgeous Tempe, Arizona. It was my first time in AZ, and I must say that I was thoroughly impressed. It was truly a great location for FUDCon. Thanks to the epic snowstorm of doom, I also was stuck there for a few extra days

Live Widgetry
Before the conference I decided to throw together a little live widget that scrolls all of the new identi.ca posts tagged with #FUDCon. Thanks to the mad design skillz of mizmo, and the feed aggregation and real-time web sockets of Moksha, I was able to throw it together pretty quickly. I plan on taking this code and integrating it in the existing fedoracommunity dashboard and hooking up many different fedora-related feeds to it.

Sessions

AutoQA
Day 1 of the FUDCon sessions were quite interesting. I got a chance to learn a bit more about the exciting AutoQA project, which is coming along nicely. You'll be seeing AutoQA commenting on bodhi updates soon enough.

Security Lab
I caught Joerg Simon's session on the Fedora Security Lab. It was exciting to see how the Security Spin has evolved ever since I created it back in 2007 for a project in my forensics class. It was also interesting to learn more about OSSTMM.

Spins
The next session I attended was about the future of spins. Almost everyone agreed that Spins are useful and a valuable part of Fedora. The problems seem to mostly lie in governance/policy and a lack of communication and coordination between the Spins SIG and QA/Releng/Infrastructure. Once of my ideas below may help with this a little bit.

The Next Big Project

Day 2 was comprised of more sessions, including my team's "Next Big Project" proposals. Of course, the Fedora RPG that Spot and Mo talked about was definitely a hot topic, and got a lot of people excited.

I talked about a handful of project ideas that I would like to work on in the future (actually, I have code written for most of them already). Here is a quick rundown:

Real-time Infrastructure
I want to see us deploy an AMQP message broker inside our production infrastructure. Then, we hook up all of our existing services and have them fire off messages when various events occur (koji builds, bodhi updates, pkgdb additions/removals/changes, git hooks, planet feeds, wiki edits, etc). From here, using some realtime web technology that we created, we could easily expose these message queues via a live dashboard that lets you filter and navigate the stream of activity, along with providing real-time metrics. We can also create desktop notification widgets, so you can get popup bubbles for things that you care about.

This is also key to the whole RPG as well. In order to build a game based on Fedora workflows, we need an underlying expert system that knows what actions can be taken within fedora, how they are accomplished, and who is getting them done.

Meeting app
Currently after a meeting, our Meetbot spits out the logs and an overview in txt/html/rst to a directory on meetbot.fedoraproject.org. Trying to track down who agreed to what when, or even to see what a given team has been up to over the past couple of months, is very tedious.

The data is there, now we just need to make it useful. I would love to see a frontend for this sytem that tracked meetings by team/people/topics/projects, kept track of actions and held people accountable for what they say they are going to do (and make it easy for people to say "I need help with this", or "I don't have time to finish this"), making it simple to go from an #idea in a meeting to implementation. There is so much great data in these logs, and I think we can do some awesome things with it.

Improved upstream monitoring
Most Fedora developers probably don't even know that we already have an Upstream release monitoring service buried in the wiki. I added almost every package I maintain to it, and it will automatically open a bug when a new upstream version is released. Extremely useful.

I wouldn't call this a "big" project, but I would like to see us integrate this service into our existing infrastructure. We could potentially store this per-package upstream data in the pkgdb/bodhi, and when a new release comes out write some code to automatically try doing a simple specfile bump, throw a scratch build at koji, run it through AutoQA, queue up for testing in bodhi, etc. Ideally, this would minimize the massive amounts of effort that our maintainers have to do to keep our packages up to speed with upstream.

Discussions app
Last year, Máirín Duffy and I came up with some interesting ideas for improving our mailing lists. I would like to make this a reality. Since then, I have already written code that can successfully parse all of fedora-devel.mbox (sounds much easier than it really is), populate it into a SQLAlchemy database model, expose a JSON API for quering, and visualize threads and various statistics with a basic widget.

Spin Master
One of the biggest problems with spins that we have right now is that there is no easy way to track how they are evolving. I would like to see us create a system that took the nightly spins and analyzed them, tracking what packages have been added/removed, which packages have grown/shrunk, etc. We could potentially get AutoQA involved here and make sure all spins pass a certain level of sanity checks before they can even be released. There are scripts floating around that can do a lot of this already -- but I want to streamline it and build a frontend.

Source-level package diff viewer
Fedora churns at such a fast pace, yet I can only imagine that a small subset of maintainers actually look at the complete code changes between upstream package releases. The recent sourceforge intrusion should be seen as a reality check to us distributions, and I think we need to step up our game quite a bit to ensure we don't let any malicious code slip into Fedora.

I would like to see a web/cli interface for viewing full source diffs of package updates in Fedora, allowing people to annotate/flag lines of code. Having more eyes view the code changes that go into our distribution is definitely a Good Thing, not just for Fedora, but for Open Source in general.

Hackfests

Now, on to my favorite part of any conference -- The Hackfests. First off, I felt that the hackfests were a bit unorganized this year. There was no opportunity to pitch hackfests, and it was not easy to figure out who was doing what in which rooms. Anyway, I had a fairly productive day of hacking...

bodhi
I integrated our package test cases into bodhi, which will now query the wiki for tests and display them in your updates, like so:

Thanks to James Laska for the code to query the wiki, and to Ian Weller for python-simplemediawiki API.

I also sat down with Máirín Duffy and talked about Bodhi v2.0 interaction design. We discussed what actions we want people to take when they arrive at bodhi's homepage, which essentially boils down to submitting, searching, browsing, and testing updates. Mo quickly threw together an awesome mockup that portrays some of our initial ideas.


Fedora Community Discussions
As mentioned above, I have already started implementing the mailing list interface that mizmo and I designed last year. I worked with Casey Dahlin during the hackfests and helped him get a working fedoracommunity/moksha development environment up and running and become familiar with the existing code.

Even though he wasn't at FUDCon, Jan Hutar has also been working on a couple of great graphs/grids of mailing list statistics for fedoracommunity as well. We'll be hacking away at this stuff over the next few months, so stay tuned.

kernel EFI framebuffer
I spun up a quick kernel patch to enable the EFI framebuffer on a handfull of Macs. I already wrote a patch that got applied upstream that enables this framebuffer on 14 different mac models, but this new patch adds 5 more. A lot of people, especially Sugar on a Stick users, are desperate to get Fedora running on their mactel machines (assuming found in may school labs), so I spun up a fresh livecd with my kernel patch for testing. See Bug #528232 for more information.

liveusb-creator
I did a bunch of work on porting the liveusb-creator from HAL to UDisks. Thankfully, Ubuntu's cleverly-named "usb-creator" already has UDisks support, so I've been happily borrowing ideas from their code :)

I also had some great discussions with Peter Robinson and Bernie Innocenti about solving the persistent overlay problem with our Live USBs. Right now, they are essentially a ticking time bomb, and real world LiveUSB use-cases are getting bit by this all of the time. Over the past few years, the "solution" has been to "wait for unionfs to get merged into the kernel". However, there are a variety of other potential solutions that we are going to look into as well.

Pyramid
The future of Python web development is extremely exciting, innovative, and still evolving at a rapid pace. As a TurboGears developer, I'm still very impressed with TG2, which along with TG1 will be supported for a long time to come -- but I'm also very eager for the next generation framework that has just emerged.

Recently, the Pylons project has merged with repoze.bfg to form Pyramid, which just released version 1.0. I'm quite amazed by the quality of the code, docs, and tests, and benchmarks already show it blowing rails/django/tg/pylons out of the water. I'm looking forward to the PyCon sprints, where all 3 communitites are going to be sitting at the same table working together on this project.

So while I was stuck in Tempe during the snow storm, I wrote 7 RPMs for Pyramid and it's dependencies, which are currently waiting to be reviewed. I plan on writing Bodhi v2.0 using Pyramid, so if this is something that interests you, let me know (and start reading the 600+ pages of docs ;)

Good times
Technical stuff aside, I had a blast at FUDCon, and I feel like it was one of the best. FUDPub was great, as usual. I played a lot of poker [poorly], ate a lot of tasty food, and had some great conversations. I had caught a cold prior to coming to FUDCon, so instead of nursing it with some nyquil and sleep, I decided to try to nurse it with a bottle of Jack, which turned out to be a bad idea.

Also, no FUDCon would be complete without mentioning TheGreatestGameYouWillEverPlay.com. Once the hackfests started winding down, I gave a quick session on nethack, where I tought people various ways to steal from shops, using a bunch of screecasts that I had on my laptop.


posted at: 21:55 | link | Tags: , , | 0 comments

Thu, 06 Jan 2011

liveusb-creator 3.9.3 windows release

I spent the majority of yesterday at a DOS prompt. Thankfully, it wasn't as painful as it sounds, as git, vim and Python make Windows development quite tolerable.

Anyway, I was finally able to track down and fix a couple of major bugs in the liveusb-creator on Windows XP and 7, and I pushed out a new build yesterday with the following changes:

Windows users, download it here: http://liveusb-creator.fedorahosted.org


posted at: 15:34 | link | Tags: , , , | 7 comments

Fri, 25 Jun 2010

Professors' Open Source Summer Experience @ RIT

Last week Red Hat put on a Professors' Open Source Summer Experience (POSSE) at RIT. Being an alumni, I was excited by the opportunity to be able to go back up to The ROC and teach some of the people that taught me. Going into it, I really had no idea what to expect. All I knew is that I was going to help lead the 'deep dive' section of the course, where I would teach professors how to dive in head first and get productively lost in a strange codebase. This is not something that can be accomplished with a set of powerpoint slides. Teaching how to hack on open source requires that you emerse yourself into a codebase, and bring your students with you.

The previous POSSE at Worcester State dove into the Sugar Measure Activity, and we were going to do the same.

Measure is an activity that turns the computer into an oscilloscope. Signals from the microphone (and sensors) can be plotted in time and frequency domains.

I had never used this activity, let alone hacked on it before. I've also never done any sugar activity development, aside from some tweaking of the OpenVideoChat, so I really had no idea what I was getting myself into. The obvious first step was to get it running. All of us were able to start the activity in virtual machines or emulators, except for one install which hit some odd errors upon startup. We were able to quickly track the bug down to a stray return statement in __init__ before some critical initialization code. Right after we fixed the problem we noticed that Walter Bender had already fixed this issue a few hours earlier. After a git pull, we were up and running.

Once we all got the activity running, we took a look at the bug list to see if there was any low-hanging fruit for us to tackle. Since the previous POSSE had already done some work on this activity the week before, there were not any trivial tickets left in the queue. So, in that case, we dove head first into the hardest one, "Measure activity gets stuck after recording". This ticket had very little information, and no log output, so we were on our own to try and track it down. We were able to reliably reproduce the issue on the XO-1.5, but not on the 1.0. In our virtual machines we hit it sporadically. We all agreed that it felt like a race-condition, most likely due to threading. So we started instrumenting the code and adding some debugging statements to try and figure out which line of code was the culprit.

In our efforts to scatter print statements all over the place to try and determine the code path, we noticed that none of our output was hitting the logs. When you have no idea if your code is even being run or not, don't be ashamed to throw in a raise Exception("WTF?!"). We finally realized that since the activity would freeze, it was never able to flush stderr/stdout to the log. A quick find/replace regex later, and we were using the proper logging module and seeing our debugging output hit the logs.

We were then able to track the bug down to a line of code that uses GTK to try and get the coordinates of the parent window. At this point, since none of us were GTK experts, we had to go upstream. So, I dropped into #fedora-devel and asked. Within 10 minutes I had responses from 3 different GTK hackers. One was a typical RTFM response, which humored the professors (who are very used to hearing/saying this), but the others pointed us in the right direction. One mentioned that gtk.gdk calls probably should not be done in a seperate thread. So, a suggested workaround was to add gtk.threads_enter()/gtk.threads_leave() calls before running any gtk.gdk code in the thread. A 2-line patch later, and we had squashed the bug.

We eventually bumped into the inevitable typo in some comments. So, we made the fix locally, and committed it to our git repo. A few minutes later and we found another one. I saw this as a great opportunity to show off some of my git-fu. Instead of sending two "Fix typo" patches upstream, I showed the professors how to use git interactive rebasing to squash multiple commits into a single one. They all followed along closely, and the workflow made sense to them.

While we were looking through the ticket queue, we saw an issue where Measure would apparently leak memory and crash when running it for a long period of time. While keeping this in mind, we kept our eyes peeled while wandering around the codebase to see if we could track the issue down. When looking at the code that takes screenshots of the waveform, I noticed that it created a temporary file with tempfile.mkstemp, saved the pixmap to it, injected it into the Journal, and then deleted the directory. This looked fine at a first glance, until I realized that it never closed the temporary file descriptor. Another 2-line patch later, and this issue was solved.

The next day both of the patches that we sent upstream were applied by Walter Bender.

Overall, POSSE definitely exceeded my expectations, and I'm extremely satisfied with how the 'deep dive' section went. I went into it feeling completely unprepared to teach, but by trusting my "hacker intuition", I feel that it turned out to be a fantastic learning experience for all of us.


posted at: 19:59 | link | Tags: , , | 0 comments

Tue, 08 Jun 2010

Fedora Updates Report

I recently wrote some code to generate detailed statistics of Fedora & EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics.

This report definitely conveys the shortcomings in how we currently utilize bodhi for "testing" updates, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria, I think we'll see these numbers drastically improve in the future.

You can find the code that generates these statistics here: metrics.py, log_stats.py. If you have any ideas or suggestions for different types of metrics to generate, or if you find any bugs in my code, please let me know. 

Bodhi Statistics Report (Generated on June 8th, 2010)
=====================================================

Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
 * notting (424)
 * mclasen (366)
 * jkeating (321)
 * adamwill (283)
 * cwickert (161)
 * rdieter (159)
 * pbrobinson (141)
 * kevin (141)
 * cweyl (122)
 * tomspur (119)
 * mtasaka (110)
 * xake (97)
 * cschwangler (86)
 * kwright (84)
 * peter (83)
 * hadess (80)
 * michich (72)
 * tagoh (69)
 * pfrields (69)
 * bpepple (69)
 * iarnell (68)
 * lkundrak (66)
 * shinobi (65)
 * sundaram (64)
 * spot (62)
 * pravins (62)
 * markmc (62)
 * thomasj (61)
 * smooge (60)
 * fab (59)

================================================================================
     Fedora 13
================================================================================

 * 3562 updates
 * 3065 stable updates
 * 427 testing updates
 * 62 pending updates
 * 8 obsolete updates
 * 2371 bugfix updates (66.56%)
 * 745 enhancement updates (20.92%)
 * 89 security updates (2.50%)
 * 357 newpackage updates (10.02%)
 * 410 critical path updates (11.51%)
 * 333 critical path updates approved
 * 1155 updates received feedback (32.43%)
 * 12120 +0 comments
 * 2477 +1 comments
 * 155 -1 comments
 * 595 unique authenticated karma submitters
 * 133 anonymous users gave feedback (1.57%)
 * 2261 out of 3562 updates went through testing (63.48%)
 * 1317 testing updates were pushed *without* karma (58.25%)
 * 21 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 11 days
   * median = 9 days
   * mode = 7 days
 * 4 updates automatically unpushed due to karma (0.11%)
   * 0 of which were critical path updates
 * 231 updates automatically pushed due to karma (6.49%)
   * 2 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2445 packages updated (top 10 shown)
    * selinux-policy: 13
    * jd: 12
    * openoffice.org: 12
    * gdb: 12
    * ibus-pinyin: 11
    * nautilus: 10
    * kernel: 10
    * evolution: 9
    * libfm: 9
    * libmx: 9

================================================================================
     Fedora 12
================================================================================

 * 4844 updates
 * 4291 stable updates
 * 371 testing updates
 * 113 pending updates
 * 69 obsolete updates
 * 2905 bugfix updates (59.97%)
 * 1054 enhancement updates (21.76%)
 * 201 security updates (4.15%)
 * 684 newpackage updates (14.12%)
 * 407 critical path updates (8.40%)
 * 960 updates received feedback (19.82%)
 * 16311 +0 comments
 * 1899 +1 comments
 * 554 -1 comments
 * 758 unique authenticated karma submitters
 * 576 anonymous users gave feedback (5.33%)
 * 2873 out of 4844 updates went through testing (59.31%)
 * 2138 testing updates were pushed *without* karma (74.42%)
 * 188 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 14 days
   * median = 13 days
   * mode = 17 days
 * 12 updates automatically unpushed due to karma (0.25%)
   * 4 of which were critical path updates
 * 133 updates automatically pushed due to karma (2.75%)
   * 13 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2902 packages updated (top 10 shown)
    * qbittorrent: 25
    * gdb: 25
    * selinux-policy: 22
    * kernel: 15
    * xorg-x11-server: 14
    * ibus: 13
    * jd: 13
    * abrt: 11
    * gvfs: 11
    * gtk2: 11

================================================================================
     Fedora 11
================================================================================

 * 6987 updates
 * 6381 stable updates
 * 183 testing updates
 * 99 pending updates
 * 324 obsolete updates
 * 3649 bugfix updates (52.23%)
 * 1566 enhancement updates (22.41%)
 * 350 security updates (5.01%)
 * 1422 newpackage updates (20.35%)
 * 383 critical path updates (5.48%)
 * 729 updates received feedback (10.43%)
 * 23427 +0 comments
 * 1197 +1 comments
 * 448 -1 comments
 * 782 unique authenticated karma submitters
 * 481 anonymous users gave feedback (3.58%)
 * 4129 out of 6987 updates went through testing (59.10%)
 * 3620 testing updates were pushed *without* karma (87.67%)
 * 278 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 15 days
   * median = 14 days
   * mode = 17 days
 * 7 updates automatically unpushed due to karma (0.10%)
   * 0 of which were critical path updates
 * 64 updates automatically pushed due to karma (0.92%)
   * 11 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 3787 packages updated (top 10 shown)
    * libguestfs: 30
    * jd: 24
    * selinux-policy: 23
    * kdebase-workspace: 19
    * kernel: 18
    * gdb: 16
    * dovecot: 16
    * qemu: 16
    * kdebase-runtime: 16
    * kdenetwork: 16

================================================================================
     Fedora EPEL 5
================================================================================

 * 1572 updates
 * 1255 stable updates
 * 198 testing updates
 * 43 pending updates
 * 76 obsolete updates
 * 734 bugfix updates (46.69%)
 * 236 enhancement updates (15.01%)
 * 93 security updates (5.92%)
 * 509 newpackage updates (32.38%)
 * 20 critical path updates (1.27%)
 * 103 updates received feedback (6.55%)
 * 6076 +0 comments
 * 156 +1 comments
 * 19 -1 comments
 * 243 unique authenticated karma submitters
 * 41 anonymous users gave feedback (1.22%)
 * 1176 out of 1572 updates went through testing (74.81%)
 * 1092 testing updates were pushed *without* karma (92.86%)
 * 19 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 24 days
   * median = 18 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 10 updates automatically pushed due to karma (0.64%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 1060 packages updated (top 10 shown)
    * libguestfs: 26
    * znc: 10
    * vrq: 8
    * cherokee: 8
    * 389-ds-base: 8
    * viewvc: 8
    * 389-admin: 7
    * pki-ca: 7
    * wordpress-mu: 7
    * Django: 7

================================================================================
     Fedora EPEL 4
================================================================================

 * 447 updates
 * 359 stable updates
 * 40 testing updates
 * 11 pending updates
 * 37 obsolete updates
 * 222 bugfix updates (49.66%)
 * 68 enhancement updates (15.21%)
 * 40 security updates (8.95%)
 * 117 newpackage updates (26.17%)
 * 5 critical path updates (1.12%)
 * 11 updates received feedback (2.46%)
 * 1592 +0 comments
 * 11 +1 comments
 * 2 -1 comments
 * 85 unique authenticated karma submitters
 * 2 anonymous users gave feedback (0.24%)
 * 320 out of 447 updates went through testing (71.59%)
 * 311 testing updates were pushed *without* karma (97.19%)
 * 5 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 18 days
   * median = 16 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 1 updates automatically pushed due to karma (0.22%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 313 packages updated (top 10 shown)
    * cherokee: 8
    * globus-common: 7
    * R: 6
    * voms: 6
    * globus-gsi-proxy-ssl: 5
    * globus-openssl-module: 5
    * globus-gsi-proxy-core: 5
    * bitlbee: 5
    * flashrom: 5
    * viewvc: 5


posted at: 20:41 | link | Tags: , , , | 0 comments

Tue, 25 May 2010

liveusb-creator trojan in the wild

I've been noticing many different copies of my Windows liveusb-creator popping up on various sketchy-looking download sites. The majority of these copies contain a variant of the Vundo Trojan.

"Vundo, or the Vundo Trojan (also known as Virtumonde or Virtumondo and sometimes referred to as MS Juan) is a Trojan horse that is known to cause popups and advertising for rogue antispyware programs, and sporadically other misbehavior including performance degradation and denial of service with some websites including Google and Facebook."

So, if you downloaded a copy of the Windows liveusb-creator from anywhere other than https://fedorahosted.org/liveusb-creator -- you could be infected. Apparently the latest variation of this trojan is undetectable by most antivirus (although, clamav was able to recognize the one that I found), so you may need to look around for some of the common symptoms. There is apparently a tool that will remove this trojan which can be found here, however I have not tested it and cannot vouch for its validity.

If anyone was actually hit by this, I'd be interested to hear about it.

Also, to state the blatantly obvious: only download the liveusb-creator from the homepage!


posted at: 21:11 | link | Tags: , , , | 4 comments

Sat, 27 Mar 2010

Fedora Community Statistics

I'm pleased to announce that version 0.4.0 of the Fedora Community dashboard has just hit production. Along with the usual batch of bugfixes, this release contains a new 'Statistics' section that contains metrics from a variety of different pieces of Fedora Infrastructure.

Thanks goes to Ian Weller for the wiki stats, Seth Vidal for the torrent stats, and Matt Domsch & Jef Spaleta for the map generation code. I ended up writing the updates metrics, package stats, and users/mirrors widgets. Enjoy!








posted at: 03:48 | link | Tags: , , | 4 comments

Mon, 18 Jan 2010

nose 0.11

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module. 

lmacken@tomservo ~/bodhi $ nosetests
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 725.111s

OK

lmacken@tomservo ~/bodhi $ nosetests --processes=50
................................................................................................
----------------------------------------------------------------------
Ran 96 tests in 10.915s

OK

Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month


posted at: 22:58 | link | Tags: , , , , | 4 comments

Thu, 31 Dec 2009

My dream machine

In an effort to optimize my home office, I recently donated my server rack to a local Boston record label, to hold their Red Hat servers. I'm also in the process of donating all of my computer hardware for re-use/recycling (~10 or so frankenstein boxen).

So, once I clear everything out, I'm going to replace it with a new machine. I usually sit in front of 1-3 laptops (thindpads and XOs) on the daily, and I absolutley love them, but I need something beefier. I do most of my work on remote machines, but I have found that I still spend too much time waiting on computers.

I'm not much of a gamer, so I probably don't need too high-end of a graphics chip, let alone SLI/Crossfire. The extent of my gaming these days consists of the occassional wesnoth, open arena, nethack, and my current favorite Cube 2: Sauerbraten. I just want a card that will work well in Linux, ideally without having to install proprietary drivers.

Anyway, I haven't built a desktop machine from the ground up in 12 years, and I've been out of the hardware game for a long time, so let me know what's good! Here is what I've been looking at so far...

Intel DX58SO Extreme Series X58 ATX Tri-Channel DDR3 16GB SLI or CrossFireX LGA1366 Overclocking Utility Desktop Board
or
ASUS Rampage II Extreme LGA1366 Intel X58 DDR3-1600 ATX Motherboard

Intel Core i7 975 Extreme Edition 3.33GHz 8M L3 Cache LGA1366 Desktop Processor
Cooler Master V8 Nickel Plated Copper Base Aluminum Fins 8 Heatpipes Core i7 1366 CPU Cooler - (RR-UV8-XBU1-GP)

Corsair 6 GB Dominator GT PC3-16000 2000Mhz 240-pin Triple Channel DDR3 Memory Kit
Corsair CMPSU-1000HX 1000-Watt HX Professional Series 80 Plus Certified Power Supply

Intel X25-E Extreme SATA Solid-State Drive
or
Corsair 256 GB Internal Solid State Drive (SSD) CMFSSD-256GBG2D

DELL ULTRASHARP 3008WFP - 30-inch Widescreen Flat Panel Monitor
EVGA 01G-P3-1180-AR GeForce GTX285 1024 MB DDR3 PCI-Express 2.0 Graphics Card

Endurapro
Cooler Master HAF 932 High Air Flow ATX Full Tower Case

Also, if you appreciate my software and want me to write it faster... donations are accepted ;)


posted at: 01:05 | link | Tags: | 15 comments

Tue, 29 Dec 2009

RIP Fedora 10

Fedora 10 (Cambridge) (2008-11-25 -- 2009-12-17)

Updates

Source: bodhi
Fedora 10 Updates
Most updates per developer in Fedora 10
Most Updated Packages in Fedora 10
Packages with best karma
Top Fedora 10 testers
Most tested Fedora 10 packages

Torrent

Source: fedoracommunity (upcoming release)

Torrent NameNumber of completed downloads
Fedora-10-i386-DVD 112,807
Fedora-10-x86_64-DVD 65,965
Fedora-10-i386-CDs 10,621
Fedora-10-ppc-DVD 6,851
Fedora-10-source-DVD 3,740
Fedora-10-x86_64-CDs 3,141
Fedora-10-ppc-CDs 1,336
Fedora-10-i686-AOS 666
Fedora-10-source-CDs 662
Fedora-10-i686-Live 599
Fedora-10-x86_64-Live 336
Fedora-10-x86_64-AOS 274
Fedora-10-i686-Live-KDE 201
Fedora-10-x86_64-Live-KDE 78
Fedora-10-i686-Live-XFCE 37
Fedora-10-i686-Live-Developer 13
Fedora-10-i686-Live-FEL 12
Fedora-10-x86_64-Live-XFCE 5
Fedora-10-i686-Live-broffice 3
Fedora-10-x86_64-Live-Developer 3
Fedora-10-x86_64-Live-FEL 2
Fedora-10-x86_64-Live-edu-math 1
Fedora-10-i686-Live-edu-math 1
Fedora-10-x86_64-Live-broffice 0
Total 207,354

Yum Data

Source: wiki/Legacy_statistics
Connections to yum
Week Dates New Unique IPsTotal Unique IPsTotal compared to F9
1 2008-11-25 -- 2008-12-0167,421 67,421 73%
2 2008-12-02 -- 2008-12-0881,674 149,095 97%
3 2008-12-09 -- 2008-12-1560,759 209,854 97%
4 2008-12-16 -- 2008-12-2262,527 272,381 93%
5 2008-12-23 -- 2008-12-2968,375 340,756 97%
6 2008-12-30 -- 2009-01-0573,585 414,341 97%
7 2009-01-06 -- 2009-01-1294,166 508,507 103%
8 2009-01-13 -- 2009-01-1985,557 594,064 106%
9 2009-01-20 -- 2009-01-2687,678 681,742 107%
10 2009-01-27 -- 2009-02-0291,014 772,756 110%
11 2009-02-03 -- 2009-02-0995,238 867,994 113%
12 2009-02-10 -- 2009-02-1695,967 963,961 115%
13 2009-02-17 -- 2009-02-23109,800 1,073,761 115%
14 2009-02-24 -- 2009-03-0285,246 1,159,007 --
15 2009-03-03 -- 2009-03-09100,610 1,259,617 --
16 2009-03-10 -- 2009-03-16100,323 1,359,940 --
17 2009-03-17 -- 2009-03-23100,819 1,460,759 --
18 2009-03-24 -- 2009-03-30102,843 1,563,602 --
19 2009-03-31 -- 2009-04-06101,978 1,665,580 136%
20 2009-04-07 -- 2009-04-1399,586 1,765,166 --
21 2009-04-14 -- 2009-04-20101,808 1,866,974 --
22 2009-04-21 -- 2009-04-27100,230 1,967,177 --
23 2009-04-28 -- 2009-05-0497,584 2,064,761 --
24 2009-05-05 -- 2009-05-1195,923 2,160,684 137%
25 2009-05-12 -- 2009-05-1895,632 2,256,316 --
26 2009-05-19 -- 2009-05-2592,377 2,348,693 --
27 2009-05-26 -- 2009-06-0191,747 2,440,440 --
28 2009-06-02 -- 2009-06-0891,513 2,531,953 --

Direct downloads

Source: wiki/Legacy_statistics
The following table shows the number of direct downloads of Fedora 10 media from unique IP addresses, as shown in the web proxy logs. The actual number of raw downloads tends to be much higher.
Week Dates Downloads this week Total downloads
1 2008-11-25 -- 2008-12-01 236,886 236,886
2 2008-12-02 -- 2008-12-08 105,994 342,880
3 2008-12-09 -- 2008-12-15 83,740 426,620
4 2008-12-16 -- 2008-12-22 76,982 503,602
5 2008-12-23 -- 2008-12-29 66,351 569,953
6 2008-12-30 -- 2009-01-05 65,102 635,055
7 2009-01-06 -- 2009-01-12 72,729 707,784
8 2009-01-13 -- 2009-01-19 73,301 781,085
9 2009-01-20 -- 2009-01-26 72,082 853,167
10 2009-01-27 -- 2009-02-02 71,788 924,955
11 2009-02-03 -- 2009-02-09 72,529 997,484
12 2009-02-10 -- 2009-02-16 69,071 1,066,555
13 2009-02-17 -- 2009-02-23 69,216 1,135,771
14 2009-02-24 -- 2009-03-02 67,669 1,203,440
15 2009-03-03 -- 2009-03-09 66,666 1,270,106
16 2009-03-10 -- 2009-03-16 65,524 1,335,630
17 2009-03-17 -- 2009-03-23 63,218 1,398,848
18 2009-03-24 -- 2009-03-30 62,930 1,461,778
19 2009-03-31 -- 2009-04-06 59,813 1,521,591
20 2009-04-07 -- 2009-04-13 57,102 1,578,693
21 2009-04-14 -- 2009-04-20 55,871 1,634,564
22 2009-04-21 -- 2009-04-27 55,117 1,689,681
23 2009-04-28 -- 2009-05-04 50,815 1,740,496
24 2009-05-05 -- 2009-05-11 48,139 1,788,635
25 2009-05-12 -- 2009-05-18 47,813 1,836,448
26 2009-05-19 -- 2009-05-25 46,077 1,882,525
27 2009-05-26 -- 2009-06-01 44,969 1,927,494
28 2009-06-02 -- 2009-06-08 44,835 1,972,329


posted at: 00:05 | link | Tags: , , , | 0 comments

Thu, 10 Dec 2009

FUDCon Toronto 2009

Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion.

My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on.

Moksha

Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later.

One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker.

I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing:

"I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is https://fedorahosted.org/moksha/ and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well http://meetbot.fedoraproject.org/fudcon-room-3/2009-12-05/fudcon-room-3.2009-12-05-22.07.log.html ... presentation slides found: http://lmacken.fedorapeople.org/moksha-FUDConToronto-2009.odp"

Fedora Community

So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform.

After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen.

During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them).

I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to come. My main concern was a blocker issue that we were hitting with our flot graphs when you quickly bounce between tabs. I ended up "fixing" the bug, so I'll be pushing what we have of the stats branch into production in the near future.

TurboGears2

TurboGears has definitely been our favorite web framework within Fedora's Infrastructure for many years now. TurboGears2, a complete re-invention of itself, has been released recently, and is catching on *very* quickly in the community. Tons of people are working on awesome new apps, and loving every minute of it. I was also able to convert a rails hacker over to it, after he was able to quickly dive into one of the tutorials with ease. See my previous blog post about getting up and running with TG2 in Fedora/EPEL.

python-fedora

One of my main tasks during the hackfests was to pull the authentication layer in Fedora Community that authenticates against the Fedora Account System, and port it over to python-fedora, so we can use it in any TurboGears2 application. I committed the initial port to python-fedora-devel, and have started working on integrating it into a default TG2 quickstart and document the process. There are still a couple of minor things I want to fix/clean up before releasing it, so expect a blog about it soon.

Bodhi

It seems like yesterday that I was an intern at Red Hat working on an internal updates system for Fedora Core. Coming up on 5 years later, and I am now working on my 3rd implementation of an updates system, Bodhi v2.0. What's wrong with the current Bodhi you ask? Well, if you talk to any user of it, you'll probably get a pretty long list. Bodhi is the first TurboGears application written & deployed in Fedora Infrastructure, and uses the vanilla components (SQLObject, kid, CherryPy2). The TG1 stack has been holding up quite nicely over the years, and is still supported upstream, but bodhi's current implemention and design does not make it easy to grow.

Bodhi v2.0 will be implemented in TurboGears2, using SQLAlchemy for an ORM, Mako for templates, and ToscaWidgets2 for re-usable widgets. It will be hook-based and plugin-driven, and will be completely distribution agnostic. Another important goal will be AMQP message-bus integration, which will allow other services or users to react to various events inside of the system as they happen.

So far I've ported the old DB model from SQLObject to SQLAlchemy, and have begun porting the old unit tests, and writing new ones. Come the new year, I'll be giving this much more of my focus.

During the hackfests I got a chance to talk to Dennis Gilmore about various improvements that we need to make with regard to the update push process. It was also great to talk to many different users of bodhi, who expressed various concerns, some of which I've already fixed. I also got a chance to talk to Xavier Lamien about deploying Bodhi for rpmfusion. On the bus ride home I helped explain to Mel how Bodhi & Koji fit into the big picture of things.

During the BarCamp sessions I also attended a session about the Update Experience, where we discussed many important issues surrounding updates.

liveusb-creator

So I got a chance to finally meet Sebastian Dziallas, of Sugar on a Stick fame, and was able to fix a few liveusb-creator issues on his laptop. I ended up pushing out a new release a couple of days ago that contains some of those fixes, along with a new version of Sugar on a Stick.

The liveusb-creator has been catching a lot of press recently (see the front page for a list). Not only did it have a 2 page spread in Linux Format, but it was also featured in this weeks Wired.com article New Sugar on a Stick Brings Much Needed Improvements. Rock.

Python

There was lot of brainstorming done by Dave Malcolm, Colin Walters, Toshio Kuratomi, Bernie Innocenti, I, and many others about various improvements that we could make to the Python interpreter. From speeding up startup time by doing some clever caching to potentially creating a new optimized compiled binary format. We also looked into how WebError/abrt gather tracebacks, and discussed ways of enabling interactive traceback debugging for vanilla processes, without requiring a layer of WSGI middleware.

There was also work done on adding SystemTap probes to Python, which is very exciting. There are many ideas for various probe points, including one that I blogged about previously.

Intel iMac8,1 support

My iMac sucks at Linux. This has been something that has been nagging me for a long time, and I've been slowly trying to chip away at the problems. First, I've been doing work on a Mac port of the liveusb-creator. I also started to work on a kernel patch for getting the EFI framebuffer working, and discussed how to do it with ajax and pjones. The screen doesn't display anything after grub, and since we don't know the base address of the framebuffer, it involves writing code to iterate over memory trying to find some common pixel patterns. I'm still trying to wrap my head around all of it, but I'll probably end up just buying them beer to fix it for me.

Thincrust

Thincrust is a project that I've been excited about for a while, and I actually have some appliances deployed in a production cloud. I was able to run some ideas for various virtual appliances by one of the authors over some beers. Some pre-baked virtual appliances that you can easily throw into a cloud that I would like to see:

dogtail

I'm glad to see that dogtail is still exciting people in the community. It still has a lot of potential to improve not only the way we test graphical software, but we also discussed ways of using it to teach people and automate various desktop tasks. What if you logged in after a fresh install and got the following popup bubble:

Hi, welcome to Fedora, what can I help you do today?

Each task would then allow Fedora to take the wheel and walk the user through various steps. I had this idea a while ago, when dogtail first came out, and I still think it would be totally awesome. Anyway, this was not a focus of the hackfests, but merely a conversation that I had while walking to lunch :)


posted at: 17:49 | link | Tags: , , , , , , , | 4 comments

Thu, 19 Nov 2009

TurboGears2 in Fedora & EPEL

I'm excited to announce that the TurboGears2 web application stack is now available in Fedora 12, 11 and EPEL-5.

What is TurboGears2?

TurboGears 2 is the built on top of the experience of several next generation web frameworks including TurboGears 1 (of course), Django, and Rails. All of these frameworks had limitations which were frustrating in various ways, and TG2 is an answer to that frustration. We wanted something that had:
  • Real multi-database support
  • Horizontal data partitioning (sharding)
  • Support for a variety of JavaScript toolkits, and new widget system to make building ajax heavy apps easier
  • Support for multiple data-exchange formats.
  • Built in extensibility via standard WSGI components

Installing the TurboGears2 stack & development tools

Fedora 12
yum install TurboGears2 python-tg-devtools
Fedora 11
yum --enablerepo=updates-testing install TurboGears2 python-tg-devtools
Red Hat Enterprise Linux 5 (with EPEL)
yum --enablerepo=epel-testing install TurboGears2 python-tg-devtools

Creating your first TG2 app

paster quickstart

Run your test suite

nosetests

Run your application

paster serve development.ini

Read the documentation

http://www.turbogears.org/2.0/docs

Contribute

If you're interested in helping maintain and improve the TG2/Pylons stack within Fedora/EPEL, please let me know. We're always looking for new Python hackers to join the team. There are still a few more components that need to be packaged and reviewed (eg: chameleon.genshi), so please take a look at the TurboGears2 page on the Fedora wiki for more details..


posted at: 06:00 | link | Tags: , , | 1 comments

Tue, 17 Nov 2009

Fedora 12 is here!



Install it with the liveusb-creator!


posted at: 06:00 | link | Tags: , | 2 comments

Mon, 09 Nov 2009

New liveusb-creator release!

So I've gotten some pretty inspiring feedback from various users of the liveusb-creator recently, so I decided to put some cycles into it this weekend and crank out another release.

"As a non-Linux person, Live-USB Creator has improved the quality of my life measurably!" --Dr. Arthur B. Hunkins
Yesterday I released version 3.8.6 of the liveusb-creator. Changes in this release include:
Windows
https://fedorahosted.org/releases/l/i/liveusb-creator/liveusb-creator-3.8.6.zip

Fedora
https://admin.fedoraproject.org/updates/liveusb-creator-3.8.6-1.fc11
https://admin.fedoraproject.org/updates/liveusb-creator-3.8.6-1.fc12

Source
https://fedorahosted.org/releases/l/i/liveusb-creator/liveusb-creator-3.8.6.tar.bz2

Trac
http://liveusb-creator.fedorahosted.org


posted at: 02:39 | link | Tags: , , , | 1 comments

Tue, 13 Oct 2009

Good Python Habits: vim + pyflakes

Here is a neat little hack for running pyflakes on Python files after you save them. I like using pyflakes for quickly catching dumb errors, but you could easily replace it with a more comprehensive tool like pychecker, or pylint for more strict PEP8 compliance.

All you have to do is throw this in your ~/.vimrc

au BufWritePost *.py !pyflakes %

This has saved me *tons* of time and frustration over the past few weeks, and I have no idea I lived without it.


posted at: 13:32 | link | Tags: , , | 3 comments

Sun, 11 Oct 2009

Fedora 12 filesystem showdown


posted at: 23:02 | link | Tags: , , , | 4 comments

Tue, 14 Jul 2009

Fedora 9 Updates Metrics

Fedora 9 Updates


Most updated packages


Packages with the best karma


Most updates per developer


Most tested packages


Top testers


posted at: 17:00 | link | Tags: , , | 18 comments

Fri, 10 Jul 2009

Bodhi EPEL support!

It's been a long time coming, but the Extra Packages for Enterprise Linux (EPEL) project is finally utilizing the Koji build system and the Bodhi updates system.

So I spent the past week hacking on EPEL support in bodhi. This was not a trivial task, and took more work than expected. Overall, it was a very beneficial experience, as I was able to hack in some higher level abstractions and also remove a lot of Fedora-specific assumptions in the code. Most of the changes were what I would normally call "hacks", mainly because I wanted to do it without changing the database schema. However, this gives me a much clearer picture as to what we need from the Bodhi v2.0 model. Anyway, 28 bodhi upgrades later, and everything seems to be working fine.

The inevitable TurboGears2 rewrite/port of Bodhi is a little further down the road. I've already ported the original model from SQLObject to SQLAlchemy, but the templates, controllers, and widgets still need to be ported. If you're interested in helping make bodhi suck less, then please come talk to me :)

Anyway, developers can now submit their EPEL updates here, or by running `make update` in their EL CVS branches. Admins can read the Bodhi SOP to learn how to push updates.

The workflow is far from perfect, but there has been some recent discussions as to how we want EPEL to be treated differently compared to Fedora updates. If you have suggestions or comments, discussions should take place on epel-devel-list.


posted at: 19:06 | link | Tags: , , | 2 comments

Sun, 14 Jun 2009

Fedora Activity Day: Fedora Development Cycle 2009

Earlier this week I ventured to Raleigh for a Fedora Activity Day, where a dozen or so people brainstormed for a few days about ways to improve our development process. I feel that it was a very productive experience, as it allowed everyone to list the things that could use improvements, without getting caught up in the technical or political details. We hashed out a huge list of items in Gobby , prioritized them, and discussed possible solutions for as many as we could. A good number of proposals came out of it:


posted at: 00:24 | link | Tags: | 0 comments

Tue, 27 Jan 2009

bodhi updates push process

Bodhi's push process is something that is usually quite opaque to Fedora package maintainers. Once an update request goes into bodhi, the developer sits back and waits for the update to go to where it needs to go. The ball is then in releng's court, as they must sign the packages, and tell bodhi to begin the push. From there, bodhi does it's thing for a while, and then updates magically end up on our users machines. Yay!

Pushing updates used to take the better part of a day, mostly due to dumb code and lots of filesystem churn over NFS. Thankfully, a lot of the code is now much smarter, and people like jkeating and mmcgrath have been helping to address the NFS & infrastructure bottlenecks.

Hopefully I can help shed some light on one of the dark corners of bodhi known as The Masher. Here are some statistics of the last updates push that happened earlier today.

Initial push request from releng
Check koji tag / bodhi status consistency38s
Move all of the build tags in Koji 9m32s
Update the comps CVS module 11s
Mash f9-updates-testing 4m16s
Mash f9-updates 1h3m8s
Mash f10-updates-testing 12m43s
Mash f10-updates 37m51s
Set update ids, state modifications, updates-testing digest generation 1m57s
Generate updateinfo.xml 5m55s
Repo sanity checks & symlinking to go live 1m4s
Cache latest repodata, and remove old 1m14s
Wait for updates to hit the master mirror 1h1s
Send update notices, update/close bugs, notify developers/commenters 11m11s
Total3h49m42s


So we've obviously made some great improvements here, and once the signing server is deployed, you can probably expect a much more frequent/consistent flow of updates. However, I definitely think there is still a lot of low-hanging fruit in this process, and many steps can probably be done in parallel. We're going to be adding DeltaRPM generation into the mix in the near future, so I'll give an update a bit later with some details as to how that effects the process.

Anyway... if you know Python, and enjoy optimizing code -- come talk to me :)


posted at: 05:15 | link | Tags: , , , | 0 comments

Wed, 31 Dec 2008

liveusb-creator 3.0

I'm pleased to announce version 3.0 of the liveusb-creator. Aside from the usual batch of bug fixes and code improvements, this release also contains a variety of enhancements:

For Fedora 9 and 10 users, you can currently find the liveusb-creator-3.0 in the updates-testing repository. Feedback is appreciated!

There are still many more great features in the pipeline, so stay tuned!


posted at: 23:19 | link | Tags: , , | 2 comments