Wed, 16 Feb 2011

FUDCon 2011 Tempe

I meant to get this summary out the door weeks ago, however, I didn't want to distract from my post-FUDCon productivity :)

Another FUDCon has come and gone. This time it was in gorgeous Tempe, Arizona. It was my first time in AZ, and I must say that I was thoroughly impressed. It was truly a great location for FUDCon. Thanks to the epic snowstorm of doom, I also was stuck there for a few extra days

Live Widgetry
Before the conference I decided to throw together a little live widget that scrolls all of the new posts tagged with #FUDCon. Thanks to the mad design skillz of mizmo, and the feed aggregation and real-time web sockets of Moksha, I was able to throw it together pretty quickly. I plan on taking this code and integrating it in the existing fedoracommunity dashboard and hooking up many different fedora-related feeds to it.


Day 1 of the FUDCon sessions were quite interesting. I got a chance to learn a bit more about the exciting AutoQA project, which is coming along nicely. You'll be seeing AutoQA commenting on bodhi updates soon enough.

Security Lab
I caught Joerg Simon's session on the Fedora Security Lab. It was exciting to see how the Security Spin has evolved ever since I created it back in 2007 for a project in my forensics class. It was also interesting to learn more about OSSTMM.

The next session I attended was about the future of spins. Almost everyone agreed that Spins are useful and a valuable part of Fedora. The problems seem to mostly lie in governance/policy and a lack of communication and coordination between the Spins SIG and QA/Releng/Infrastructure. Once of my ideas below may help with this a little bit.

The Next Big Project

Day 2 was comprised of more sessions, including my team's "Next Big Project" proposals. Of course, the Fedora RPG that Spot and Mo talked about was definitely a hot topic, and got a lot of people excited.

I talked about a handful of project ideas that I would like to work on in the future (actually, I have code written for most of them already). Here is a quick rundown:

Real-time Infrastructure
I want to see us deploy an AMQP message broker inside our production infrastructure. Then, we hook up all of our existing services and have them fire off messages when various events occur (koji builds, bodhi updates, pkgdb additions/removals/changes, git hooks, planet feeds, wiki edits, etc). From here, using some realtime web technology that we created, we could easily expose these message queues via a live dashboard that lets you filter and navigate the stream of activity, along with providing real-time metrics. We can also create desktop notification widgets, so you can get popup bubbles for things that you care about.

This is also key to the whole RPG as well. In order to build a game based on Fedora workflows, we need an underlying expert system that knows what actions can be taken within fedora, how they are accomplished, and who is getting them done.

Meeting app
Currently after a meeting, our Meetbot spits out the logs and an overview in txt/html/rst to a directory on Trying to track down who agreed to what when, or even to see what a given team has been up to over the past couple of months, is very tedious.

The data is there, now we just need to make it useful. I would love to see a frontend for this sytem that tracked meetings by team/people/topics/projects, kept track of actions and held people accountable for what they say they are going to do (and make it easy for people to say "I need help with this", or "I don't have time to finish this"), making it simple to go from an #idea in a meeting to implementation. There is so much great data in these logs, and I think we can do some awesome things with it.

Improved upstream monitoring
Most Fedora developers probably don't even know that we already have an Upstream release monitoring service buried in the wiki. I added almost every package I maintain to it, and it will automatically open a bug when a new upstream version is released. Extremely useful.

I wouldn't call this a "big" project, but I would like to see us integrate this service into our existing infrastructure. We could potentially store this per-package upstream data in the pkgdb/bodhi, and when a new release comes out write some code to automatically try doing a simple specfile bump, throw a scratch build at koji, run it through AutoQA, queue up for testing in bodhi, etc. Ideally, this would minimize the massive amounts of effort that our maintainers have to do to keep our packages up to speed with upstream.

Discussions app
Last year, Máirín Duffy and I came up with some interesting ideas for improving our mailing lists. I would like to make this a reality. Since then, I have already written code that can successfully parse all of fedora-devel.mbox (sounds much easier than it really is), populate it into a SQLAlchemy database model, expose a JSON API for quering, and visualize threads and various statistics with a basic widget.

Spin Master
One of the biggest problems with spins that we have right now is that there is no easy way to track how they are evolving. I would like to see us create a system that took the nightly spins and analyzed them, tracking what packages have been added/removed, which packages have grown/shrunk, etc. We could potentially get AutoQA involved here and make sure all spins pass a certain level of sanity checks before they can even be released. There are scripts floating around that can do a lot of this already -- but I want to streamline it and build a frontend.

Source-level package diff viewer
Fedora churns at such a fast pace, yet I can only imagine that a small subset of maintainers actually look at the complete code changes between upstream package releases. The recent sourceforge intrusion should be seen as a reality check to us distributions, and I think we need to step up our game quite a bit to ensure we don't let any malicious code slip into Fedora.

I would like to see a web/cli interface for viewing full source diffs of package updates in Fedora, allowing people to annotate/flag lines of code. Having more eyes view the code changes that go into our distribution is definitely a Good Thing, not just for Fedora, but for Open Source in general.


Now, on to my favorite part of any conference -- The Hackfests. First off, I felt that the hackfests were a bit unorganized this year. There was no opportunity to pitch hackfests, and it was not easy to figure out who was doing what in which rooms. Anyway, I had a fairly productive day of hacking...

I integrated our package test cases into bodhi, which will now query the wiki for tests and display them in your updates, like so:

Thanks to James Laska for the code to query the wiki, and to Ian Weller for python-simplemediawiki API.

I also sat down with Máirín Duffy and talked about Bodhi v2.0 interaction design. We discussed what actions we want people to take when they arrive at bodhi's homepage, which essentially boils down to submitting, searching, browsing, and testing updates. Mo quickly threw together an awesome mockup that portrays some of our initial ideas.

Fedora Community Discussions
As mentioned above, I have already started implementing the mailing list interface that mizmo and I designed last year. I worked with Casey Dahlin during the hackfests and helped him get a working fedoracommunity/moksha development environment up and running and become familiar with the existing code.

Even though he wasn't at FUDCon, Jan Hutar has also been working on a couple of great graphs/grids of mailing list statistics for fedoracommunity as well. We'll be hacking away at this stuff over the next few months, so stay tuned.

kernel EFI framebuffer
I spun up a quick kernel patch to enable the EFI framebuffer on a handfull of Macs. I already wrote a patch that got applied upstream that enables this framebuffer on 14 different mac models, but this new patch adds 5 more. A lot of people, especially Sugar on a Stick users, are desperate to get Fedora running on their mactel machines (assuming found in may school labs), so I spun up a fresh livecd with my kernel patch for testing. See Bug #528232 for more information.

I did a bunch of work on porting the liveusb-creator from HAL to UDisks. Thankfully, Ubuntu's cleverly-named "usb-creator" already has UDisks support, so I've been happily borrowing ideas from their code :)

I also had some great discussions with Peter Robinson and Bernie Innocenti about solving the persistent overlay problem with our Live USBs. Right now, they are essentially a ticking time bomb, and real world LiveUSB use-cases are getting bit by this all of the time. Over the past few years, the "solution" has been to "wait for unionfs to get merged into the kernel". However, there are a variety of other potential solutions that we are going to look into as well.

The future of Python web development is extremely exciting, innovative, and still evolving at a rapid pace. As a TurboGears developer, I'm still very impressed with TG2, which along with TG1 will be supported for a long time to come -- but I'm also very eager for the next generation framework that has just emerged.

Recently, the Pylons project has merged with repoze.bfg to form Pyramid, which just released version 1.0. I'm quite amazed by the quality of the code, docs, and tests, and benchmarks already show it blowing rails/django/tg/pylons out of the water. I'm looking forward to the PyCon sprints, where all 3 communitites are going to be sitting at the same table working together on this project.

So while I was stuck in Tempe during the snow storm, I wrote 7 RPMs for Pyramid and it's dependencies, which are currently waiting to be reviewed. I plan on writing Bodhi v2.0 using Pyramid, so if this is something that interests you, let me know (and start reading the 600+ pages of docs ;)

Good times
Technical stuff aside, I had a blast at FUDCon, and I feel like it was one of the best. FUDPub was great, as usual. I played a lot of poker [poorly], ate a lot of tasty food, and had some great conversations. I had caught a cold prior to coming to FUDCon, so instead of nursing it with some nyquil and sleep, I decided to try to nurse it with a bottle of Jack, which turned out to be a bad idea.

Also, no FUDCon would be complete without mentioning Once the hackfests started winding down, I gave a quick session on nethack, where I tought people various ways to steal from shops, using a bunch of screecasts that I had on my laptop.

posted at: 21:55 | link | Tags: , , | 2 comments

Thu, 10 Dec 2009

FUDCon Toronto 2009

Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion.

My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on.


Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later.

One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker.

I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing:

"I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well ... presentation slides found:"

Fedora Community

So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform.

After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen.

During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them).

I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to come. My main concern was a blocker issue that we were hitting with our flot graphs when you quickly bounce between tabs. I ended up "fixing" the bug, so I'll be pushing what we have of the stats branch into production in the near future.


TurboGears has definitely been our favorite web framework within Fedora's Infrastructure for many years now. TurboGears2, a complete re-invention of itself, has been released recently, and is catching on *very* quickly in the community. Tons of people are working on awesome new apps, and loving every minute of it. I was also able to convert a rails hacker over to it, after he was able to quickly dive into one of the tutorials with ease. See my previous blog post about getting up and running with TG2 in Fedora/EPEL.


One of my main tasks during the hackfests was to pull the authentication layer in Fedora Community that authenticates against the Fedora Account System, and port it over to python-fedora, so we can use it in any TurboGears2 application. I committed the initial port to python-fedora-devel, and have started working on integrating it into a default TG2 quickstart and document the process. There are still a couple of minor things I want to fix/clean up before releasing it, so expect a blog about it soon.


It seems like yesterday that I was an intern at Red Hat working on an internal updates system for Fedora Core. Coming up on 5 years later, and I am now working on my 3rd implementation of an updates system, Bodhi v2.0. What's wrong with the current Bodhi you ask? Well, if you talk to any user of it, you'll probably get a pretty long list. Bodhi is the first TurboGears application written & deployed in Fedora Infrastructure, and uses the vanilla components (SQLObject, kid, CherryPy2). The TG1 stack has been holding up quite nicely over the years, and is still supported upstream, but bodhi's current implemention and design does not make it easy to grow.

Bodhi v2.0 will be implemented in TurboGears2, using SQLAlchemy for an ORM, Mako for templates, and ToscaWidgets2 for re-usable widgets. It will be hook-based and plugin-driven, and will be completely distribution agnostic. Another important goal will be AMQP message-bus integration, which will allow other services or users to react to various events inside of the system as they happen.

So far I've ported the old DB model from SQLObject to SQLAlchemy, and have begun porting the old unit tests, and writing new ones. Come the new year, I'll be giving this much more of my focus.

During the hackfests I got a chance to talk to Dennis Gilmore about various improvements that we need to make with regard to the update push process. It was also great to talk to many different users of bodhi, who expressed various concerns, some of which I've already fixed. I also got a chance to talk to Xavier Lamien about deploying Bodhi for rpmfusion. On the bus ride home I helped explain to Mel how Bodhi & Koji fit into the big picture of things.

During the BarCamp sessions I also attended a session about the Update Experience, where we discussed many important issues surrounding updates.


So I got a chance to finally meet Sebastian Dziallas, of Sugar on a Stick fame, and was able to fix a few liveusb-creator issues on his laptop. I ended up pushing out a new release a couple of days ago that contains some of those fixes, along with a new version of Sugar on a Stick.

The liveusb-creator has been catching a lot of press recently (see the front page for a list). Not only did it have a 2 page spread in Linux Format, but it was also featured in this weeks article New Sugar on a Stick Brings Much Needed Improvements. Rock.


There was lot of brainstorming done by Dave Malcolm, Colin Walters, Toshio Kuratomi, Bernie Innocenti, I, and many others about various improvements that we could make to the Python interpreter. From speeding up startup time by doing some clever caching to potentially creating a new optimized compiled binary format. We also looked into how WebError/abrt gather tracebacks, and discussed ways of enabling interactive traceback debugging for vanilla processes, without requiring a layer of WSGI middleware.

There was also work done on adding SystemTap probes to Python, which is very exciting. There are many ideas for various probe points, including one that I blogged about previously.

Intel iMac8,1 support

My iMac sucks at Linux. This has been something that has been nagging me for a long time, and I've been slowly trying to chip away at the problems. First, I've been doing work on a Mac port of the liveusb-creator. I also started to work on a kernel patch for getting the EFI framebuffer working, and discussed how to do it with ajax and pjones. The screen doesn't display anything after grub, and since we don't know the base address of the framebuffer, it involves writing code to iterate over memory trying to find some common pixel patterns. I'm still trying to wrap my head around all of it, but I'll probably end up just buying them beer to fix it for me.


Thincrust is a project that I've been excited about for a while, and I actually have some appliances deployed in a production cloud. I was able to run some ideas for various virtual appliances by one of the authors over some beers. Some pre-baked virtual appliances that you can easily throw into a cloud that I would like to see:


I'm glad to see that dogtail is still exciting people in the community. It still has a lot of potential to improve not only the way we test graphical software, but we also discussed ways of using it to teach people and automate various desktop tasks. What if you logged in after a fresh install and got the following popup bubble:

Hi, welcome to Fedora, what can I help you do today?

Each task would then allow Fedora to take the wheel and walk the user through various steps. I had this idea a while ago, when dogtail first came out, and I still think it would be totally awesome. Anyway, this was not a focus of the hackfests, but merely a conversation that I had while walking to lunch :)

posted at: 17:49 | link | Tags: , , , , , , , | 54 comments