Mon, 16 Jun 2014

Bodhi2 Fedora Activity Day

The Bodhi2/Taskotron Fedora Activity Day happened earlier this month! A bunch of us gathered in Denver for a few days and worked on some of our critical releng & qa infrastructure. The hackfest was held in a conference room in my apartment building, which worked out quite nicely for the amount of people that we had. The hotel was right up the road, and we were able to walk to a lot of awesome spots, like the 1UP Barcade :).

It was great to have folks from various corners of Fedora together in the same room for a few days. As it is, we get a lot done remotely, but being able to leverage the high-bandwidth face-to-face time is extremely valuable, especially when coming to consensus on difficult architectural decisions. We used Gobby to collaborate on a long list of action items, and chipped away most of it. Thankfully, Bodhi has enough layers that we were all able to split up and dive into different corners of the code without stepping on each others' toes.

Up until now, our releng stack in staging has always been less than useful. We've never able to do a full build->update->push->run, and have had to rely on testing certain codepaths in production. Not only that, but Bodhi's "masher" never really had proper unit tests, so pushing out major changes to that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our staging setup and made it so we can actually use it. I made a lot of headway on porting the Bodhi masher to a new fedmsg-driven architecture, while writing unit tests for it along the way. I'm hopeful that we can write tests for every part of the "push" process, and then optimize the hell out of it :)

While Mathieu and I mainly focused on back-end hacking, Ralph and Ricky made some fantastic headway on the front-end. Ralph started working on the revamped New Update Form, which is already significantly better than the original. The idea here is that the maintainer should only need to provide the package name(s), and Bodhi will automatically find the corresponding candidate builds, related bugs, and eventually it will pull in all candidate deps as well (or tell if you if any need to be rebuilt). It would also be very convenient to be able to "push this entire stack again". Ideally, I'd love to make it so that folks maintaining large stacks, like GNOME, shouldn't need to use a Google doc to coordinate their mega-updates ;)

Ralph also started revamping the new karma system (check out his screencast here). We don't have any of the policy in place to enforce it yet, but eventually we'd like the maintainers to be able to define custom policy constraints for their updates. For example, you could say "only allow this update to go to the stable repo once this specific bug or test case has been confirmed as fixed/passing".

Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on the front page using the fedmsg datagrepper widget that Ralph blogged about recently. Other front-end improvements include libravatar support for all users, proper markdown rendering with previews and image support, and of course a konami code easter-egg ;)

I was going to post a bunch of screenshots here, but Ralph just deployed a development instance of Bodhi2 that folks can play around with instead: (it's a dev instance, so don't expect it to always be up/functional).

Some other corners of Bodhi that have seen improvements recently:

The API. The Bodhi webapp is written in Python using the excellent Pyramid web framework. On top of that we are using Cornice, which makes it really easy to build & document standards-compliant web services with Pyramid. Thanks to colander validation/deserialization schemas and our custom ACLs and validators, we are able to write dead-simple views that can safely assume that all of the data we are dealing with is valid and in the right format, and that the user has the appropriate permissions. Cornice also has a Sphinx plugin that auto-generates our API documentation. So not only do we now have a semi-RESTful standards-compliant self-documenting API, but Ralph also added a support for rendering almost every service as an RSS feed as well.

Regarding the Bodhi Python APIs, I've begun porting them to the new python-fedora OpenIDBaseClient (see bodhi/ Since a large percentage of the API usage is through the current python-fedora BodhiClient, I'd like to try our best to maintain compatibility--at least for a version or two with deprecation warnings if we have to. I am really looking forward to finally being able to trash our old TurboGears1 FAS visit/identity layer in favor of FedOAuth.

On top of the Python API lies the bodhi-client. I recently ported the basic functionality over using the click module, which makes it really easy to write complex command-line tools (see bodhi/ Since the current bodhi-client is an absolute mess, this is one area that I'm actually okay with breaking backwards-compatibility to a certain extent. Having a proper command structure, similar to the Koji cli, is well worth some cli flag changes in my opinion.

In a similar fashion, Mathieu implemented a great release management tool for Bodhi admins. Currently, creating a release and changing it's pre-release status involves using the TurboGears Python shell, creating SQLObject instances by hand, editing config files, etc. This tool will make it dead simple for releng to create new releases and manage all of the pre-GA state changes that happen along the way.

Performance was another key area of development. The app is fairly snappy now, but there is still a ton of room for improvement. The pyramid-debugtoolbar has been amazingly useful for us so far. It let's us analyze every SQL statement made, it does full-stack profiling, and it lets us execute commands in every layer of a traceback. Along with that, Ralph added a SQLAlchemy event hook to our unit tests to ensure that certain changes don't drastically change how many SQL statements are getting executed. With regard to Masher performance, there is still a lot of low-hanging fruit there. We saw a drastic boost in mash speed recently when Kevin discovered that a couple of releng machines weren't using virtio. This brought the mash time of EPEL5 updates from 45 minutes down to around 15. In the new version of the masher, updates are grouped by tag and then processed in a separate threads. Security updates and critical bugfixes will take priority over enhancements, and there are also things that we can do to make the bits hit the mirrors faster once we're done mashing.

Another corner of Bodhi that was the topic of discussion was around notifications. Bodhi currently sends way too much email, most of which I'm assuming gets ignored. There are a couple of mails that are a bit more important, like the update announcement emails that get sent to package-announce, and the updates-testing digest that goes to the test-list. The consensus that we came to was that we are going to attempt to use FMN to allow people to configure what messages they want to receive, and where (IRC, email, etc). This alleviates the need to build an email layer into Bodhi2, and allows us to focus on publishing fedmsgs only, letting FMN do the rest.

As far as the transition to Bodhi2 goes, we're going to try our best to not break the world in the process. Ralph mentioned the potential timeline in his blog post, and we still have lots of work to do before then. In order to help ease this transition, I created a wiki page to track the consumers of Bodhi's APIs, so we can make the appropriate changes to those codebases before launch. Please feel free to update it with any that I left out.

Overall, it was a very successful FAD. We got a ton of stuff done, ate a bunch of great food, and had a lot of fun in the process. I didn't cover everything that we worked on, so checkout the blog posts from threebean, bochecha, and nirik for more details on other things that got done. If you're interested in getting involved with Bodhi2, grab the code, checkout the open issues, and hop in #fedora-apps on Freenode.

Also, I'll be giving a presentation at Flock in Prague this year on "Evolving the Fedora Updates process", which will cover the history of pushing updates as well as an in-depth dive into the new bodhi2 stack.

posted at: 16:49 | link | Tags: , , | 1 comments

Tue, 08 Jun 2010

Fedora Updates Report

I recently wrote some code to generate detailed statistics of Fedora & EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics.

This report definitely conveys the shortcomings in how we currently utilize bodhi for "testing" updates, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria, I think we'll see these numbers drastically improve in the future.

You can find the code that generates these statistics here:, If you have any ideas or suggestions for different types of metrics to generate, or if you find any bugs in my code, please let me know.

Bodhi Statistics Report (Generated on June 8th, 2010)

Out of 17412 total updates, 2958 received feedback (16.99%)
Out of 1045 total unique karma submitters, the top 30 are:
 * notting (424)
 * mclasen (366)
 * jkeating (321)
 * adamwill (283)
 * cwickert (161)
 * rdieter (159)
 * pbrobinson (141)
 * kevin (141)
 * cweyl (122)
 * tomspur (119)
 * mtasaka (110)
 * xake (97)
 * cschwangler (86)
 * kwright (84)
 * peter (83)
 * hadess (80)
 * michich (72)
 * tagoh (69)
 * pfrields (69)
 * bpepple (69)
 * iarnell (68)
 * lkundrak (66)
 * shinobi (65)
 * sundaram (64)
 * spot (62)
 * pravins (62)
 * markmc (62)
 * thomasj (61)
 * smooge (60)
 * fab (59)

     Fedora 13

 * 3562 updates
 * 3065 stable updates
 * 427 testing updates
 * 62 pending updates
 * 8 obsolete updates
 * 2371 bugfix updates (66.56%)
 * 745 enhancement updates (20.92%)
 * 89 security updates (2.50%)
 * 357 newpackage updates (10.02%)
 * 410 critical path updates (11.51%)
 * 333 critical path updates approved
 * 1155 updates received feedback (32.43%)
 * 12120 +0 comments
 * 2477 +1 comments
 * 155 -1 comments
 * 595 unique authenticated karma submitters
 * 133 anonymous users gave feedback (1.57%)
 * 2261 out of 3562 updates went through testing (63.48%)
 * 1317 testing updates were pushed *without* karma (58.25%)
 * 21 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 11 days
   * median = 9 days
   * mode = 7 days
 * 4 updates automatically unpushed due to karma (0.11%)
   * 0 of which were critical path updates
 * 231 updates automatically pushed due to karma (6.49%)
   * 2 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2445 packages updated (top 10 shown)
    * selinux-policy: 13
    * jd: 12
    * 12
    * gdb: 12
    * ibus-pinyin: 11
    * nautilus: 10
    * kernel: 10
    * evolution: 9
    * libfm: 9
    * libmx: 9

     Fedora 12

 * 4844 updates
 * 4291 stable updates
 * 371 testing updates
 * 113 pending updates
 * 69 obsolete updates
 * 2905 bugfix updates (59.97%)
 * 1054 enhancement updates (21.76%)
 * 201 security updates (4.15%)
 * 684 newpackage updates (14.12%)
 * 407 critical path updates (8.40%)
 * 960 updates received feedback (19.82%)
 * 16311 +0 comments
 * 1899 +1 comments
 * 554 -1 comments
 * 758 unique authenticated karma submitters
 * 576 anonymous users gave feedback (5.33%)
 * 2873 out of 4844 updates went through testing (59.31%)
 * 2138 testing updates were pushed *without* karma (74.42%)
 * 188 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 14 days
   * median = 13 days
   * mode = 17 days
 * 12 updates automatically unpushed due to karma (0.25%)
   * 4 of which were critical path updates
 * 133 updates automatically pushed due to karma (2.75%)
   * 13 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 2902 packages updated (top 10 shown)
    * qbittorrent: 25
    * gdb: 25
    * selinux-policy: 22
    * kernel: 15
    * xorg-x11-server: 14
    * ibus: 13
    * jd: 13
    * abrt: 11
    * gvfs: 11
    * gtk2: 11

     Fedora 11

 * 6987 updates
 * 6381 stable updates
 * 183 testing updates
 * 99 pending updates
 * 324 obsolete updates
 * 3649 bugfix updates (52.23%)
 * 1566 enhancement updates (22.41%)
 * 350 security updates (5.01%)
 * 1422 newpackage updates (20.35%)
 * 383 critical path updates (5.48%)
 * 729 updates received feedback (10.43%)
 * 23427 +0 comments
 * 1197 +1 comments
 * 448 -1 comments
 * 782 unique authenticated karma submitters
 * 481 anonymous users gave feedback (3.58%)
 * 4129 out of 6987 updates went through testing (59.10%)
 * 3620 testing updates were pushed *without* karma (87.67%)
 * 278 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 15 days
   * median = 14 days
   * mode = 17 days
 * 7 updates automatically unpushed due to karma (0.10%)
   * 0 of which were critical path updates
 * 64 updates automatically pushed due to karma (0.92%)
   * 11 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 3787 packages updated (top 10 shown)
    * libguestfs: 30
    * jd: 24
    * selinux-policy: 23
    * kdebase-workspace: 19
    * kernel: 18
    * gdb: 16
    * dovecot: 16
    * qemu: 16
    * kdebase-runtime: 16
    * kdenetwork: 16

     Fedora EPEL 5

 * 1572 updates
 * 1255 stable updates
 * 198 testing updates
 * 43 pending updates
 * 76 obsolete updates
 * 734 bugfix updates (46.69%)
 * 236 enhancement updates (15.01%)
 * 93 security updates (5.92%)
 * 509 newpackage updates (32.38%)
 * 20 critical path updates (1.27%)
 * 103 updates received feedback (6.55%)
 * 6076 +0 comments
 * 156 +1 comments
 * 19 -1 comments
 * 243 unique authenticated karma submitters
 * 41 anonymous users gave feedback (1.22%)
 * 1176 out of 1572 updates went through testing (74.81%)
 * 1092 testing updates were pushed *without* karma (92.86%)
 * 19 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 24 days
   * median = 18 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 10 updates automatically pushed due to karma (0.64%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 1060 packages updated (top 10 shown)
    * libguestfs: 26
    * znc: 10
    * vrq: 8
    * cherokee: 8
    * 389-ds-base: 8
    * viewvc: 8
    * 389-admin: 7
    * pki-ca: 7
    * wordpress-mu: 7
    * Django: 7

     Fedora EPEL 4

 * 447 updates
 * 359 stable updates
 * 40 testing updates
 * 11 pending updates
 * 37 obsolete updates
 * 222 bugfix updates (49.66%)
 * 68 enhancement updates (15.21%)
 * 40 security updates (8.95%)
 * 117 newpackage updates (26.17%)
 * 5 critical path updates (1.12%)
 * 11 updates received feedback (2.46%)
 * 1592 +0 comments
 * 11 +1 comments
 * 2 -1 comments
 * 85 unique authenticated karma submitters
 * 2 anonymous users gave feedback (0.24%)
 * 320 out of 447 updates went through testing (71.59%)
 * 311 testing updates were pushed *without* karma (97.19%)
 * 5 critical path updates pushed *without* karma
 * Time spent in testing:
   * mean = 18 days
   * median = 16 days
   * mode = 16 days
 * 0 updates automatically unpushed due to karma (0.00%)
   * 0 of which were critical path updates
 * 1 updates automatically pushed due to karma (0.22%)
   * 0 of which were critical path updates
 * Time spent in testing of updates that were pushed by karma:
   * mean = 11 days
   * median = 7 days
   * mode = 7 days
 * Time spent in testing of updates that were unpushed by karma:
   * mean = 9 days
   * median = 5 days
   * mode = 5 days
 * 313 packages updated (top 10 shown)
    * cherokee: 8
    * globus-common: 7
    * R: 6
    * voms: 6
    * globus-gsi-proxy-ssl: 5
    * globus-openssl-module: 5
    * globus-gsi-proxy-core: 5
    * bitlbee: 5
    * flashrom: 5
    * viewvc: 5

posted at: 20:41 | link | Tags: , , , | 1 comments

Mon, 18 Jan 2010

nose 0.11

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module.

lmacken@tomservo ~/bodhi $ nosetests
Ran 96 tests in 725.111s


lmacken@tomservo ~/bodhi $ nosetests --processes=50
Ran 96 tests in 10.915s


Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month

posted at: 22:58 | link | Tags: , , , , | 261 comments

Tue, 29 Dec 2009

RIP Fedora 10

Fedora 10 (Cambridge) (2008-11-25 -- 2009-12-17)


Source: bodhi
Fedora 10 Updates
Most updates per developer in Fedora 10
Most Updated Packages in Fedora 10
Packages with best karma
Top Fedora 10 testers
Most tested Fedora 10 packages


Source: fedoracommunity (upcoming release)

Torrent NameNumber of completed downloads
Fedora-10-i386-DVD 112,807
Fedora-10-x86_64-DVD 65,965
Fedora-10-i386-CDs 10,621
Fedora-10-ppc-DVD 6,851
Fedora-10-source-DVD 3,740
Fedora-10-x86_64-CDs 3,141
Fedora-10-ppc-CDs 1,336
Fedora-10-i686-AOS 666
Fedora-10-source-CDs 662
Fedora-10-i686-Live 599
Fedora-10-x86_64-Live 336
Fedora-10-x86_64-AOS 274
Fedora-10-i686-Live-KDE 201
Fedora-10-x86_64-Live-KDE 78
Fedora-10-i686-Live-XFCE 37
Fedora-10-i686-Live-Developer 13
Fedora-10-i686-Live-FEL 12
Fedora-10-x86_64-Live-XFCE 5
Fedora-10-i686-Live-broffice 3
Fedora-10-x86_64-Live-Developer 3
Fedora-10-x86_64-Live-FEL 2
Fedora-10-x86_64-Live-edu-math 1
Fedora-10-i686-Live-edu-math 1
Fedora-10-x86_64-Live-broffice 0

Total 207,354

Yum Data

Source: wiki/Legacy_statistics
Connections to yum
Week Dates New Unique IPsTotal Unique IPsTotal compared to F9
1 2008-11-25 -- 2008-12-0167,421 67,421 73%
2 2008-12-02 -- 2008-12-0881,674 149,095 97%
3 2008-12-09 -- 2008-12-1560,759 209,854 97%
4 2008-12-16 -- 2008-12-2262,527 272,381 93%
5 2008-12-23 -- 2008-12-2968,375 340,756 97%
6 2008-12-30 -- 2009-01-0573,585 414,341 97%
7 2009-01-06 -- 2009-01-1294,166 508,507 103%
8 2009-01-13 -- 2009-01-1985,557 594,064 106%
9 2009-01-20 -- 2009-01-2687,678 681,742 107%
10 2009-01-27 -- 2009-02-0291,014 772,756 110%
11 2009-02-03 -- 2009-02-0995,238 867,994 113%
12 2009-02-10 -- 2009-02-1695,967 963,961 115%
13 2009-02-17 -- 2009-02-23109,800 1,073,761 115%
14 2009-02-24 -- 2009-03-0285,246 1,159,007 --
15 2009-03-03 -- 2009-03-09100,610 1,259,617 --
16 2009-03-10 -- 2009-03-16100,323 1,359,940 --
17 2009-03-17 -- 2009-03-23100,819 1,460,759 --
18 2009-03-24 -- 2009-03-30102,843 1,563,602 --
19 2009-03-31 -- 2009-04-06101,978 1,665,580 136%
20 2009-04-07 -- 2009-04-1399,586 1,765,166 --
21 2009-04-14 -- 2009-04-20101,808 1,866,974 --
22 2009-04-21 -- 2009-04-27100,230 1,967,177 --
23 2009-04-28 -- 2009-05-0497,584 2,064,761 --
24 2009-05-05 -- 2009-05-1195,923 2,160,684 137%
25 2009-05-12 -- 2009-05-1895,632 2,256,316 --
26 2009-05-19 -- 2009-05-2592,377 2,348,693 --
27 2009-05-26 -- 2009-06-0191,747 2,440,440 --
28 2009-06-02 -- 2009-06-0891,513 2,531,953 --

Direct downloads

Source: wiki/Legacy_statistics
The following table shows the number of direct downloads of Fedora 10 media from unique IP addresses, as shown in the web proxy logs. The actual number of raw downloads tends to be much higher.
Week Dates Downloads this week Total downloads
1 2008-11-25 -- 2008-12-01 236,886 236,886
2 2008-12-02 -- 2008-12-08 105,994 342,880
3 2008-12-09 -- 2008-12-15 83,740 426,620
4 2008-12-16 -- 2008-12-22 76,982 503,602
5 2008-12-23 -- 2008-12-29 66,351 569,953
6 2008-12-30 -- 2009-01-05 65,102 635,055
7 2009-01-06 -- 2009-01-12 72,729 707,784
8 2009-01-13 -- 2009-01-19 73,301 781,085
9 2009-01-20 -- 2009-01-26 72,082 853,167
10 2009-01-27 -- 2009-02-02 71,788 924,955
11 2009-02-03 -- 2009-02-09 72,529 997,484
12 2009-02-10 -- 2009-02-16 69,071 1,066,555
13 2009-02-17 -- 2009-02-23 69,216 1,135,771
14 2009-02-24 -- 2009-03-02 67,669 1,203,440
15 2009-03-03 -- 2009-03-09 66,666 1,270,106
16 2009-03-10 -- 2009-03-16 65,524 1,335,630
17 2009-03-17 -- 2009-03-23 63,218 1,398,848
18 2009-03-24 -- 2009-03-30 62,930 1,461,778
19 2009-03-31 -- 2009-04-06 59,813 1,521,591
20 2009-04-07 -- 2009-04-13 57,102 1,578,693
21 2009-04-14 -- 2009-04-20 55,871 1,634,564
22 2009-04-21 -- 2009-04-27 55,117 1,689,681
23 2009-04-28 -- 2009-05-04 50,815 1,740,496
24 2009-05-05 -- 2009-05-11 48,139 1,788,635
25 2009-05-12 -- 2009-05-18 47,813 1,836,448
26 2009-05-19 -- 2009-05-25 46,077 1,882,525
27 2009-05-26 -- 2009-06-01 44,969 1,927,494
28 2009-06-02 -- 2009-06-08 44,835 1,972,329

posted at: 00:05 | link | Tags: , , , | 0 comments

Thu, 10 Dec 2009

FUDCon Toronto 2009

Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion.

My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on.


Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later.

One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker.

I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing:

"I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well ... presentation slides found:"

Fedora Community

So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform.

After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen.

During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them).

I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to come. My main concern was a blocker issue that we were hitting with our flot graphs when you quickly bounce between tabs. I ended up "fixing" the bug, so I'll be pushing what we have of the stats branch into production in the near future.


TurboGears has definitely been our favorite web framework within Fedora's Infrastructure for many years now. TurboGears2, a complete re-invention of itself, has been released recently, and is catching on *very* quickly in the community. Tons of people are working on awesome new apps, and loving every minute of it. I was also able to convert a rails hacker over to it, after he was able to quickly dive into one of the tutorials with ease. See my previous blog post about getting up and running with TG2 in Fedora/EPEL.


One of my main tasks during the hackfests was to pull the authentication layer in Fedora Community that authenticates against the Fedora Account System, and port it over to python-fedora, so we can use it in any TurboGears2 application. I committed the initial port to python-fedora-devel, and have started working on integrating it into a default TG2 quickstart and document the process. There are still a couple of minor things I want to fix/clean up before releasing it, so expect a blog about it soon.


It seems like yesterday that I was an intern at Red Hat working on an internal updates system for Fedora Core. Coming up on 5 years later, and I am now working on my 3rd implementation of an updates system, Bodhi v2.0. What's wrong with the current Bodhi you ask? Well, if you talk to any user of it, you'll probably get a pretty long list. Bodhi is the first TurboGears application written & deployed in Fedora Infrastructure, and uses the vanilla components (SQLObject, kid, CherryPy2). The TG1 stack has been holding up quite nicely over the years, and is still supported upstream, but bodhi's current implemention and design does not make it easy to grow.

Bodhi v2.0 will be implemented in TurboGears2, using SQLAlchemy for an ORM, Mako for templates, and ToscaWidgets2 for re-usable widgets. It will be hook-based and plugin-driven, and will be completely distribution agnostic. Another important goal will be AMQP message-bus integration, which will allow other services or users to react to various events inside of the system as they happen.

So far I've ported the old DB model from SQLObject to SQLAlchemy, and have begun porting the old unit tests, and writing new ones. Come the new year, I'll be giving this much more of my focus.

During the hackfests I got a chance to talk to Dennis Gilmore about various improvements that we need to make with regard to the update push process. It was also great to talk to many different users of bodhi, who expressed various concerns, some of which I've already fixed. I also got a chance to talk to Xavier Lamien about deploying Bodhi for rpmfusion. On the bus ride home I helped explain to Mel how Bodhi & Koji fit into the big picture of things.

During the BarCamp sessions I also attended a session about the Update Experience, where we discussed many important issues surrounding updates.


So I got a chance to finally meet Sebastian Dziallas, of Sugar on a Stick fame, and was able to fix a few liveusb-creator issues on his laptop. I ended up pushing out a new release a couple of days ago that contains some of those fixes, along with a new version of Sugar on a Stick.

The liveusb-creator has been catching a lot of press recently (see the front page for a list). Not only did it have a 2 page spread in Linux Format, but it was also featured in this weeks article New Sugar on a Stick Brings Much Needed Improvements. Rock.


There was lot of brainstorming done by Dave Malcolm, Colin Walters, Toshio Kuratomi, Bernie Innocenti, I, and many others about various improvements that we could make to the Python interpreter. From speeding up startup time by doing some clever caching to potentially creating a new optimized compiled binary format. We also looked into how WebError/abrt gather tracebacks, and discussed ways of enabling interactive traceback debugging for vanilla processes, without requiring a layer of WSGI middleware.

There was also work done on adding SystemTap probes to Python, which is very exciting. There are many ideas for various probe points, including one that I blogged about previously.

Intel iMac8,1 support

My iMac sucks at Linux. This has been something that has been nagging me for a long time, and I've been slowly trying to chip away at the problems. First, I've been doing work on a Mac port of the liveusb-creator. I also started to work on a kernel patch for getting the EFI framebuffer working, and discussed how to do it with ajax and pjones. The screen doesn't display anything after grub, and since we don't know the base address of the framebuffer, it involves writing code to iterate over memory trying to find some common pixel patterns. I'm still trying to wrap my head around all of it, but I'll probably end up just buying them beer to fix it for me.


Thincrust is a project that I've been excited about for a while, and I actually have some appliances deployed in a production cloud. I was able to run some ideas for various virtual appliances by one of the authors over some beers. Some pre-baked virtual appliances that you can easily throw into a cloud that I would like to see:


I'm glad to see that dogtail is still exciting people in the community. It still has a lot of potential to improve not only the way we test graphical software, but we also discussed ways of using it to teach people and automate various desktop tasks. What if you logged in after a fresh install and got the following popup bubble:

Hi, welcome to Fedora, what can I help you do today?

Each task would then allow Fedora to take the wheel and walk the user through various steps. I had this idea a while ago, when dogtail first came out, and I still think it would be totally awesome. Anyway, this was not a focus of the hackfests, but merely a conversation that I had while walking to lunch :)

posted at: 17:49 | link | Tags: , , , , , , , | 64 comments

Fri, 10 Jul 2009

Bodhi EPEL support!

It's been a long time coming, but the Extra Packages for Enterprise Linux (EPEL) project is finally utilizing the Koji build system and the Bodhi updates system.

So I spent the past week hacking on EPEL support in bodhi. This was not a trivial task, and took more work than expected. Overall, it was a very beneficial experience, as I was able to hack in some higher level abstractions and also remove a lot of Fedora-specific assumptions in the code. Most of the changes were what I would normally call "hacks", mainly because I wanted to do it without changing the database schema. However, this gives me a much clearer picture as to what we need from the Bodhi v2.0 model. Anyway, 28 bodhi upgrades later, and everything seems to be working fine.

The inevitable TurboGears2 rewrite/port of Bodhi is a little further down the road. I've already ported the original model from SQLObject to SQLAlchemy, but the templates, controllers, and widgets still need to be ported. If you're interested in helping make bodhi suck less, then please come talk to me :)

Anyway, developers can now submit their EPEL updates here, or by running `make update` in their EL CVS branches. Admins can read the Bodhi SOP to learn how to push updates.

The workflow is far from perfect, but there has been some recent discussions as to how we want EPEL to be treated differently compared to Fedora updates. If you have suggestions or comments, discussions should take place on epel-devel-list.

posted at: 19:06 | link | Tags: , , | 4 comments

Tue, 27 Jan 2009

bodhi updates push process

Bodhi's push process is something that is usually quite opaque to Fedora package maintainers. Once an update request goes into bodhi, the developer sits back and waits for the update to go to where it needs to go. The ball is then in releng's court, as they must sign the packages, and tell bodhi to begin the push. From there, bodhi does it's thing for a while, and then updates magically end up on our users machines. Yay!

Pushing updates used to take the better part of a day, mostly due to dumb code and lots of filesystem churn over NFS. Thankfully, a lot of the code is now much smarter, and people like jkeating and mmcgrath have been helping to address the NFS & infrastructure bottlenecks.

Hopefully I can help shed some light on one of the dark corners of bodhi known as The Masher. Here are some statistics of the last updates push that happened earlier today.

Initial push request from releng
Check koji tag / bodhi status consistency38s
Move all of the build tags in Koji 9m32s
Update the comps CVS module 11s
Mash f9-updates-testing 4m16s
Mash f9-updates 1h3m8s
Mash f10-updates-testing 12m43s
Mash f10-updates 37m51s
Set update ids, state modifications, updates-testing digest generation 1m57s
Generate updateinfo.xml 5m55s
Repo sanity checks & symlinking to go live 1m4s
Cache latest repodata, and remove old 1m14s
Wait for updates to hit the master mirror 1h1s
Send update notices, update/close bugs, notify developers/commenters 11m11s

So we've obviously made some great improvements here, and once the signing server is deployed, you can probably expect a much more frequent/consistent flow of updates. However, I definitely think there is still a lot of low-hanging fruit in this process, and many steps can probably be done in parallel. We're going to be adding DeltaRPM generation into the mix in the near future, so I'll give an update a bit later with some details as to how that effects the process.

Anyway... if you know Python, and enjoy optimizing code -- come talk to me :)

posted at: 05:15 | link | Tags: , , , | 0 comments

Sat, 13 Dec 2008

Time spent in updates-testing purgatory

Will Woods asked me on IRC earlier today how easy it would be to determine the amount of time Fedora updates spend in testing within bodhi. It turned out to be fairly easy to calculate, so I thought I would share the code and results.

from datetime import timedelta
from bodhi.model import PackageUpdate

deltas = []
occurrences = {}
accumulative = timedelta()

for update in
    for comment in update.comments:
        if comment.text == 'This update has been pushed to testing':
            for othercomment in update.comments:
                if othercomment.text == 'This update has been pushed to stable':
                    delta = othercomment.timestamp - comment.timestamp
                    occurrences[delta.days] = occurrences.setdefault(delta.days, 0) + 1
                    accumulative += deltas[-1]

all =
percentage = int(float(len(deltas)) / float(all) * 100)
mode = sorted(occurrences.items(), cmp=lambda x, y: cmp(x[1], y[1]))[-1][0]

print "%d out of %d updates went through testing (%d%%)" % (len(deltas), all, percentage)
print "mean = %d days" % (accumulative.days / len(deltas))
print "median = %d days" % deltas[len(deltas) / 2].days
print "mode = %d days" % mode

4878 out of 10829 updates went through testing (45%)
mean = 17 days
median = 11 days
mode = 6 days

So, it seems that the majority of updates leave updates-testing in less than a week. This is interesting when taking into consideration the testing workflow mechanisms that bodhi employs. An update can go from testing to stable in two ways: 1) The update's karma can reach an optional stable threshold, and automatically get pushed to the stable repository based on positive community feedback. 2) The developer can request that the update be marked as stable. After an update sits in testing for two weeks, bodhi will send the developer nagmail, which seems to help mitigate stale updates. When initially deploying bodhi, I thought that we would get bogged down with a ton of stale testing updates and would have to implement a timeout to have them automatically get marked as stable. This is still a viable option (which would require FESCo rubberstamping), but I'm quite surprised to see how effective this community-driven workflow is already. Now we just need to encourage more people to use it :)

Due to the limitations of the current model I couldn't figure out an easy way to determine which updates were marked as stable by positive community feedback. This issue will be assessed with the long-awaited SQLAlchemy port that I will hopefully finish up at some point early next year.

posted at: 08:13 | link | Tags: , , , | 3 comments

Wed, 10 Sep 2008

bodhi 0.5!

As many have already noticed, I performed a large bodhi upgrade recently. A few weeks ago, during The Incident, I was forced to perform what was originally going to be a week long bodhi migration and upgrade, overnight. During the past two weeks I've pushed out 24 revisions of bodhi to our infrastructure, fixing various show-stoppers, and helping to make sure that updates got out the door.

One of the most noticable changes is that bodhi is much more responsive. Previously, bodhi was a single python process, running on a single server. This single server was also responsible for composing the updates repositories, and rawhide, among lots of other bodhi-related churn. This lead to much pain and suffering for all.

The bodhi deployment has since changed. All bodhi requests are now load balanced to a bunch of app servers, each running mod_wsgi with multiple bodhi processes, each with multiple threads. All of the hard work is now done on an isolated releng server. This separate bodhi "masher" is now responsible for composing repositories, updating bugs, generating update notices, sending emails, extended metadata generation, and calculating metrics. I also added support for inter-bodhi communication, which allows our bodhi web frontends to kick off push requests to our bodhi-masher instance.

Some of the new features in this release:

Bodhi is far from being feature complete. Some new features in the pipeline:

As always,

Also, if you're currently having issues with the bodhi client, a fixed version will be going out with the next batch of updates. For the impatient, you can pull fixed versions from koji (also, make sure your Makefile.common is up to date):

    koji download-build --arch=noarch python-fedora-0.3.5-1.fc10
    koji download-build --arch=noarch bodhi-0.5.2-1.fc9

posted at: 21:20 | link | Tags: , , | 1 comments

Mon, 16 Jun 2008

Fedora 7 Update Metrics

Fedora 7 reached End of Life on Friday, June 13th. Here are some graphs that I generated with bodhi.

posted at: 20:24 | link | Tags: , , | 2 comments

Sun, 09 Dec 2007

Fedora update metrics

Using flot, a plotting library for jQuery, I threw together some shiny metrics for bodhi. It's pretty amazing to see how a Fedora release evolves over time, with almost as many enhancements as bugfixes. This could arguably be a bad thing, as our "stable" bits seem to change so much; but it definitely shows how much innovation is happening in Fedora.

I should also note that the data on the graphs may look different than the numbers you see next to each category in the bodhi menu. This is due to the fact that updates may contain multiple builds, and the graphs account for all builds in the system.

When I get some free cycles I'd like to generate some metrics from the old updates system for FC4-FC6. I can imagine that the differences will be pretty drastic, considering how the old updates tool was internal to Red Hat, and that the majority of our top packagers are community folks.

posted at: 01:05 | link | Tags: , , , , , | 2 comments

Tue, 20 Nov 2007

make update

I updated Fedora's Makefile.common to support an 'update' target, which has now been comitted. This gives developers the ability to push an update into bodhi with a single command.

Once your package has been built, jump into your branch and do the following:

[lmacken@tomservo F-8]$ make update
This will drop you into a template where you can specify various fields. Any bug numbers mentioned in the most recent RPM ChangeLog will be pre-populated as well.
# [ nethack-3.4.3-15.fc8 ]
# type=[S|B|E] (S=security, B=bugfix, E=enhancement) (required)
# request=[T|S] (T=testing, S=stable) (default: testing)
# bug=123,456
# all other text will be considered to be part of the update notes

Make and save your changes, and then your update will be submitted to bodhi.

Creating new update for nethack-3.4.3-15.fc8
Update successfully created

    Release: Fedora 8
     Status: pending
       Type: bugfix
      Karma: 0
    Request: testing
       Bugs: 221948 - nethack-recover looks in the wrong directory
           : 221692 - Nethack font warning
  Submitter: lmacken
  Submitted: 2007-11-20 14:25:55

See the UpdatingPackageHowTo for more information on updating your packages. You can find more details on the bodhi console client here.

posted at: 06:00 | link | Tags: , | 2 comments

Tue, 13 Nov 2007

bodhi command-line client

The bodhi-client package should be making its way to an updates-testing repository near you! Not only does this command-line tool give developers easier access to bodhi, but also provides some new features to help people get more involved with testing updates and providing useful feedback.

I wrote up some documentation on various usage examples of the tool, which can be found on the bodhi wiki. I also submitted a Makefile.common patch that, once applied, will allow you to run `make update` from your package branch. This will drop you into a new update template, and will then submit your update straight to bodhi.

Some noteworthy features in the bodhi-client, aside from the normal bodhi functionality:

I also upgraded our production bodhi instance yesterday, which pulled in a ton of bugfixes and some new features, such as:

As always, patches/questions/criticisms/comments are welcome. You can file tickets in the usual place. Happy hacking!

posted at: 15:42 | link | Tags: , , | 3 comments

Mon, 01 Oct 2007

Use your Nose!

Every programmer out there [hopefully] knows that unittests are an essential part of any growing body of code, especially in the open source world. However, most hackers out either never write test cases (let alone comments), or usually put them off until "later" (aka: never). Having to deal with Java and JUnit tests in college not only made me not want to write unit tests, but it made me want to kill myself and everyone around me. Thankfully, I learned Python.

So, I just happen to maintain a piece of software in Fedora called nose (which lives in the python-nose package). Nose is a discovery-based unittest extension for Python, and is also a part of the TurboGears stack. If you're hacking on a TurboGears project, the turbogears.testutil module provides some incredibly useful features that make writing tests powerfully trivial.

For example, in the code below (taken from bodhi), I create a test case that utilizes a fresh SQLite database in memory. Inheriting from the the testutil.DBTest parent class, this database will be created and torn down automagically before and after each test case is run -- ensuring that my tests are executed in complete isolation. With this example, I wrote a test case to ensure that unauthenticated people cannot create a new update.

import urllib, cherrypy
from turbogears import update_config, database, testutil, url

update_config(configfile='dev.cfg', modulename='bodhi.config')

class TestControllers(testutil.DBTest):

    def test_unauthenticated_update(self):
        params = {
                'builds'  : 'TurboGears-',
                'release' : 'Fedora 7',
                'type'    : 'enhancement',
                'bugs'    : '1234 5678',
                'cves'    : 'CVE-2020-0001',
                'notes'   : 'foobar'
        path = url('/save?' + urllib.urlencode(params))
        testutil.createRequest(path, method='POST')
        assert "You must provide your credentials before accessing this resource." in cherrypy.response.body[0]
In the above example, the TestControllers class is automatically detected by nose, which then executes each method that begins with the word 'test'. To run your unittests, just type 'nosetests'.
[lmacken@tomservo bodhi]$ nosetests
Ran 33 tests in 16.798s

Now, for the fun part. Nose comes equipped with a profiling plugin that will profile your test cases using Python's hotshot module. So, I went ahead and added a 'profile' target to bodhi's Makefile:
    nosetests --with-profile
    python -c "import hotshot.stats ; stats = hotshot.stats.load('') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
Now, typing 'make profile' will execute and profile all of our unit tests, and spit out the top 20 method calls -- ordered by internal time and call count.
[lmacken@tomservo bodhi]$ make profile
nosetests --with-profile
Ran 33 tests in 42.878s

python -c "import hotshot.stats ; stats = hotshot.stats.load('') ; stats.sort_stats('time', 'calls') ; stats.print_stats(20)"
         800986 function calls (702850 primitive calls) in 42.878 CPU seconds

   Ordered by: internal time, call count
   List reduced from 3815 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       14   13.675    0.977   13.675    0.977 /usr/lib/python2.5/
       31   10.683    0.345   10.683    0.345 /usr/lib/python2.5/
2478/2429    9.297    0.004    9.677    0.004 :1()
        1    0.604    0.604    0.604    0.604 /usr/lib/python2.5/
     2999    0.536    0.000    0.539    0.000 /usr/lib/python2.5/site-packages/sqlobject/sqlite/
   105899    0.448    0.000    0.773    0.000 Modules/pyexpat.c:871(Default)
       60    0.327    0.005    1.102    0.018 /usr/lib/python2.5/site-packages/kid/
   105899    0.325    0.000    0.325    0.000 /usr/lib/python2.5/site-packages/kid/
     3396    0.280    0.000    0.420    0.000 /usr/lib/python2.5/site-packages/cherrypy/
     2965    0.263    0.000    0.263    0.000 /usr/lib/python2.5/logging/
44964/6587    0.238    0.000    0.252    0.000 /usr/lib/python2.5/site-packages/kid/
       60    0.116    0.002    0.116    0.002 /usr/lib/python2.5/site-packages/kid/
     8127    0.114    0.000    0.114    0.000 /usr/lib/python2.5/site-packages/cherrypy/
     8982    0.110    0.000    0.137    0.000 /usr/lib/python2.5/site-packages/sqlobject/
13740/4044    0.108    0.000    2.176    0.001 /usr/lib/python2.5/site-packages/kid/
24353/4026    0.107    0.000    2.143    0.001 /usr/lib/python2.5/site-packages/kid/
     3170    0.093    0.000    0.398    0.000 /usr/lib/python2.5/logging/
        1    0.082    0.082    0.082    0.082 /usr/lib/python2.5/site-packages/rpm/
     4777    0.081    0.000    1.320    0.000 /usr/lib/python2.5/site-packages/kid/
  759/176    0.074    0.000    0.210    0.001 /usr/lib/python2.5/

posted at: 14:40 | link | Tags: , , , , | 425 comments

Sat, 15 Sep 2007

bodhi 0.2.0

I'm pleased to announce that bodhi 0.2.0 has been released and deployed. This release has fixed a ton of issues and introduces many new features, such as:

This release introduces many database changes from the previous version, so it will be much easier to jump back into the release-early-release often cycle.

Soon to come:

I would also like to thank Till Maas and Tim Lauridsen for taking the time to help out and do some great work.

There is still much to be done with bodhi, so if you're interested in helping out, you can setup a local bodhi development playground with just a few commands and dive in.

As always, please file any bugs or enhancement requests here.

posted at: 15:44 | link | Tags: , , | 2 comments

Thu, 16 Aug 2007

Bodhi Presentation at Virtual FUDCon

I'm going to be giving a presentation on Bodhi at the Virtual FUDCon August 15th at 1900-2000 UTC (3pm EST). I'll be covering the past, present, and future of Fedora updates, as well as diving into some technical aspects of the Bodhi project. If all goes to plan, this will be a live audio-enabled presentation, where people can follow the slides as well as converse on IRC. I try to post my slides, and [hopefully] a recording of the presentation next week.

*Update* You can find the IRC transcripts from my presentation here, and my slides here.

posted at: 04:49 | link | Tags: , , | 1 comments

Thu, 15 Feb 2007


So I've been spending the majority of my free time recently getting the new Fedora Updates System ready for Fedora 7, which is quickly approaching. Since the code is going to exist in multiple instances for different projects (Fedora, RH, etc), and is already fairly modular and distro-independent, I decided it would be best to give the code a new name, and a new home: Bodhi emerged. Being hardcore into Zen recently, I feel like the name seems to fit the goals of the project nicely.

A Buddhist term for the wisdom by which one attains enlightenment. Bodhi is the opposite of ignorance, the insight into reality which destroys mental afflictions and brings peace.

Previously, the updates system code was in CVS, and bugs were to be filed in OTRS, which ended up being extremely painful to work with. After switching to our new shiny hosting setup, Bodhi is now using Mercurial for source control, and Trac for managing the project milestones, bugs, and documentation. Overall, I've been extremely impressed with Mercurial and Trac so far.

Here are some screenshots of the current development version. More screenshots can be found on the bodhi wiki.

So what's going to change for Fedora developers? Well, currently any [extras] developer can push out updates to their packages for any release with ease (cvs commit && make tag build). If your updated package fixes 4 bugs and 2 CVE's, the end user who is blindly installing this update has no idea. Not only is there no differentiation between bugfix/enhancement/security updates, but none of them go through any sort of QA whatsoever. This is what bodhi is aiming to change.

I've been designing this system completely independent of any Buildsystem (mainly because the unveiling of brew is still churning slowly), so the process of building packages is going to remain the same (for now). Once your packages is built, you will enter it into bodhi via a web form, or [eventually] a command-line tool (Ideally, I'd like to see the process of preparing/testing/releasing a package to be a single point of interaction; either completely from command-line or in web interface). From here, your update will undergo various checks (and will eventually dispatch tests to the beaker test lab). If all goes well, your update will get signed and pushed, all referenced Bugzillas will be updated/closed, and an update notification mail will be sent to the appropriate list.

updates-testing is currently our "testing" repository that developers can choose to push their updates out to before going to 'final'. This is not very appealing for devs, as it requires them to 'Move to final' after a certain amount of time when they feel it is necessary, and the system will nag them if they don't. The updates-testing repo is not widely used, and gives testers no incentive or way to give feedback, other than filing a bug. One of the goals of bodhi is to require all updates to go through the updates-testing process, and provide a simple interface for developers/testers/users to provide positive/negative feedback regarding an update, and also make it trivial to submit bugs about them as well (we threw around some ideas about making a reusable QA feedback widget for this at a recent Fedora QA meeting). After a certain number of positive responses (or after a given length of time with no negative responses), an update will then able to make it's way to the stable updates repository.

I've written a ton of code so far, but there is still much work to be done to accomplish all of the goals mentioned above. I layed out a few milestones for bodhi, 1.0 being the minimal functionality needed for the release of Fedora 7. I also added a 1.1 milestone for features that aren't crucial for minimal functionality, but are definitely important, and also a wishlist milestone for features that might be nice to have someday (metrics, rss feeds, etc).

With FUDCon and the Fedora Infrastructure HackFest coming up, I'm encouraging anyone interested to dive in help out in any way they want. If you can't make it out to Boston, just hop on IRC. If you would like to see a feature in bodhi, feel free to add a ticket to any of the milestones.

This system holds much potential for revolutionizing the way Fedora releases evolve, and how developers and testers interact with it. Since this is going to be utilized by all package maintainers, it's important that the system is molded to fit the needs of the developers and testers; so if you have any suggestions/improvements/fixes, don't hesitate: Contribute.

posted at: 03:12 | link | Tags: , , | 13 comments