Evolving the Fedora Updates process

Luke Macken <lmacken@redhat.com>

Overview

Past: How we have been pushing updates since Fedora's inception Present: bodhi1, the push process, metrics, growing pains Future: bodhi2

FC1 (Yarrow) ➡ FC3 (Heidelberg)

Build package, edit a flat-file in CVS, assign themselves an ID, provide details like testing/stable, security, embargo, etc. Then had to email releng by hand, wait for a response, then generate the mail template and send it off.

FC4 (Stentz) ➡ FC6 (Zod)

Internal to Red Hat, used for Fedora Core only, meaning only RH developers could use it.

I wrote it without ever using it myself.

Zod EOL

That all ended with Zod.

bodhi "1.0"

First TurboGears1 app in FI, along with MM.

Best of breed at the time

Reduced effort for maintainers, testers, infra, and releng. Less blocking & context switching.

Started to encourage and acknowledge testers

BaseClient was developed by Toshio and I and made it really easy to write CLI for TG1 apps.

This also involved writing a FAS identity layer for TG1, which is still in production today.

The Push Process

Fedora Updates

Types

Growing Pains

One of the many issues with a "Best of breed" framework, is that a lot of the components don't remain the best for very long, and many become abandoned.

was a good first step, but turns out to be extremely limiting.

Fires

[X] Duplicate update IDs

[X] f9.newkey

[X] sha256 updateinfo hashing

[_] Various explosions during pushes that still occur to this day

Examples of production explosions that I have lost sleep over

Unsigned packages stale testing packages that pop back up when newer builds go to stable failures with external services

Bodhi2

Aims to solve a lot of the previously mentioned limitations and issues. Currently under development, and planning to have it deployed shortly after F21.

Now I'm going to talk about some of the new features and improvements that bodhi2 brings to the table.

Modern look-and-feel

Props to dfong. Has held up over the years. I wanted to port it wholesale, but the community felt that it needed an update.

fedora-bootstrap, thanks to threebean. need help from designers to improve it

REST-ish JSON API

Clean, standards-compliant, self-documenting, RESTful JSON API Thanks to Pyramid (part of the Pylons project) and Cornice (from Mozilla)

Standards compliant meaning automatic error codes that adhere to the HTTP spec.

Cornice provides a Sphinx plugin that automatically generates API documentation for your services So that we can spend more time writing code and less time writing docs.

HTML/JSON/RSS endpoints

From the same URL, you can view an HTML grid in your web browser, retrieve the data in JSON, or subscribe to it as an RSS feed.

JSON

$ http get http://localhost:6543/updates/
HTTP/1.1 200 OK
Content-Length: 70141
Content-Type: application/json; charset=UTF-8
Date: Sat, 02 Aug 2014 02:57:30 GMT
Server: waitress

{
    "page": 1,
    "pages": 6381,
    "rows_per_page": 20,
    "total": 127615,
    "updates": [
        {
            "alias": null,
            "bugs": [],
            "builds": [
                {
                    "nvr": "json-c-0.12-1.el6",
                    "override": null
                }
            ],
            "close_bugs": true,
            "comments": [
                {
                    "anonymous": false,
                    "bug_feedback": [],
                    "id": 881059,
                    "karma": 0,
                    "karma_critpath": 0,
                    "testcase_feedback": [],
                    "text": "This update has been submitted for testing by cicku. ",
                    "timestamp": "2014-07-28 01:40:20",
                    "update": 124875,
                    "update_id": 124875,
                    "user": {
                        "buildroot_overrides": [],
                        "name": "bodhi"
                    },
                    "user_id": 2
                }
            ],
            "critpath": false,
            "cves": [],
            "date_approved": null,
            "date_modified": null,
            ...

RSS

$ http get http://localhost:6543/updates/ Accept:application/rss
HTTP/1.1 200 OK
Content-Length: 8008
Content-Type: application/rss+xml; charset=UTF-8
Date: Sat, 02 Aug 2014 02:56:20 GMT
Server: waitress

<?xml version="1.0" ?>
<rss version="2.0">
    <channel>
        <title>updates</title>
        <link>http://localhost:6543/updates/</link>
        <description>updates</description>
        <language>en</language>
        <lastBuildDate>Mon, 28 Jul 2014 01:40:17 -0000</lastBuildDate>
        <item>
            <title>json-c-0.12-1.el6</title>
            <link>http://localhost:6543/updates/json-c-0.12-1.el6</link>
            <description>* Address security issues:
            ...

This behavior works for all "services" in bodih2, like updates, users, builds, packages, overrides, comments, etc.

Improved command-line interface

@cli.command()
@click.argument('builds')
@click.option('--username', envvar='USERNAME')
@click.option('--password', prompt=True, hide_input=True)
@click.option('--type', default='bugfix', help='Update type', required=True,
              type=click.Choice(['security', 'bugfix',
                                 'enhancement', 'newpackage']))
@click.option('--notes', help='Update description')
@click.option('--bugs', help='Comma-seperated list of bug numbers', default='')
@click.option('--close-bugs', default=True, is_flag=True, help='Automatically close bugs')
@click.option('--request', help='Requested repository',
              type=click.Choice(['testing', 'stable', 'unpush']))
@click.option('--autokarma', default=True, is_flag=True, help='Enable karma automatism')
@click.option('--stable-karma', help='Stable karma threshold')
@click.option('--unstable-karma', help='Unstable karma threshold')
@click.option('--suggest', help='Post-update user suggestion',
              type=click.Choice(['logout', 'reboot']))
@click.option('--file', help='A text file containing all the update details')
def new(username, password, **kwargs):
    client = BodhiClient()
    client.login(username, password)

    if kwargs['file'] is None:
        updates = [kwargs]
    else:
        updates = client.parse_file(os.path.abspath(kwargs['file']))

    for update in updates:
        resp = client.new(**update)
        print_resp(resp)

The new CLI provides sub-commands, similar to the koji CLI, which is nicely organized. As opposed to a giant list of flags that the current bodhi-client offers. This let's us sanely group functionality into seperate commands, each with their own help messages.

Also, we're now using the click module, which makes it trivial to implement new commands

Search typeahead.js auto-completion

Fedora Notifications Integration

Currently, bodhi sends way too many emails, and you can't configure which ones you get.

FMN let's you configure what notifications you'd like to receive, either via Email, IRC, and eventually android, based on Fedmsgs.

Bodhi doesn't have to send out emails, all it has to do is fire off fedmsgs

Datanommer Fedmsg News Feed

Fedmsg widget: http://threebean.org/blog/a-fedmsg-widget-for-your-site

On the topic of Fedmsg, we have a system called datanommer & datagrepper that essentially datamine fedmgs.

It also provides a nifty widget, which you can even embed on your own website, that shows you a custom stream of fedmsgs based on a certain criteria.

Will eventually be live.

Debugging & Profiling Improvements

The Architecture

A high level view of bodhi's architecture. If you have any specific questions about any of these components, I'd be happy to elaborate

The current masher is just another instance of bodhi with a different communication. This approach has a variety of shortcomings, mainly due to it running within apache. The new bodhi masher however runs within a Twisted reactor, and is triggered by fedmsg.

The Database Model

So now that I've talked about how the architecture has evolved, I'd like to highlight the various ways that bodhi2.0 will benefit various people in the Fedora community.

Benefits to testers

Automated Testing

Prioritized critical/security updates

As you can see, bodhi2 makes it obvious which updates are the most important and in need of testing.

Recognition

Bodhi2 also does a much better job of giving credit where credit is due. The frontpage dashboard shines the spotlight on the top testers.

Rewards

If you're interested in Fedora Badges, I highly recommend checking out the "Badges & Badge design" talk later on today given by Marie Nordin and Chris Roberts. Marie actually designed the majority of the badges you see here during her internship with the GNOME Women's Outreach Program.

Improved Feedback

fedora-easy-karma

$ fedora-easy-karma
Getting list of installed packages...
Getting list of packages in updates-testing...
================================================================================
     fedora-easy-karma-0-0.7.20100709git561718c8.fc12
================================================================================
  Update ID: FEDORA-2010-11029
    Release: Fedora 12
     Status: testing
       Type: enhancement
      Karma: 1
      Notes: Introduce the --critpath-only parameter.
  Submitter: till
  Submitted: 2010-07-09 15:30:58
   Comments: bodhi - 2010-07-09 15:30:58 (karma 0)
             This update has been submitted for testing by till.
             bodhi - 2010-07-13 07:45:42 (karma 0)
             This update has been pushed to testing
             robatino (proventesters) - 2010-07-13 08:49:36 (karma 1)
             Seems to work properly.

  https://admin.fedoraproject.org/updates/F12/FEDORA-2010-11029

 inst. RPMS: fedora-easy-karma-0-0.7.20100709git561718c8.fc12.noarch - Fedora update feedback made easy (installed 0 days ago)

Comment? -1/0/1 ->karma, other -> skip> 1
Comment> It works perfectly.

https://fedoraproject.org/wiki/Fedora_Easy_Karma

Created by Till Maas

On the topic of feedback, there are multiple tools already available in Fedora that makes it easy to provide karma. They're not a part of Bodhi2 itself, but I feel they're definitely worth mentioning, and discussing how to improve them along with bodhi2.

fedora-gooey-karma

https://fedoraproject.org/wiki/Fedora_Gooey_Karma

Created by Branislav Blaskovic & Tomas Meszaros

Benefits to Package Maintainers

Easier update submission

  • Auto-completes package name
  • List of candidate builds
  • Populates list of potential bugs
  • Eventually could list deps that need to be included as well

Customizable karma requirements

Benefits to Fedora Users

Benefits for Fedora Infrastructure Developers

Action Items for the Initial Release

The Future

Fedora.next, ability to push cloud images. As long as it can be tagged in koji, bodhi should be able to handle it. Instead of 'mashing' a repo, for the AMI example it could simply send out a fedmsg that triggers oddshocks' AMI uploader, etc.

Getting Involved

Thank you!

https://github.com/fedora-infra/bodhi

#fedora-apps on Freenode

SpaceForward
Left, Down, Page DownNext slide
Right, Up, Page UpPrevious slide
POpen presenter console
HToggle this help