rhelmer’s blog

The Tinderbox waterfall page is pretty detail-heavy. I see a lot of complaints specifically that the page gets too wide, making it hard to see at a glance if anything is failing to compile or failing a test (which for Mozilla means that you can’t check in until it’s fixed).

One way to make this better is to collapse the information and provide visual cues:

Ok, well maybe not that exact picture But the point is that all Linux, Windows and Mac machines are represented by one column per build-type, not one column per machine.

Another useful thing to do would be to make use cases, and provide different front-ends instead of just the waterfall page. For example, there’s the “can I check in right now” use case, which is different than the “are all machines reporting ok” and “what do the performance numbers look like today” use cases.

Right now, we have the graph server, and to track down e.g. a perf regression window can be pretty painful (mostly because you can’t see checkins along the graph; this is probably most appropriate for the new graph server).

For the “can I check in now” use case, making the front page of Tinderbox more friendly would be a good first step.

1 Comment »

preed always makes fun of my release process diagram (as seen on whiteboards everywhere):

So I made a fancier one, showing the inside of each of these steps:

However, I still feel that preed doesn’t appreciate it, so I dedicate this diagram to morgamic.

No Comments »

We’ve been using the release automation scripts, aka Bootstrap, for the past several releases. We’ve hit some bumps but overall we’ve improved quality as we’ve been pushing changes into the scripts instead of having to document and remember to re-check a large list of “gotchas” whenever we run into problems, or need to add new steps to the release (e.g. DLL/EXE signing). Every release “step” has a set of verification tests, which we’ve been augmenting and then not having to think about next release.

Repeating the same set of steps every 6-8 weeks sounds pretty terrible; it’s short enough so that it feels like you’ve done it a million times already, and it’s just long enough that you get a little fuzzy on the details and have to constantly refer to the documentation. Even worse, you have about a dozen individual files to edit, hundreds of commands to run, and if any of it is incorrect you (generally) need to go back and start over from there. After all, you can’t build without a tag, repack without a build, or generate (all of the) updates without repacks.

The part that’s exciting, challenging and fun is when you start to break down the big, scary problem into a set of small, manageable problems. Abstract the small problems into discrete steps and automate them, so that you don’t need to worry about the individual details every time you use them. You can examine each step separately, optimize it, test it, and try to get as close to absolute consistency as possible. Do paranoid, pedantic tests for correctness that would drive a person mad.

In short, it’s basically refactoring and unit testing. When you start doing it after-the-fact there’s a high ramp-up cost, but once you’ve got the ball rolling it starts picking up serious momentum.

Continue Reading »

No Comments »