rhelmer’s blog

If you care about the behavior of the Firefox perf test machines, please check out my post moving Mozilla1.8 tinderboxes to Buildbot – perf impact in the mozilla.dev.performance newsgroup.

The big question is whether we can move to a model where we only build on checkin rather than continuously. This change would mean faster build turnaround times for developers, and a reduced load on build machines. It also means that the perf machines cycle less often. Currently, there’s no way to disambiguate the start time of the run versus the latest revision in the build (for CVS, revision in this sense is branch+timestamp), Tinderbox and graph servers all expect build and perf run to be the same thing.

In case you’re wondering why I’m worried about the Mozilla1.8 tree, if all goes well with this rollout we’ll want to do it this way on Firefox tree as well; the Mozilla1.8 branch is stable and already on release automation, so we think it makes sense to start there first.

2 Comments »

I’ve set up the release automation staging server for the Mozilla 1.8 branch (Firefox 2.x) to also generate nightly builds and depend builds on checkin to the branch (using buildbot’s BonsaiPoller). I outlined some of the advantages to this release automation/nightly+depend integration in my previous post.

You can see the results on the Mozilla1.8-Staging Tinderbox tree. 

The main impediment to taking this live are the performance test machines. These machines currently only cycle when a new build is available, but ideally we’d want them to keep re-testing the same build as many times as possible, to get more stable test results. Since the Tinderbox-driven depend builds currently continously cycle instead of waiting for checkin, we tend to get several test builds for the same source code as a side effect.

These machines forge their start time to match that of the build it came from which allows for easily matching up checkins and build times to performance results, but this doesn’t really make sense if we’re doing multiple test runs per build.

I’ve started a thread in the mozilla.dev.builds newsgroup with the subject “moving Mozilla1.8 tinderboxes to Buildbot” for general discussion about this idea.

1 Comment »

As I mentioned previously, I’ve been working on incrementally moving our Tinderbox client installs over to Buildbot.

The first step is to switch from driving Tinderbox from Buildbot and our release automation system, instead of being driven on each machine from the multi-tinderbox script. The release automation still calls Tinderbox client underneath, so features like CLOBBER support and all of your other favorites remain.

I have the staging automation builders publishing to the Tinderbox MozillaStage tree. Note that it’s using Mozilla1.8 builders but firing off builds when it sees trunk checkins; this is because I want to make sure the Bonsai polling is working and Mozilla1.8 doesn’t get very many checkins Also, I’m trying to stay out of the way of the ongoing trunk automation work that bhearsum is driving (AKA, letting him find and fix the trunk+release_automation bugs before I add nightly support). Expect to see trunk nightlies up there in the next few weeks.

This has several advantages right off the bat:

• same release process we use for production (currently a very small subset
• only build on checkin, should help cycle times
• same builders used for nightly and production releases (admittedly, this is how it used to be before release automation; but now we can let Buildbot handle the queuing instead of either interrupting depend/nightly builds or running multiple builds on the same machine, which is slow)

As we continue to make the nightly and final release process more alike, we can start taking advantage of things like that only final releases have but are missing on nightlies:

• updates for l10n (only en-US gets updates currently )
• update verification
• publishing the source tarball used to buld
• using the same timestamp for all platforms

On the administration side, it should let us manage builders centrally, more quickly and easily deploy new builders, and with a little more work will let us parallelize builds (multiple build machines per column, or buildslaves per builder in Buildbot parlance), which should further help cycle time (no having to wait for the current build to finish to get a build started with your fresh checkin).

Comments/questions/concerns welcome! Feel free to email me, the build group, or post in mozilla.dev.builds newsgroup if you’d like to discuss further.

3 Comments »