perf impact on nightly release automation move
Posted by rhelmer on 28 Dec 2007 at 08:26 pm | Tagged as: automation, buildbot, mozilla, tinderbox
If you care about the behavior of the Firefox perf test machines, please check out my post moving Mozilla1.8 tinderboxes to Buildbot – perf impact in the mozilla.dev.performance newsgroup.
The big question is whether we can move to a model where we only build on checkin rather than continuously. This change would mean faster build turnaround times for developers, and a reduced load on build machines. It also means that the perf machines cycle less often. Currently, there’s no way to disambiguate the start time of the run versus the latest revision in the build (for CVS, revision in this sense is branch+timestamp), Tinderbox and graph servers all expect build and perf run to be the same thing.
In case you’re wondering why I’m worried about the Mozilla1.8 tree, if all goes well with this rollout we’ll want to do it this way on Firefox tree as well; the Mozilla1.8 branch is stable and already on release automation, so we think it makes sense to start there first.
How about having the perf boxes run perf tests repeatedly with the builds they have, but interrupting any “repeated” tests when a new build becomes available? Then you’ll get fast first-cycle turnarounds for checkins while also getting enough data to notice the subtle perf regressions.
@Jesse:
Sure we can do that, but how do we report the results? We’d need to stop perf testing before the next checkin happened, or it wouldn’t be possible to match up checkin times and test run times.
The method the perf machines currently use is to lie about start time, to be the same as build start time. This only works because we do continous builds, and is of course not very optimal (we could be getting more test cycles in for each build).
If we teach Tinderbox and the graph server that version control revision and start time of the test are different things, I think this would work.