Tinderbox
Posted by robert on 15 Feb 2009 at 01:50 pm | Tagged as: mozilla, tinderbox
I have been meaning to respond to a bit of Aki’s post which linked to me a while back.
I totally agree on quite a bit, although I’d argue that unless someone really steps up, takes a leadership role, and sets a clear future direction, then sticking with Tinderbox indefinitely is going to continue to give you diminishing returns. Tinderbox 1 has been in maintenance mode for a very long time, although cls, bear and reed do a great job of keeping it secure and limping along. Tinderbox 2 was maintained by bear for a while when he was at OSAF but he suggested Buildbot as a better alternative, and Tinderbox 3 looks like a great proof of concept but has been inactive for a very long time.
I feel that it’s better to contribute to an already active community that has a lot of momentum behind it, instead of trying to build support behind home-grown products like Tinderbox and Bonsai, given the amount of work it is to build and maintain an active community and the current state of these projects. There were no active competing projects when these tools were released, and they really set the bar at a time when “continuous integration” had yet to be coined. Overall they’ve been hugely successful and delivered a lot of value to Mozilla and others, but without a driving force behind new development, they are not keeping up with demand. I could give you a bunch of little examples, but I think that the fact that the “blame” column (which is a critical feature) has been empty since the switch to hg says it all.
rhelmer covered the current tinderbox/buildbot split, and is among the voices I’ve heard/read calling for a move away from the waterfall view, which I don’t completely understand. I do understand that the waterfall is far from ideal as a solitary view. But it does represent the activity of builds and build machines over a brief amount of time quite well. Even better when you have a guilty column
So, why not have both? Or multiple? Not to clutter, but to present different ways of accessing the data. Each with their own strengths.
I don’t think that the waterfall is bad, it is actually quite brilliant for certain use cases; however the waterfall is at one end of the spectrum, with something like Dolske’s isthetreegreen.com on the other side, and things like tinderboxpushlog somewhere in the middle. So in essence I agree, but I think the waterfall is actually not that useful in most cases. It’s a pretty low-level, diagnostic type of interface.
Why do people visit Tinderbox? Here is what I think:
- Should I pull the tree (“Will It Build?”)
- Can I check in (“Is the tree open?”)
- Who broke the build (and how)?
- Has there been a regression in performance or other metrics?
Out of these, only the latter two are served by the waterfall, and that’s only a starting point for this kind of investigation (which the waterfall does an OK job at).
I think that the first two are a much larger subset of users, and a huge and complex display is actively hurting them. Regression hunters need a much larger arsenal of tools, and the waterfall may not be the best place for them to start, and certainly isn’t the last place to visit (they’ll need build logs, graphs, etc.).
There’s a ton of innovation going on around build and release right now, for example I really like how Hudson approaches the problems here, and also has direct support for release processes. Like Buildbot, it doesn’t do everything Tinderbox does, and it has it’s own tradeoffs. It’s not a drop-in replacement for Tinderbox.
A drop-in replacement for Tinderbox is an interesting notion, but I think it’s worth taking a step back and figuring out if you’re really getting the value you could be. I think this says it better than I can:
The telephone destroyed the telegraph.
Here’s why people liked the telegraph: It was universal, inexpensive, asynchronous and it left a paper trail.
The telephone offered not one of these four attributes. It was far from universal, and if someone didn’t have a phone, you couldn’t call them. It was expensive, even before someone called you. It was synchronous–if you weren’t home, no call got made. And of course, there was no paper trail.
If the telephone guys had set out to make something that did what the telegraph does, but better, they probably would have failed. Instead, they solved a different problem, in such an overwhelmingly useful way that they eliminated the feature set of the competition.
The list of examples is long (YouTube vs. television, web vs. newspapers, Nike vs. sneakers). Your turn.
Looks like we agree on quite a bit =)
I think the main challenge is making that cutover… which is why I was thinking along the lines of multiple and customizable views, in which the most useful will win out.
The basis of the initial post was the idea of putting a database for queueing and build history behind buildbot, a database shared by multiple buildbot masters. With such a database, creating multiple views would be easier than trying to massage the existing tinderbox perl data structure to show details it wasn’t designed to show. So I wasn’t necessarily talking about keeping Tinderbox1 around forever, but running in parallel until it was clear that the new view(s) were as good as or better than the existing ones.
Yes we do, sorry to pick on that one part of your post; I just wanted to respond since I was mentioned
I do have ideas on the actual switchover, and also the whole “build database” idea which has been discussed quite a bit in the past.
I have been working on a few things, such as a new AUS3 web/db system, which lead me to conclude that something similar would also be suitable for pulling data out of Tinderbox/Buildbot – I made a little progress which you can see in my hg repo.
I will try to flesh this out more and respond to your post as a whole, again sorry for taking it out of context; I wanted to use it as an opportunity to drive home my points, but I did hear and agree with you and think you’re on the right track.
I think you missed a reason to visit Tinderbox – how long will it be until the tree is green or my checkin has cycled green?
It sort of fits under your point 2 – Can I check in? However, I think it is partially addressed by a waterfall display (tinderbox/buildbot) – you can normally get a reasonable idea of how long it will take (as long as you haven’t caused a complete rebuild for some reason) by looking at previous and current builds.
tinderboxpushlog and isthetreegreen definitely don’t address this at the moment, though buildbot waterfall would do out of the box due to its etas (which aren’t perfect, but are something). Tinderbox waterfall is probably the best current way of working out roughly how long.
Hrm. Looking at the datamodel in millicent, I don’t think that’s going to help us significantly. I think that plenty of views useful for presenting build results as well as debugging depend on build properties, most likely the full list we have in buildbot.
One fine day, I’d love to understand what you like about Hudson, as I haven’t figured that out yet. And right now, I can’t even get close, as the website is down (at least the wiki part with the actual info).
Well the point behind Millicent is to be a drop-in replacement for Tinderbox in the sense that it can import data (from email right now, but ideally from JSON too) but all it does is export that data in a saner JSON format, and the consumer has to deal with it (whether that’s Javascript, or a web app, etc).
It’s one approach toward moving incrementally away from Tinderbox, an alternative to improving the Tinderbox code and the JSON exporter. I think this would be a great way to do it, but there is no good separation of front-end/back-end code in Tinderbox. Millicent is just a really simple backend.
Right, like I said the waterfall is brilliant for certain cases.. I’m not saying it should go away, and if you want to start there you can.
I would say that ideally going to http://tinderbox.mozilla.org (or http://build.mozilla.org, whatever) would provide an interface that’s actually useful, instead of just a list of projects.
If you wanted to then click through and bookmark the waterfall interface and use that exclusively, I think that makes a lot of sense for certain Tinderbox users.
I use tinderbox for: is the tree green and, if not, what is the first time that it’s likely to be green in the future (i.e. what is the estimated time that all currently red or orange boxes will have completed another full cycle)?
This is actually the question that anyone who wants to check in is really asking.
Gerv
Gerv this is an excellent point, this is a crucial detail that I’ve been leaving out. I know that both Buildbot and Hudson have ETA support, although I have not looked into them in any real depth; Hudson gives last build time and last successful build time for each project.
In principle it seems like there should be enough information to say how long a successful build generally takes, how long the last build took, etc. at a higher level. Having a (reliable!) system do this for you seems better than having to slog through the waterfall and estimate yourself; I have not had to futz with Hudson at all, although I use it on a much smaller project than Mozilla.
The kind of thing I am proposing, just to be clear, would be a better front page, such as the one on http://tinderbox.mozilla.org
Way more information could be provided there than is currently, and anyone who needs more information could click through (just as they do now) to get to a more detailed page.
[...] http://roberthelmer.com/blog/?p=87 [...]