rhelmer’s blog

People are often very confused by the state of where Mozilla is with regard to Tinderbox versus Buildbot. They are both continuous integration systems, and you’d think that just jumping wholesale would be easier than the unholy marriage I’ve described in the past.

The big distinctions are these:

• server vs. client – Buildbot clients and server are tightly coupled, and communicate through an active TCP connection (managed by Twisted). Tinderbox clients simply send email to the server, one for build start and one for build stop (build stop has the status specified, which changes color on Tinderbox server). The logfile for the build may be attached to the “end” email.
•  Tinderbox server vs. Buildbot server – tinderbox.mozilla.org puts up with a lot of load. Buildbot server can probably not handle this. Also, Tinderbox server has a bunch of features that Mozilla developers depend on, like setting status, etc.

Personally I feel that Tinderbox is the wrong way to visualize what developers actually need, but I’ll save that for a later and more productive post For now, suffice to say that Tinderbox server does a lot more and can handle way more load than Buildbot server.

However, Buildbot server does have some very nice qualities, like being able to see the log in real-time, and being able to stop and force builds. So, an interim solution is to have Buildbot server send email to Tinderbox server on behalf of it’s clients, so you get Buildbot as an administrative, developer-only interface, and Tinderbox server as the general, public interface.

The 1.8 and 1.9 nightly builders are already exposed to nightly users; there are a couple kinks to work out, so I won’t link to it right now (I’ll let the people that are actually maintaining it do that ), but the glorious future is that developers can stop and kick builds as well as see real-time logs.

So, that’s all well and good, and I think fairly well understood. Now here’s the hairy part – the 1.8 and 1.9 nightly Buildbot clients are turning around and calling Tinderbox! WTF! (note that the unittest and moz2 buildbots do not do this, only the 1.8/1.9 nightly boxes). This is because Tinderbox client contains code to do a bunch of things:

• mozilla-specific build process
• performance testing
• create updates
• publish updates (nightly AUS only)
• rebooting windows 9x between builds (not joking)
• support for a bajillion products and platforms (mostly through huge “if” blocks)
• support for hybrid depend/clobber builders
• support for uploading to various locations on FTP
• much, much more

Some of these features are very useful and not available elsewhere, and some are obviously not useful anymore. The error and log handling leaves a lot to be desired; it’s not something trivially fixable, unfortunately (lots of people have tried, resulting in not one but two attempted rewrites).

Getting all of the useful bits of this into Buildbot has been a real challenge, but Ben Hearsum has all of the important bits worked out for moz2. I’m hoping to spend some time packaging that up as a BuildFactory, to make it easy to reuse this code for other branches and products (mostly because I’d really like to see bug 421586 get fixed), strictly as a community member of course

You can read more about Buildbot process-specific factories (that’s a nice example of what a GNU Autoconf style project could use, which comes with Buildbot) but suffice to say it’s a way of encapsulating the basic build process so you don’t need to copy and paste “cvs co client.mk”, “make -f client.mk MOZ_CO_PROJECT=blah” for each builder in your Buildbot master.cfg

This brings up the other big missing piece, which is that Buildbot’s awesome Source class can’t be used because it doesn’t understand that it can’t just update the whole “mozilla” CVS module, but needs to use the client.mk instead. This means that built-in clobber support and the built-in “tryserver” support can’t be used (the current Mozilla implementations have a lot of custom code).

Bug 414031 suggests a possible way to implement support for it. Although it’s kind of a pain to implement, using a driver script like this is fairly common in Java projects, so I think some kind of generic support might be feasible.

If you’re not sure what I’m talking about here and why Source can’t be used out of the box, the client.mk only does a partial checkout of the “mozilla” CVS module depending on which MOZ_CO_PROJECT is specified. Also, it can and does check out different versions of subdirectories, such as NSPR and NSS.

In other words, this is not your typical “checkout module && ./configure && make” project, although it is deceptively close in some ways It’d probably be better to have basic support for this flow, just based on principle of least surprise. I think that it also has material effect on tool support and new developers, too.

No Comments »

Since I’m going to be doing GIS programming and working with scientific mumbo jumbo again, I’ve recently started brushing up my rusty (and in some cases non-existent) math skills. I recently stumbled somewhat serendipitously on the Steve Yegge blog rant Math for Programmers.

Warning – His posts tend to be very long, I rather enjoy them but I see tl;dr in some of your futures

Anyway, a nice reminder for me is this little section:

The Right Way To Learn Math

The right way to learn math is breadth-first, not depth-first. You need to survey the space, learn the names of things, figure out what’s what.

This is absolutely true, for me anyway. I’ve started doing math every day in addition to music (time spent on both are of course are dwarfed by baby-care duties, but I can carve out a few solid hours per day here and there ).

I just spent the past week going through:

I really enjoyed this (and other algorithm books, but especially this one) last time I read it (several years ago). However, I didn’t really take the time to do most of the exercises, and just skipped the math that went over my head, or that I didn’t recall readily.

This is fine for a first pass, but the difference this time is that I’m taking notes on bits I don’t understand and following those up, and committing to coming back and working through the exercises to make sure that I grok it. JS might be a fun language to use for this

I’m finding algorithm analysis way more interesting this time around, for some reason.

Next on my list:

I tried reading this when getting hung up in parts of “The Art of Computer Programming”. “Introduction to Algorithms” also refers to this book a lot. It’s pretty dense. I’m thinking of jumping back to TAoCP once I do the first pass, and returning to specific areas as-needed, as I find progress in TAoCP very rewarding.

Focusing on the breadth-first strategy is making a lot of sense to me; it’s way easier to find the solution to a problem when you know how to recognize the type of problem, as opposed to the rote method of memorizing algorithms that I’ve found in many (but not all) classrooms.

This is highly motivating stuff:

And I’ll keep getting better at this. I have lots of years left, and lots of books, and articles. Sometimes I’ll spend a whole weekend reading a math book, and sometimes I’ll go for weeks without thinking about it even once. But like any hobby, if you simply trust that it will be interesting, and that it’ll get easier with time, you can apply it as often or as little as you like and still get value out of it.

That’s exactly how I feel about music and computers (programming, server stuff, etc) in general, I don’t know why I’ve always had such a block against math, it has always felt like more work than fun (well, except Geometry. Visuals ftw!).

I think that doing simple games can be my “in” to getting along better with Physics. I can’t remember ever _wanting_ to remember trig before I threw together that little Breakout! demo Now I am kicking Past Rob for not paying more attention. I guess the most I can hope to do is spare Future Rob that same trauma.

No Comments »

I put this set of slides together to explain what state the release automation project is in. It probably makes more sense when I am sitting there to explain what each point means, but I figured I’d put it out there anyway

The current setup mimics ye olde manual release process, forged by Chase. Over the past few years we’ve worked on wrapping that process in scripts with this perl framework (aka “bootstrap”), which auto-generates configs for underlying systems like tinderbox and patcher, checks logs for errors, etc.

A lot of the current bugs come from underlying systems, especially the tinderbox client. Reducing some of the complexity here would both make the system more understandable and most likely faster as well. It’s pretty tough to make changes when you’re doing this level of wrapping, too.

Now that everything is driven by Buildbot, it probably makes the most sense to call the build system directly, instead of buildbot->bootstrap->tinderbox->build_system that we have today. There are bugs on all of this already, hopefully the slides and this post will make it clearer how they tie together.

No Comments »

My last day at Mozilla Corporation will be April 14th. It’s been an incredibly awesome experience, and I’d like to thank everyone for the support, guidance and encouragement I’ve received.

I’m going to be taking some time off before starting work at CustomWeather, Inc. I won’t be working directly on any Mozilla projects, but I do hope to help out from time to time. I will be available to help with Firefox 3 for sure.

I’ve changed my bugmail address to my personal email address – robert@roberthelmer.com – feel free to contact me.

5 Comments »