This Week in Bazaar

This is the 9th in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, unlicensed health professional. This week we are joined by Paul Hummer, who works on integrating Bazaar into Launchpad.

How to integrate bzr into your build and release process

Once you are happily using bzr on your project, the next step is some basic integration into your build process. A common desire is getting revision number to store during build process, so that you can tell what revision your program was built with. This is easy to do with 'bzr revno', which prints the current revision number. Thats not very exciting though.

There is a much more sophisticated command in bzr called version-info. For example, running:

 bzr version-info --custom \
  --template="#define VERSION_INFO \"Project 1.2.3 (r{revno})\"\n"

Will produce a C header file with formatted string containing the current revision number. Other supported variables in the templates are: date, build date, revno, revision id, branch nickname, and clean (which shows whether the tree contained uncommitted changes). This makes integrating into make or another build system very easy. The templates make it very easy to generate a version file for whatever language you are writing in.

What else could be automated other than version info? The bzr-stats plugin has a credits command. This is useful for getting a list of contributors to fill out a credits page, easter egg, etc. Also, changelogs can be generated with the gnulog plugin.

Andrew Bennetts has been working on a new server side push hook that can be used to run tests before allowing a push to complete. Wow, this could replace PQM! Well, not quite. This is more of a poor-man's PQM. It doesn't scale as well, but would work for smaller teams that don't necessarily need PQM. Blocking push while tests are running is not a good idea if you have a very long test suite, and PQM will merge and commit, making it easier to deal with multiple people trying to merge changes at the same time. If you're working in a very small group (1-3 people) with a smaller test suite, using these hooks might be just the trick, but for a larger work group you should still set up PQM.

Right now PQM is a fair amount of work to set up, but that should be changing soon. Daniel Watkins has started work on making PQM easier to set up and use, and others have been submitting cleanup patches too.

Finally, if you are using bzr on a project that builds .deb packages, check out the builddeb plugin. It would be great to have plugins for other packaging tools as well! RPM, MSI, JAR, WAR, etc.

This Week in Bazaar

This is the eighth (wow, 2 whole months of solid updates, yipee!) in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, who drinks the rain. This week we are joined by Martin Albisetti, talking about Loggerhead, and dreaming of a cold pint.

bzr-search, loggerhead, gnome, and you

Robert Collins recently published his awesome bzr-search plugin, and John Carr has been doing a lot of work on setting up a bzr mirror of Gnome. A neat search module and a bunch of source trees is just begging to be combined in some sort of web interface!

There are a few web front ends for Bazaar at the moment, such as Loggerhead, webserve, viewbzr, and bzrweb. Today we are going to be focusing on Loggerhead (you can also go to its Launchpad project page to watch the development activity). It is probably the one with the most active development at the moment. An installation of the latest stuff in action is available at the bzr mirror of Gnome. Loggerhead shows side-by-side diffs, has RSS feeds, and lets you download specific changes, just like you would expect.

You can get the latest version of it yourself by doing:
bzr branch lp:loggerhead
You'll need python-simpletal and python-paste. Then by running "serve-branches.py" in the directory where you're branches live, you should be up and running with your own web interface. Eventually serve-branches.py is to expected to become a bzr plugin which will let you easily serve your branches with a single bzr command.

We hinted at it above; recent versions have started integrating with bzr-search. So for branches that you've run "bzr index" on, it can give hints in the search dialog, and quickly find revisions that match your search terms. You can try it yourself by just typing a few letters into the search dialog.

In the coming weeks, Loggerhead will be getting a bit of a face lift with a new theme to make its externals as shiny and new as its internals.

So give it a poke, and send any feedback to either bazaar@lists.canonical.com, or https://bugs.launchpad.net/loggerhead.

This Week in Bazaar

This is the seventh in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, who is sentimental today.

MySQL Switches to Bazaar

Very big news for the Bazaar team today, as MySQL announces switching from Bitkeeper to Bazaar.

One of the things that was important in doing this conversion was doing a very high quality import of all the existing history. John did a great job working on that, and even added a new feature to Bazaar and bzr-gtk to enable this: per-file commit messages. Since per-file commit messages had been used for years in the MySQL code base, it was not acceptable to lose them, and none of the DVCS systems under consideration supported these messages. Although this feature is debated by some, it was important to preserve that history, and so support for per-file commit messages was added to Bazaar in a non-invasive way, where projects who wanted to use them could, but existing projects were not forced to adopt them. At the moment, to enter per-file commit messages you need to use the bzr-gtk GUI commit tool, but we'd love it if someone came up with a clean way to enable this in the standard CLI also.

It was also important to have a smooth transition period that did not interrupt delivering MySQL releases. This meant we needed a stable importer where the imports could be periodically refreshed without causing all of the developers around the world working on the project to re-download all their trees. At one point we were doing continuous imports of over 30 trees.

It's been a fun and challenging project providing support to MySQL during this time. Although we're really excited about this milestone, we still have plenty of work to do. Here are a few things we've learned, where we are working to make Bazaar even better.

Stacked Branches. We've talked previously about stacked branches, and for a project like MySQL this new feature will make uploading a new branch to Launchpad much faster.

Merging - Bazaar has several good merge algorithms, but we still have some ideas to make merging go even smoother, particularly for some of the complicated ancestries that MySQL has. All merge algorithms have their own set of trade offs, edge cases that they handle better or worse than other algorithms.

We also need to continue to add GUI tools, and make further enhancements to existing tools. If you are looking for a valuable way to contribute to Bazaar, try lending a hand to one of the Bazaar GUI projects.

Last week we asked about bzr screencasts, and James Westby told me about a screencast that he recorded - if anyone else is interested in getting involved in producing a series of screencasts, please do let us know.

This Week in Bazaar

This is the sixth in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, who just wants a nice story and a nap.

1.6 on the way

We decided to change the release process a bit for the bzr 1.6 release. We're introducing a bit more than normal in this relase (such as Stacked Branches), so we've decided to delay the final release a couple of weeks to ensure that everything gets an extra coat of polish. We've already had 2 beta releases, which are available in the Source.

Please give it a poke and let us know what you think.


Diff and Merge Tools

When you start working with other people on a project, you need some way of seeing what code has changed, doing code reviews, resolving conflicts, etc. The 'bzr diff' command has a '--using=foo' argument that allows you to plug in your favorite diff/merge tool if you don't want the built-in text based diff. You can also add an alias for your favorite tool. For example, Elliot uses meld all the time, so he has 'alias mdiff=diff --using=meld'. You also might want to install the difftools plugin, which adds some smarts to Bazaar about whether a particular tool understands how to diff a full tree or needs to handle the files one at a time. Here are some of the more interesting diff tools that you might want to try out:

• Meld
• Kdiff3
• vimdiff
• Wikipedia lists many more file comparision tools

One technique for easily reviewing a lot of incoming code is to keep around a pristine branch of your project that you use for conducting reviews. You can apply a patch to the tree, then run 'bzr mdiff' (or your own favorite tool), and take a look at all the changes in the patch with a lot more context than is included in the patch itself. This also gives you a spot to run the automated tests for that project, see if it compiles, etc. Once you are done with the review you can simply 'bzr revert' to get back to a clean tree and move on to the next patch to be reviewed.

Another neat trick is to use the 'merge --preview' switch. You might want to use this command to take a look at any conflicts that might have been introduced if there have been changes since the patch was generated. It shows you the patch of exactly what would be merged into the branch at that moment in time, which can sometimes have differences from what you would be reviewing by reading the patch.

Another interesting (but commercial) tool is Changes.app. It is a Mac OS X client which integrates with Finder and provides a comparison tool. It has direct support for Bazaar as well as several other version control systems.


Screencasts

Screencasts are becoming a very popular way to show people how to use your fancy tool, and we'd like to get some volunteers to help with putting together some screencasts explaining how to use various parts of bzr and related tools. If you want to help with this, email elliot at canonical dot com. The great thing about screencasts is that they use a different avenue for conveying information (audio, motion, etc) so while it won't replace a written tutorial, it is a wonderful supplement.

DVCS Comparison: On mainline merges and fast forwards

DVCS Comparison: On mainline merges and fast forwards has a discussion about whether 'fast forward' is a "better" method for merging in a distributed topology.

I can understand where he is coming from, and we respect that some users prefer other workflows. Bazaar even has direct support for 'fast forward' with 'bzr merge --pull', and with our aliasing functionality, you can set:

[ALIASES]
merge = merge --pull

In ~/.bazaar/bazaar.conf and change the default meaning of 'bzr merge'. However, I still fall of the side of the fence that fast forward should not be the default.

I can agree that if you have 2 people collaborating on the same feature that you would want fast forward. Though I would argue that is because they are effectively working on the same branch. For my personal workflow, I have a different alias set:

log = log --short -r -10..-1 --forward

What this means is that when I type 'bzr log' I see just the mainline commits of a branch, without the merge cruft. (Where I define the merge cruft as the individual revisions that make up a feature change, not the 'merge foo' node.)

Take this view of bzr.dev:

3466 Canonical.com Patch Queue Manager 2008-06-02 [merge]
    (jam) Give Aaron the benefit of bug #202928

3467 Canonical.com Patch Queue Manager 2008-06-03 [merge]
    (Martin Albisetti) Better message when a repository is locked.

3468 Canonical.com Patch Queue Manager 2008-06-03 [merge]
    (mbp) merge 1.6b1 back to trunk

3469 Canonical.com Patch Queue Manager 2008-06-04 [merge]
    (mbp) Update more users of default file modes from control_files to bzrdir

3470 Canonical.com Patch Queue Manager 2008-06-04 [merge]
    (Jelmer) Move update_revisions() implementation from BzrBranch to
      Branch.

3471 Canonical.com Patch Queue Manager 2008-06-04 [merge]
    (vila) Split a test

3472 Canonical.com Patch Queue Manager 2008-06-04 [merge]
    (jam) Fix bug #235407, if someone merges the same revision twice,
      don't record the second one.

3473 Canonical.com Patch Queue Manager 2008-06-05 [merge]
    Isolate the test HTTPServer from chdir calls (Robert Collins)

3474 Canonical.com Patch Queue Manager 2008-06-05 [merge]
    Add the 'alias' command (Tim Penhey)

3475 Canonical.com Patch Queue Manager 2008-06-05 [merge]
    (mbp) #234748 fix problems in final newline on Knit add_lines and
      get_lines

You get to see a nice short summary of everything that has been happening (in proper chronological order.) Admittedly, seeing "Patch Queue Manager" on each of those commits is less optimal (which is why we add the author names.) That is just a temporary limitation of our PQM. Bazaar already supports setting an --author, separate from the committer, we just need to teach our integration bot to use it.

The big difference, IMO, is whether you are bringing in someone else's changes to enhance your work, or whether you are collaborating on the same item. I would argue that collaborating on the same item is slightly less common. It also depends what you do with the merge commits. Just saying "merge from branch A" is certainly not helpful. But when you can say "merge Jerry's changes to Command.foo", it can indeed be helpful when tracking back through and figuring out where and when "foo" changed, without being lost in the forest for having too many trees.

This Week in Bazaar

This is the fifth in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot "fresh needle" Murphy. The two topics for this week are not related, but it's our blog and we get to write what we want.

Hosting of Bazaar branches

One of the first questions people ask when moving to Bazaar is "Where can I host my branches?" Even with distributed revision control, it is often handy to have a shared location where you publish your code, and merge code from others. Canonical has put a lot of work into making launchpad.net an excellent place to host code, but there are many other options available.

Because bazaar supports "dumb" transports like sftp, you can publish your branches anywhere that you can get write access to some disk space. For example, sourceforce.net gives projects some web space with sftp access, and you can easily push branches up over sftp. It's also easy to use bzr on any machine that you have ssh access to, you don't even need to install bazaar on the remote machine.

As bazaar is a GNU project, we've been working with the Savannah team to enable bazaar hosting on Savannah also.

Another option is serving bazaar branches over HTTP. You can do this for both read and write access, and there is a great HOWTO in the bazaar documentation. Do you know of anywhere else that is offering Bazaar hosting? Let us know in the comments!


Bazaar review and integration process

How do you ensure high quality code, when working on a fast moving codebase in a widely distributed team? Here are some things that we've been doing with the Bazaar project, and we think they are useful practices for most projects.


Automated Test Suite

One very important key towards having a stable product is proper testing. As people say "untested code is broken code". In the Bazaar project, we recommend that developers use Test Driven Development as much as possible. However, what we *require* is that all new code has tests. The reason it is important for the tests to be automated, is because it transfers knowledge about the code base between developers. I can run someone else's test, and know if I conformed to their expectations about what this piece of code should do.

This actually frees up development tremendously. Especially when you are doing large changes. With a good test suite, you can be confident that after your 2000 line refactoring, everything still works as before.


Code Review

Having other people look at your changes is a great way to catch the little things that aren't always directly testable. It also helps transfer knowledge between developers. So one developer can spend a couple weeks experimenting with different changes, and then there is at least one other person who is aware of what those are.

The basic rules for getting code merged into Bazaar is that:

1. It doesn't reduce code clarity
2. It improves on the previous code
3. It doesn't reduce test coverage
4. It must be approved by 2 developers who are familiar with the code base.

We try to apply those rules to avoid hitting the rule "The code must be perfect before it is merged", and the associated project stagnation. Code review is a very powerful tool, but you have to be cautious of "oh, and while you are there, fix this, and that, and this thing over here." Sometimes that is useful to catch things that are easy (drive-by fixes). It can also lead to long delays before you actually get to see the improvements from someone's work, and long delays are demotivating.

Item number 3 is a pragmatic way to approach how much testing is required. In general, the test coverage should improve, jsut like the code quality. But that doesn't mean you have to test all code 100 different ways.


Integration Robot (PQM)
Now that you have a good automated test suite, and proper code reviews, the next step is to make sure that you have a version of the code base which has all the tests passing. Often when developing a new feature, it is quite reasonable to "break" things for a bit, while you work out just how everything should fit together. Requiring the tests to pass at each level of development puts an undue burden on developers, preventing them from publishing their changes (to get feedback, to snapshot their progress, etc.) Very often I commit when things are still somewhat broken, as it gives me a way to 'bzr revert' back to something I wanted to try.

However, you don't want the official releases of your project to have all of these broken bits in them. The Bazaar project uses a "Patch Queue Manager". Which is simply a program that responds to requests to merge changes. When your patch has passed the review stages, you submit it to the PQM, which grabs your changes, applies them, runs the full test suite, and commits the changes to "mainline" if everything is clean.

The reasons to use a robot are:

1. Humans are very tempted to say "ah, this is a trivial fix, I'll just merge it". Without realizing there is a subtle typo or far-reaching effect. When you have a large test suite, it can often take a while to run all the tests (the bzr.dev test suite runs in 5-10 minutes, but some projects have test suites that take hours.) Having a program doing the work means a human is relieved of the tedium of checking it.
2. There is generally only a single mainline, but there may be 50 developers doing work on different branches. When they all want to merge things, it isn't feasible to require the users to run the test suite with the latest development tip. If the development pace is fast enough versus the time to run the test suite, you can get into an "infinite loop". Where you merge in the latest tip, wait for the tests to pass, and by the time you go to mark this as the new mainline tip, someone else beat you to it. And you go around again. PQM does this for you in a fairly natural way.
3. Running in a "clean" environment is a safety net for when you forget about a new dependency that you added.
4. There are similar ways to do this, such as Cruise Control for Subversion. There is one key difference, though. With Cruise Control, you find out after the fact that your mainline has broken. With PQM, we know that every commit to the mainline of Bazaar passed the test suite at that time. This helps a lot when tracking down bugs. It also helps with "dogfooding"...
   

Dogfooding

If you want people to do regular testing of the development version, it must be easy to run different versions of the project without needing a complex install. Bazaar does this by being runnable directly out of the source tree, without any need to set $PYTHONPATH or mess around with installing different versions. You can also easily change out the plugins that are loaded using the BZR_PLUGIN_PATH env variable. This means that developers can run the latest development version, and easily switch to a particular version when trying to reproduce a bug or help a user.

By having the PQM running the test suite, developers can run on the bleeding edge, and know that they won't get random breakage. It is always possible that something will break, but the chance is quite low. (In 2000 or so commits since we started using PQM, I believe bzr.dev has never been completely unusable, and has had < 5 revisions which we would not recommend people use.)

It also means that you can be fairly confident in creating a release directly from your integration branch (mainline).

This Week in Bazaar

This is the fourth in a series of posts about current topics in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, imaginary boy and part-time impostor.


Stacked branches

Some projects are very big with lots of files and lots of history. Many projects want to maintain the policy that development is done on independent branches, which are then merged back when the development is complete. However, the overhead of downloading, branching, and uploading the full history is prohibitive. There are a couple of different ways to solve this problem.

Dealing with a large branch can be split into two problems: downloading and uploading.

Bazaar has had a storage optimization called shared repositories for quite a while. This serves to dramatically reduce the amount of data downloaded for the second, third, etc branches of a project. A shared repository is a big pool of revisions which multiple branches point to. When you grab a new branch into a shared repository, bzr figures out how much of the history it already has, and only downloads the new revisions. So the first branch of a large project transfers most of the data, and grabbing additional branches is very cheap. In extreme cases, like working on a multi-gigabyte project from a 56k dial-up connection, you could even do things like distribute the initial data on a DVD to prime the shared repository, and then the user only needs to download incremental changes.

This technique can also be used for solving the uploading problem. If the upload location uses a shared repository, then uploading a new branch can just copy the new data. The problem with this, is once you start introducing multiple users, who decide that they may not want to give access to other people to push data into their repository.

Another approach to minimizing the data uploaded is called server side forking, and you can see a nice implementation of this on github.com. The user places a request with the code host to do the copy for them, and when it finishes, they have their own location already primed with the current branch.

The Bazaar project is approaching it in a different way. If some data is already public, then you can just reference the other public location when you start uploading your new branch. The first steps in this direction are being termed "Stacked Branches". Basically, instead of requiring all branches to contain the full history, you are allowed to "Stack" a branch on top of another. Because the uploader does not have write access to the lower levels of the stack, this addresses the security risks of shared repositories.

Stacking also opens up possibilites for the "download" side of the equation. For many users, they don't need a very deep copy of history to get their work done. If there is a location that can be trusted to be available when they need it, they can copy just the tip revisions. Which would allow them to do most of their work (commit, diff, etc) without consulting the remote host. And when they need more information (such as during a complex merge), the bzr client is able to fall back to the original source to get any needed information.

The goal of all this is to make it very easy to start working with a large project, while still making all the history available in a meaningful way. The bulk of this work has been completed, and it is likely that it will land in bzr 1.6 (to be released in a couple of weeks.)