Showing posts with label win32. Show all posts
Showing posts with label win32. Show all posts

Thursday, August 14, 2008

This Week in Bazaar

Ah, to take a break from reporting to the world, but now we are back. This used to be a completely weekly series of posts about the on-going events in the world of Bazaar (and may be yet again). Written by co-authors John Arbash Meinel, one of the primary developers on Bazaar, and Paul Hummer, who works on integrating Bazaar into Launchpad.


Bazaar 1.6rc3 Released

With Martin Pool going on vacation for the next two weeks, John has stepped up to marshall 1.6 out the door. And he started with not 1 but 2 release candidates in 2 days. We're trying hard to get back into a time-based release schedule. The problem with sneaking in a feature-based release, is that they always end up slipping, as everyone tries to get "one-more-thing" in to the delayed release. However, with RC3, we've actually gotten the list of things that must be in 1.6 down to 0, so there is a very good chance it will become 1.6-final next week.

Since it has been a delayed release, there are lots of goodies inside to partake of. Stacked Branches, improved Weave merge, significantly faster 'bzr log --short', improvements to the Windows installation, better server side hooks, and the list goes on. Most of this we have mentioned in previous "This Weeks", the big difference is that it is available in a release, rather than just in the bzr.dev trunk.

The Windows install is one of the major changes, in that it will now (by default) bundle TortoiseBzr as part of the standalone install. TortoiseBzr still needs work before it is as much of a joy to work with as the rest of the system, but this release is mostly about testing our ability to bundle them together.


Looking forward to Bazaar 1.7

As 1.6 nears it's official release, the development community has started planning the 1.7 development process. As it stands now, bzr 1.7 has a planned release date of September 8th. This means there are two whole weeks two get various bugfixes and contributions to bazaar in before getting down to release time (mentoring available).

Among the proposed potential features, there are a few that really stand out. Mark Hammond has been polishing Bazaar on Windows, and there is much desire for someone to help getting the bazaar test suite to run cleanly in Mac OS X. These features will greatly add to the existing portability strengths of Bazaar. While the majority of changes needed are actually in the test suite, and not the core functionality, the community could really use someone who could step up, and learn how to do unit testing in Python. Bazaar 1.7 will also see some increased merge flexibilities, especially with criss cross merges.

Improvements to the indexing layer are likely to land in 1.7, though as always, not on the default format. (We want at least 1 release supporting a format before we suggest it as the default, to give people time for compatibility.) The new b+tree layout for indexes makes them smaller (by approx 2:1) and makes them faster to search (eg, bzr log file being 3x faster).

We also have a chance to land Group Compress, which has shown to compress repositories by as much as 3:1 over their current size. This change needs a bit more tweaking, though. There are generally tradeoffs between how much time you spend compressing, and how small the result is. And we want to make sure that we make the right tradeoffs. It is currently being evaluated as a test plugin.


Bazaar Bug Day

As Bazaar development speeds up, so do the incoming bugs. There are currently 1062 open bugs in Launchpad, and 287 of them have a "New" status, meaning they have not yet been triaged and categorized. At a past Bazaar sprint, a "bug day" was talked about, and it has been brought up again on the mailing list. Often, we fix many bugs and just haven't gotten around to marking them fixed. This is a great opportunity for members of the community who use Bazaar but don't directly develop it t o contribute back to the Bazaar community. You can help out by verifying bugs have been fixed, that they no longer exist, or that they still exist and provide more information on them. Come give your karma a boost, and help us squish some bugs!

Monday, July 28, 2008

Last Week in Bazaar

Well, I'm late this week, so I'm officially marking this post as Last Week in Bazaar. In my defense, I got busy last Thursday, and then my cohort (Paul Hummer) flew off to New Zealand for a work-related sprint. So today, I (John Arbash Meinel, a developer on Bazaar) get to exercise full control over the content.


Keyword Expansion

People often request the ability to expand keywords, like they are used to in SVN and CVS. We've sort of postponed the implementation, because probably 90% of the time, it isn't really the right solution to the problem users are having. Also, they are kind of a mess in CVS anyway. Where I used to work we tried to use $Id$ style expansion, only to find out that they conflict on every attempt at merging, and we started working hard to strip them out of our files. In a distributed VCS, you usually merge at least an order-of-magnitude more often, which also tends to reveal this problem.

SVN at least works around the problem, in that when you commit, it actually strips the texts of their expanded keywords, so that the repository never stores the expanded form. And merges are also done on the non-expanded form. Which fixes that little problem. Though it introduces a couple others. Specifically, what you have on disk is not what sits in the repository, nor is it exactly what you will get out of a fresh checkout. The biggest reason is that if you commit revno 1354, it will update the tags of files that are touched. But if you checkout revno 1354 it will update the tags of *all* files. (I'm not positive on this, but I know there was a bug which was causing problems for people trying to do conversions. Because they couldn't quite find the right invocation to have 'update -r 1354' (from 1353) give the exact same tree as 'checkout -r 1354').

The other reason keyword expansion is not usually what you want, is because it expands only for the given file. If you make a commit to 5 other files, the *tree* is at revno 1359, but the file with your:

my_version = "$Version$";

Tag is still pointing at 1354. (Again, if 'svn update' would force all the tags to get re-expanded it might work correctly, though you run into performance problems expanding every keyword in every file on every update.) Bazaar has supported the bzr version-info command for a while, which lets you generate a version file (possibly from a template) which can store all the real details. Including the last-modified version for every file, whether any files in the working tree have been modified since commit, etc.

The only case that I've really heard a good reason for keyword expansion is for a Website. Where each individual file is spread out into the world. So having a little "last modified" at the bottom can be pretty convenient. You also don't tend to have a "build" process which lets you generate the version information at that time.

However, as Bazaar is meant to be a flexible system, Ian Clatworthy has done a wonderful job of adding the ability to support content munging via plugins. And has continued on to write a plugin specifically for expanding keywords.

http://bazaar-vcs.org/KeywordExpansion

So for all those people who feel they really need keyword expansion, look it up.I would imagine that once people get a good feel for it, and it matures a bit, it has a good chance to be brought into core code. Or at least make it into the release tarball as an "official" plugin.


Open Source, Python, and Counting My Blessings

Now onto something a bit more personal. This last week I had cause to re-visit an old library I had written, and try to get it up and running again. (Specifically, the project was pydcmtk, python wrappers for the Dicom Toolkit.)
It took me several hours times several days just to get it to build and run the test suite again. All without changing any of the code. It was simply a build-step problem.
Which revealed a couple wonderful things about my current work:

  1. I get to work in Python. Which is a nice language, flexible, and *doesn't* need a build step. Not having to deal with C/C++ and all the complexities of getting dependencies built, with the right version of the compiler, and the right flags to the compiler.
  2. Microsoft has a much harder time on their hands than Open Source does, at least when it comes to compatibility. Specifically, each version of their compiler comes with a different runtime. And code compiled for Visual Studio 7.1 doesn't like to work with the 8.0 objects nor the 9.0 objects. And they all have different msvcrtX.X.dll files. However, because the official method for getting your program to users is in binary (object) form, they have to provide ways to support your binary files for a long time. So in VS 8.0 they introduced a new step, which is to post-process your linked binaries with a manifest, declaring what runtimes they use. Further complicating this is that if you try to run a 8.0 compiled dll, it just gives an opaque "This process has tried to access the runtime incorrectly."
    Not realizing this, I spent a long time comparing the exact compiler flags with other examples to fix it. (The boost build tool, bjam, knows how to do it, but there was a line "if exists foo.manifest: do stuff", which I originally read as "if not exists foo.manifest: create the manifest.")
  3. Open source has generally handled the binary compatibility issue by punting and requesting software compatibility. And then you have a whole bunch of groups that spend their time recompiling everything for you (distributions like Ubuntu or Red Hat). And then they give you all the dependencies with a few simple commands (apt-get install zlib-dev dcmtk-dev boost-dev). On Windows, if you want to switch to developer mode, you generally have to grab the source code for all of those dependencies, and recompile them for your exact configuration.
    Software-level compatibility is *much* easier to handle. Not the least of which because if something becomes incompatible you can fix it. (I remember a Microsoft memory issue, where they had to switch in bug-for-bug compatibility because fixing it broke SimCity, how much better if they could have just patched SimCity.)
    Binary compatibility (for C/C++) means that you can't even add members to structs, because then the size changes and malloc starts failing (plus members are referenced by offset, so adding something in the *middle* is a big no-no).
    Source-level turns this way down into not removing things people are using. And, if something does change, with source-level you can even write a patch to fix the code. This does make it quite a bit harder for people who want to release binary-only packages, that they then don't have to modify for years. (Though when updating is a simple process, people are willing to do it more.)

1.6rc1 soon to come

We are working on putting the final polish to stacked branches. We are trying to release something that people can feel comfortable using right away, and there are a few tricks to get there. (For example, bzr has a general policy of always preserving the source format when you do 'bzr branch'. It helps maintain compatibility within a project that hasn't chosen to upgrade to a newer format yet. However if you do 'bzr branch --stacked' that indicates you want to use the new feature, so we have to work out logic to create an upgraded target at the right time. This also turns out to conflict a bit with bzr-svn, which had its own logic to trick 'bzr branch' into not copying the source format.)

You can already play with the Stacked Branches feature in the beta releases, but they'll appear much more polished in the final rc.

Thursday, May 8, 2008

This Week In Bazaar First Edition

This is the first in a mostly-every-week series of posts about whats been happening in the development world of the Bazaar distributed version control system. The series is co-authored by John Arbash Meinel, one of the primary developers on Bazaar, and Elliot Murphy, Launchpad developer and wanted criminal.

We get to talk about anything we want. This week:
  • What's been happening for a better GUI on Windows
  • What's new in the 1.4 release
  • Importing from other VCS's with bzr fast-import
... details ...

GUI on Windows

We found this guy named Mark Hammond who claims to know how to make python stuff work well on windows. There is an existing GUI tool for Bazaar on Windows called TortoiseBZR now, modeled after TortoiseSVN. If you haven't used a Tortoise before, they are extensions that integrate into Windows Explorer; allowing you to see and control the versioning of your files without needing to change to a separate tool.

Mark has taken a look and proposed a series of enhancements to make the tool work even better. Bazaar already works very well from the Windows command prompt, but we want to provide excellent GUI tools as well. Take a look at the TortoiseBZR web page for screenshots of it in action.

What's new in the 1.4 release

The Bazaar team releases a new version of Bazaar just about every month, with both bugfixes and new features. The bzr-1.4 release came out last Thursday, May 1st.

The major changes for 1.4 include improvements in performance of 'log' and 'status', and a new Branch hook called post-change-branch-tip, which will trigger any time a Branch is modified (push, commit, etc). This should enable server generated emails whenever somebody publishes their changes. Write something cool with it and tell us what you did!

The full list of changes for 1.4 can be found at: https://launchpad.net/bzr/1.4/1.4
The list of all changes is at http://doc.bazaar-vcs.org/bzr.dev/en/release-notes/NEWS.html

bzr fast-import

Bazaar fast-import is a plugin for bazaar that allows you to import from many different version control systems. The fast-import stuff is intended to support any system that can use the fast-export format. This format was originated by git developers, and quickly adopted elsewhere. So if a source format can generate a "fast-import" stream, you should be able
to import it into Bazaar.

  • CVS
    To convert from cvs, you currently use the cvs2svn converter. Which has a flag to generate a "fast-import" stream.
  • Mercurial
    There is a script called hg-fast-export.py bundled with the plugin (in the exporters/ directory).
  • SVN
    The svn-fast-export script is also bundled with the bzr-fastimport plugin.
  • git
    Bundled with the standard git distribution is the git-fast-export command.
  • Your own exotic system here.
Give fast-import a try. It's mostly designed for 1-time conversions, rather than mirroring, but there are already some rudimentary mirroring capabilities.


That's all for the first installment of "This Week in Bazaar".

(edited for formatting)

Wednesday, March 14, 2007

Reading and Writing to Files ('r+', 'w+' mode) on Windows

It turns out that Windows has a small oddity when reading and writing to the a file. It is reported in the 'fopen' documentation at MSDN:
http://msdn2.microsoft.com/en-us/library/yeby3zcb(vs.71).aspx

The specific quote is:
When the "r+", "w+", or "a+" access type is specified, both reading and writing are allowed (the file is said to be open for "update"). However, when you switch between reading and writing, there must be an intervening fflush, fsetpos, fseek, or rewind operation. The current position can be specified for the fsetpos or fseek operation, if desired.
As an example, here is what you might do in python:
>>> f = open('test', 'wb+')
>>> f.write('initial text\n')
>>> f.close()
>>> f = open('test', 'rb+')
>>> f.read()
'initial text\n'
>>> f.write('this should go at the end\n')

On most platforms, that succeeds. But on Windows, if you don't do
>>> f.seek(0, 2) # Seek to the end of the file

before you call f.write(), you will get an IOError, with e.errno = 0. (Yeah, having an error of SUCCESS is a little hard to figure out).

Anyway, it took a while for me to figure out, so I figured I'd let other people know.