Redmine has many of the features I like from Trac, such as the Roadmap view, version control integration, an Activity view, and built in Wiki.  It also has a bunch of plugins, one of which looks like it’ll generate burndown charts, which I’m planning to use.

It looked like Redmine was fairly well supported on Windows (box I have available) so I decided to try it out.  Redmine is based on Ruby on Rails, so the first step to installation is to get Ruby and Rails installed.

I started off trying to install Ruby and Rails under cygwin, but ran into problems with building mysql from source (no cygwin port for this!).

So instead I went down the route of installing everything outside of cygwin.  Here’s a summary of the steps:

• Install the Windows Ruby binary, along with Gem package system.
• Install the Rails 2.1.2 gem, which Redmine 0.8.X depends on
• Download and unpack the Redmine distribution
• Setup the database (running on MySQL in my case) and edit database.yml per instructions.
• I had to install older MySQL client library due to some Rails/MySQL library incompatibility (described here)
• Create the database schema for Redmine via:

  rake redmine:load_default_data RAILS_ENV="production"

• Populate a default configuration via:

  rake redmine:load_default_data RAILS_ENV="production"

• You can then run Redmine via:

  ruby script/server webrick -e production

Programmable Web is a great resource for those building mashups – it has a directory of APIs and Mashups that make use of those APIs.  The blog post “Top Twenty Open APIs and Mashup Resources for Web Developers” lists some of the most popular APIs.

Google Transit Feed Specification

• Provides information on stops, routes, calendar, fares, frequencies, usage policies
• Unfortunately, only available for a limited set of metro areas

Google Maps AJAX API:

• AJAX API
• Display map with road, satellite, hybrid, and terrain, by latitude & longitude
• Can respond to user events within the map – e.g. clicking, zooming in/out, panning, etc
• Can manipulate map via JavaScript – adding markers, changing location, zoom, etc
• Can create various types of overlays on the map
• Includes google street view

Google AJAX Feed API

• Include any RSS or ATOM feed on your site, with various presentation options

OpenSocial

• Common API for multiple social networking sites, including hi5, LinkedIn, MySpace, Netlog, Ning, orkut, and Yahoo.  Unfortunately Facebook is not one of them.
• JavaScript APIs for client side and REST/RPC APIs for server side
• Can be used to create apps that are embedded in social networking sites, as well as use user social information (user profile, friend lists, events) from social networks in your site.
• OAuth is used to allow users to authorize use of their social network information

YouTube APIs

• Integrate YouTube functionality into your site: video searches, upload videos, etc
• Integrate video player into your site
• Allow users to see/manage their favorites on your site
• RSS/ATOM data feeds from YouTube

YELP APIs

• REST APIs
• Retrieve business info and reviews for a geographic region and business category.  This includes the business location, and their categories.
• Retrieve neighborhood name and info by location
• Retrieve reviews for a particular business
• Retrieve business info and pictures
• It looks like this isn’t a complete directory – only those businesses and features that have been rated are listed

Zillow APIs

• “Neighborhood and city affordability statistics: Zillow Home Value Index, Zestimate distribution, median single family home and condo values, average tax rates, and percentage of flips.”
• Demographic data at the city and neighborhood level – local market data, affordability, household income, average age, commute time, etc.
• Home Valuation: “Search results list, Zestimate^® home valuations, home valuation charts, comparable houses, and market trend charts.”
• Property Details: “Property-level data, including historical sales price and year, taxes, beds/baths, etc.”
• Lists of counties, cities, ZIP codes, and neighborhoods, as well as latitude and longitude data for these areas so you can put them on a map.
• “boundaries for nearly 7,000 neighborhoods and 150 cities”  available via Creative Commons license.
• License restricts site from charging money for Zillow data

Yahoo! Local Web Services

• REST APIs
• Provides access to public collections created with Yahoo! Local Collections
• Can perform local searches based on location, radius, route, categories: returns results with location info, ratings, categories, etc
• Yahoo Local search web site: http://local.yahoo.com/ – You can see some of the information that’s available

Walkscore API:

• Get walk score by location

I’ve been using Mercurial to work on a couple projects I’m involved with that have Subversion repositories. GIT has a good bridge to Subversion built in. Unfortunately, Mercurial is a little bit behind on this front, but it looks like things are getting better quickly.

I initially tried out Tailor, a general purpose version control bridge tool, which supports Mercurial and Subversion among a bunch of others. I had difficulty getting Tailor to work with the Subversion repository I was working on, and it didn’t seem like Tailor was being used much for Mercurial-Subversion.

I’ve been using the hgsvn package lately and it’s worked out pretty well. The only caveat is that hgsvn doesn’t directly support uploading changes from Mercurial back to Subversion. hgsvn does a good job of downloading changesets from Subversion to Mercurial though, and there are a couple solutions to handle the other direction. I’ve been pushing changes back by using the excellent Mercurial MQ extension.

A new Mercurial-Subversion bridge, hgsubversion, looks like it supports both downstream and upstream changes from Mercurial to/from Subversion, and looks very promising. I’m just waiting for it to support importing starting at a particular Subversion revision before I can start using it.

Until I can use hgsubversion, here is how I’ve been using hgsvn. I installed hgsvn as a Python egg package via easy_install:

sudo easy_install hgsvn

The hgimportsvn command is used to import change sets from an existing Subversion repository into a local Mercurial repository it creates. You can optionally have it start at a particular Subversion revision if you don’t need the full history imported:

hgimportsvn [-r svn rev] <svn URL> <local directory name>

This creates a combined Subversion working directory and Mercurial repository in the local directory you give it. Once this is done you can update the Mercurial repository with the latest Subversion changesets by executing the following in the local directory:

hgpullsvn

You run hgpullsvn anytime you want to refresh the Mercurial repository with the latest from Subversion. You can then do anything you would do with a Mercurial repository, except committing changes of your own. Cloning, MQ, revision histories, diffs, etc all work. I tried committing a change to the Mercurial repository, then committing the same changes to Subversion, then doing an hgpullsvn to update both from the Subversion repository. This worked at first but the Mercurial repository got out of sync after a while, so I wouldn’t recommend it.

What I’ve been doing is managing the upstream changes via Mercurial Queues (MQ). This also helps me manage patches I want to apply on top of the latest code.

The first time using Mercurial Queues in a repository, you have to initialize the queue repository:

hg qinit -c

The -c option makes it so the queues repository is version controlled. You can then see what older versions of a patch were in case you need to roll back. I’m basically using patches as changesets, so it’s useful for me to be able to version control and have a history for them.

Once you have the queue repository, you can create a patch before making any changes to files:

hg qnew -g <patch name>

When you create a new patch, it automatically becomes an active patch at the top of the queue. You can then edit files as you normally would. ‘hg add’ and ‘hg remove’ commands mark files to be added or removed in the patch.

Once you’re ready to commit any changed/added/removed files to the patch, do a commit:

hg qcommit

Once committed, you can look in your .hg/patches/ directory and see the patch file.

Whenever I want to update from SVN, I pop everything off the patch queue so the Mercurial repository & Subversion working directory are pristine, and do the update:

hg qpop -a

hgpullsvn

I can then push the patches I’m working on back on top:

hg qpush -a

Whenever I’m ready to push changes into Subversion, I export the patch, then apply it to a separate Subversion working directory for commit:

hg export -g <patch name> > ../my.patch

in separate Subversion working directory,

patch -p1 < ../my.patch

Change management is one of those software software development practices that I rarely see implemented in IT projects, although it helps to organize development and cuts down on the more tedious project management activities (gathering status, updating detailed project plans). Change management is similar to defect tracking, which is frequently used, but is broader is scope. Defect tracking is really a subset of change management.

In a change management process, every proposed and actual change to an artifact is recorded and tracked in a change management tool. I say artifact rather than code because it can be used to track changes to requirements, test scripts, documentation, design; not just code. Every change to an artifact and commit to VCS has an associated record in the change management tool associated with it, describing the nature of the change, who requested it, what release it was planned for, etc.

Open source projects seem to more frequently use change management, which isn’t surprising considering they are made up of distributed teams who need a good way to keep track of what changes are planned for, and tracking the progress of those changes.

There’s a bunch of open source change management tools out there. Roundup, JTrac, XPlanner, Scarab, Mantisbt, ITracker, Trac, and Bugzilla are the tools I took a look at.

I have several criteria for a change management tool, including:

• Ability to link one change request to another
• This can be used to set up dependencies between changes or defects (this bug has to be fixed before that one can) or to break up large changes into smaller ones to be worked on by different people (e.g. have a change request to track the development of a feature, linked to smaller change requests to track the changes to implement the feature).
• RSS feeds for reports/searchs
• Support for representing enhancements and defects
• Support for components, with the ability to assign change requests to components, and to assign owners for each component
• Easy to administer – Ability to administer products, components, and users through a web UI
• Ability to authenticate against LDAP
• Active development community
• Easy to use, web-based on-demand reporting and charting
• Can run on Windows (it’s the box I have)
• Support by Mylyn

I narrowed the choice down to Bugzilla and Trac, and settled on Bugzilla after installing both. I’ve used and installed Bugzilla before, so I was already pretty comfortable with it. The Bugzilla Windows installation has come a long way since the last time I tried it (~2001). Back then I gave up and ended up installing Linux on a box just to be able to run Bugzilla. It still requires quite a few steps, but it worked without too much trouble.

I installed stable version 3.0.4 following the instructions at http://www.bugzilla.org/docs/win32install.html. At a high level, the steps were:

• Download and unpack tarball
• Install MySQL 5.0.51a
• Create Bugzilla database and database user
• Installe ActiveState Perl 5.8.8
• Installed a bunch(!) of Perl packages via bundled PPM tool
• Install & Configure Apache 2.2
• Had to upgrade Email and Mime related Perl packages (see this blog posting).
• Download and install Graphviz, and tell Bugzilla where the ‘dot’ executable can be found (Bugzilla parameters->Dependency Graphs)
• Execute perl collectstats.pl (not sure if this is doing anything)
• I had to execute the following to get graphical reports to work: ppm install Template-GD

What got me to try it out was the Groovy/Grails support. I’ve been doing some work with Grails and had heard they had a good plugin. After working with it a while, the Groovy support is the best out of the three major Java IDEs (Eclipse, NetBeans, IDEA). The Maven 2 support is great too.. It’s by far the best support for Maven 2 in any IDE. There are some features I would still like them to add (such as a Maven repository index/search feature), but the important things work well (handling Maven dependencies).

In general, I find myself having to fiddle with the UI less than I do with Eclipse (configuring and switching perspectives, resizing panes, etc).

The other plugin I was really interested in was the Mercurial IDEA plugin. This plugin is still early in the development cycle. Unfortunately it gave me an error when starting up – it was compiled for JDK 1.6 and I’m stuck with 1.5 on my Mac. I imported the MercurialIdea plugin source into Idea, reset it to use the Idea platform runtime, and rebuilt the plugin without success. No biggie, the Mercurial command line is easy enough to use.

After looking into both, I ended up going with Mercurial over GIT. Both Mercurial and GIT seem to fit my needs; decentralized, fast, and flexible, but GIT didn’t seem as well supported on non-Linux platforms and the Java IDE plugins are further behind. Mercurial has binaries available for Mac, Windows, and of course Linux. On the Mac and Windows side, there are a couple different options for how to install Mercurial; using a ports packaging system such as Cygwin on Windows or MacPorts on Mac, or installing natively on the OS. The Eclipse IDE plugin is still early in development, but it sounds like the NetBeans plugin is pretty full featured.

When working with Mercurial, everything is a copy of the repository, including branches and the local repository that developers work out of. Each repository is as fully capable as any other and includes all of the change history. It sounds inefficient to have a full copy of the repository on each workstation, but it turns out that Mercurial repositories are usually smaller than a Subversion working directory. This is partly because Mercurial stores changes to files in a very efficient manner, and partly because Subversion stores a significant amount of extra data in a working directory in order to avoid accessing the server for some operations.

In Mercurial, changes to files are committed to a local repository as a unit of work, called a change set. The change sets can then be pushed or pulled from other repositories in order to share work with others. It’s then up to you how you organize your work and manage the exchange of change sets between repositories. You can have a single master repository that everyone pushes/pulls from, you can have multiple masters for better performance and reliability, you can have a hierarchical model where committers receive change sets from the community, then pass it up to module owners, who pass them up to release owners (from what I’ve heard the Linux Kernel is organized in this way), or even have no master and have members of the team exchange change sets directly with each others’ repositories (this sounds like too much work though).

Most of all, I was impressed with Mercurial’s attention to details in their design decisions:

• Mercurial never makes updates to a revision log of a file, just appends to it. This minimizes the opportunities for data getting corrupted; if you never update or delete data, there is very little chance of it being accidentally corrupted beyond repair.
• Mercurial stores the diffs to files (text and binary) rather than storing the full copy of a file for every revision. It then “replays” the diffs in order to reconstruct the full copy of a revision of a file when needed. This is much more efficient than it may seem, but after a large number of changes it can become expensive. To compensate for this, Mercurial stores a snapshot copy of a file revision for fast access when the chain of diffs becomes too long.
• Minimize locking – read and update operations have been ordered in such a manner that locking isn’t necessary for most commands. Reads and clone/pull operations don’t lock the repository at all. Mercurial uses locks to ensure only one process writes to a repository at a time, but read and clone/pull operations are not affected by this.
• Minimizing filesystem seek operations – Seek operations are relatively expensive, so Mercurial has been designed to minimize this. This is one of the reasons that Mercurial keeps repository metadata in a single directory (.hg at the root) rather than stored with each directory as Subversion does (.svn directories) – having separate metadata directories would require a seek for every directory.

I’ll follow up later with a post on setting up & working with Mercurial.