Migrating a Subversion repository

In my current project, there are a number of development teams whose code repositories have grown, separately or together, in a somewhat haphazard fashion. As part of the initiative to introduce a more streamlined development process, therefore, I want to host all repositories on one server and reorganise them in a more uniform and logical way.

Using Subversion, it is relatively straightforward to move whole repositories, but taking one branch out of an existing one to add to a new repository (without losing the change history!) turns out to be surprisingly hard. After much experimentation, the best way turns out to be via Mercurial's excellent "convert" extension. Unfortunately, conversion from Hg to Subversion simply fails under Windows, so you need to use Linux.

This particular customer has a totally Windows-based infrastructure, as well as a particularly draconian firewall that uses Windows authentication exclusively. The following recipe therefore had to take that into account. Names have been blanked out where necessary to preserve client confidentiality.

1. Install the VMWare player and a Linux appliance, such as the Ubuntu 10.04 LTS image with VMware Tools.


2. Lightly edit the VMWare virtual machine configuration file (Ubuntu.vmx) to ensure that it has at least 1024MB of RAM. Start up the Linux VM and upgrade existing packages. This turns out to be a little more difficult than you might expect, because the Synaptic package manager (unlike Firefox) is unable to pass your Windows username and password as NTLM credentials to the network proxy server. Therefore you have to proceed as follows:


1. Make sure that the VMWare player's network connection is set to "NAT"


2. In System -> Preferences -> Network Proxy, set the manual HTTP proxy to the server and and port that your Windows web browser uses. If the Web browser's Internet settings nominate a proxy auto-configuration script, you may be able to use that instead. Don't forget to set your Windows username and password. Apply system-wide (you will have to supply your Linux user password twice).


3. Open Firefox and navigate to http://sourceforge.net/projects/cntlm/ - download the .deb package and allow the package installer to install it. This is a local proxy server that logs onto the firewall on your behalf.


4. Now configure the cntlm service as follows:
   • As superuser, edit /etc/cntlm.conf and set the following values (see the documentation):
     • Username (your Windows user ID)
     • Domain (your Windows domain)
     • Workstation (your PC name - not the full FQDN)
     • Proxy (proxy FQDN:proxy port)
     • Proxy (backup proxy FQDN:proxy port)
     • Listen 3128
   • Make sure that the permissions on cntlm.conf are -rw-r--r--
   • As a normal user, run the following command:
     cntlm -v -I -M http://test.com
   • Copy and paste the resulting profile (the lines between -------------- markers) into the cntlm.conf file - search for "#Auth" to find the insertion point
   • As the superuser, start up the cntlm service:
     /etc/init.d/cntlm start


5. In System -> Preferences -> Network Proxy, set the manual HTTP and FTP proxy servers to localhost and port 3128. Username and password should still be your Windows credentials. Apply system-wide as before. Shutdown Firefox and open it again to check that the proxy settings are correct.


6. Open System -> Administration -> Synaptic Package Manager (it will request your Linux user password). Go to Settings and configure a manual proxy just as in System Preferences.


7. Still in the package manager, click Reload. You should see the package files being downloaded successfully. Click "Mark All Upgrades".


8. Still in the package manager, type "subversion" into the search box. Mark both "subversion" and its dependencies and "python-subversion" for installation. Similarly for "mercurial". Finally, click "apply", click the Apply button, and sit back.


9. Part way through the installation process, you may encounter a warning that the Linux system image cannot be safely installed without GRUB. However, the grub loader has been superseded by grub-pc, so you can safely ignore this (click the "go ahead without grub" checkbox).


3. You can also upgrade the VMWare tools, which the player will claim are out of date. This is of unproven usefulness. It doesn't take terribly long, so you may as well. After downloading the upgrade, VMWare Player mounts it as a CD image under /media (run "df" to find out what it's called). The trick is to unzip from there into /tmp or your home directory, because the mounted volume is read-only. Then run "sudo perl <install-script>" to upgrade the software.


4. Assuming all has gone well, you should now be able to run Mercurial and Subversion from the command line. Test it by running the following commands:
   hg help
   svn help
   svnadmin help


5. You need to cache the subversion authentication parameters in order that Mercurial can invoke subversion non-interactively. To do this, make sure you are logged in as an ordinary user (not root) and run:
   svn log -l 2 REPOSITORY_URL
   where REPOSITORY_URL is the address of the repository from which you wish to migrate a branch.
   When prompted for the "user" password, just hit return. Now enter the username of a real user with access to the target repository, and hit RETURN. Next enter that user's password and hit RETURN. You should be rewarded with two lines of history from that repository.


6. You must enable the convert extension by uncommenting the line containing "hgext.convert =" in /etc/mercurial/hgrc.d/hgext.rc.


7. Now you're all set to do the export. Create a folder (e.g. "repo_migration") and change directory into it. Create a file-mapping file such as myproject_mapping.txt:
   
   include branches/Development/myproject
   rename branches/Development/myproject trunk
   
   See the hg convert documentation for details of the mapping format.


8. The following command does the business. It will probably process around 100 revisions per minute on average, so you've got time to make lots of cups of tea:
   - hg convert -d svn --filemap myproject_mapping.txt -s svn REPOSITORY_URL NEW_REPOSITORY
   where REPOSITORY_URL is the address of the repository from which you are migrating, and NEW_REPOSITORY is the name of the local Subversion repository you want to create as the result.


9. If the conversion starts to slow down, you can open a second terminal and, as superuser, set the nice level of the running hg process to -20, which should increase its priority to maximum. I'm not sure how much this helps, since the process seems mainly network-bound.


10. Finally, copy the new local repository to a location from which your Subversion server can serve it, and Bob's your parent's brother.

It has just occurred to me that the reason this doesn't work under Windows is probably because we didn't have the python-subversion package installed. However, that is just a guess because the diagnostics given (exit code 1) provided no clue.

Ideally of course, you should just export the existing repository to Mercurial and let everyone use that instead of Subversion from now on. Unfortunately some tools we want to use, such as JIRA are not yet Mercurial-aware. Besides, we don't want to incur the overhead of training developers to work with a new SCM tool - it isn't that long since they even began to use Subversion.

Passwords passé?

I've just read a highly amusing and (for Twitter) embarrassing story about the loss of confidential information stored by a Twitter staffer under Google Apps.

The article closes by trotting out the usual advice about making usernames and passwords secure - make them long and cryptic, don't use the same credentials for more than one service, and don't write them down. However, I heard a piece on BBC Radio 4 not long ago in which Damian Grammaticus investigated computer security and hacking. It showed that any Windows password under 14 characters long could be cracked in seconds, regardless of how many punctuation characters and digits it contained. Since we're all forced to register with dozens, if not hundreds of different IT services on the Internet and our employers' networks, it simply isn't possible for standard-issue mortals to memorise that many cryptic usernames and passwords (particularly where IT policy demands that the password be changed regularly).

Common approaches to overcome these wetware limitations include

• Writing usernames and passwords down - not recommended because anyone could find them and steal your precious secret information
• Using a standard username and password for many different sites - again not recommended because if a thief finds out one valid combination, (s)he can access all your accounts
• Generating usernames and passwords by means of some function, which you can compute mentally without too much effort and which preferably takes the current month and/or the URL of the service as parameters - this works OK in certain situations, but because the algorithm can't be terribly complicated, someone could reverse-engineer it and have access to all your accounts
• Using a password manager application: something that you install on your laptop (say) and which stores all your usernames and passwords for you, encrypted of course, so that you can only gain access using a master password or even biometric authentication - disadvantages may include the risk of losing the device or the master access key, as well as the inconvenience of having to have the device to hand whenever you need one of your account credentials (not to mention the cost of the password manager itself)
• Trusting an Internet service such as DataInherit to manage the keys for you, which has the advantage that should you die or be incapacitated, a nominated person or persons can automatically be given access to selected accounts in your portfolio - disadvantages include the price and the number of steps you have to go through to retrieve any given username and password

Surely the time is ripe for a security architecture that fully separates authentication from authorisation. Depending on your spending power and level of paranoia, you could procure authentication services from any reputable trust authority of your choosing, which could make use of a single very secure identifier and pass phrase, optionally combined with more stringent authentication factors such as digital certificates, RSA SecurID devices, or even biometrics. The identity thus established would be securely and trustworthily conveyed to whatever service you were trying to access - regardless of whether it is on a company intranet, the public Internet or your ATM network.

Unfortunately the Twitters and Googles of this world, not to mention the myriad online shopping sites, still want you to register with them directly and not with some third-party authentication provider. This is because they believe that there's value in knowing your real world (meatspace) name and address, age, marital status and what have you so that they can push targeted marketing at you. It doesn't seem to bother them that the majority of the data thus gathered is probably untrue, because most Internet users are paranoid about giving away such information.

I want single sign-on to be ubiquitous, effortless, virtually unhackable and foolproof (as well as heritable, as directed by me, when I leave this world, so that my digital assets are not locked away forever). By "ubiquitous", I mean that I should be able to use the same sign-on for my employer's network as for any client's network which wants to authorise me to access some selection of their IT services, for Internet services on which I register and preferably at the bank's ATM too. By "effortless", I mean that authenticating myself to all these services is necessary only once per computer per day and ideally doesn't require me to memorise anything, or at most one thing. A physical token in addition, or a biometric, would be acceptable.

The theoretical foundations already exist, underpinning such standards as Kerberos and WS-Trust. But to make it work in the real world of competing and highly disparate IT services, the community of service providers would need to discover some kind of shared interest in implementing it as standard. A grand computing challenge for the new millennium, perhaps?