Sunday 5 December 2010

Shocking - www.amazon.co.uk disappears

This evening, I tried to buy a Christmas present for my wife that she had selected from Amazon's UK site. Imagine my surprise when I got a domain-parking page advertising naughty dating services!


Has Amazon allowed its domain registration to run out? It appears not - I checked the WHOIS entry. More probably, someone has hacked the DNS entry in "revenge" for last week's eviction of the WikiLeaks service from Amazon's cloud. Grow up, people!

In fact as I was trying to upload the above image, blogger.com also started to display the same kind of holding page. I had to hurriedly change to a different DNS. Who knows how long that one will last - we may be under some kind of sustained Internet hacking attack.

Trying to send an alert to anyone at Amazon by e-mail is, apparently, impossible. Nowhere on the amazon.com site is there a "contact us" link. The Amazon staff is better protected against any communication from their customers than EasyJet (see posts passim).

Saturday 13 November 2010

File Handles in the JVM

I recently spent three or four fruitless hours (spread across several days) trying to get a bunch of Fit tests to run reliably. The setup fixture was used to clean out the folders in which the program under test was intended to find its input data and write its output data. Subsequent fixtures created the input files, executed the program under test, and analysed the output files.

So far so good, but as the number of tests gradually increased we started to get failures apparently at random. I suspected that the setup fixture was not deleting all the files out of the input and output folders, so increased the rigour of this method by invoking the FileUtils deleteDirectory method and throwing a runtime exception if it failed even after five retries.

It turned out that this was frequently failing because some process was keeping file handles open on files within the test folder hierarchy. Under Windows, this prevents the containing folder from being deleted or moved. We suspected virus checkers, search indexers and the like, but it turned out to be the Fit test harness itself. Even though we had taken great care to flush and close every file stream used by our Fit fixtures, the VM was not always releasing the corresponding file handles and so the next invocation of the setup fixture failed.

The answer turned out to be surprisingly simple: invoke the garbage collector and finalizer before retrying the call to deleteDirectory. Now it works every time!

Code sample:

...
import static org.apache.commons.io.FileUtils.deleteDirectory;
...
public static void zapDirectory(String path) {
...
File directory = new File(path);
for (int i = RETRY_COUNT; directory.exists(); i--) {
try {
deleteDirectory(directory);
} catch(IOException e) {
if (i <= 0) {
throw new RuntimeException("Cannot delete folder \"
+ path + "\" because: " + e.getMessage());
}
try {
System.gc();
System.runFinalization();
Thread.sleep(SLEEP_INTERVAL);
} catch(InterruptedException e1) {
// don't care if interrupted
}
}
if (directory.exists()) {
throw new RuntimeException("Cannot delete folder \"
+ path + "\" for an unknown reason");
}
}
}

Monday 2 August 2010

Migrating a Subversion repository

In my current project, there are a number of development teams whose code repositories have grown, separately or together, in a somewhat haphazard fashion. As part of the initiative to introduce a more streamlined development process, therefore, I want to host all repositories on one server and reorganise them in a more uniform and logical way.

Using Subversion, it is relatively straightforward to move whole repositories, but taking one branch out of an existing one to add to a new repository (without losing the change history!) turns out to be surprisingly hard. After much experimentation, the best way turns out to be via Mercurial's excellent "convert" extension. Unfortunately, conversion from Hg to Subversion simply fails under Windows, so you need to use Linux.

This particular customer has a totally Windows-based infrastructure, as well as a particularly draconian firewall that uses Windows authentication exclusively. The following recipe therefore had to take that into account. Names have been blanked out where necessary to preserve client confidentiality.

  1. Install the VMWare player and a Linux appliance, such as the Ubuntu 10.04 LTS image with VMware Tools.

  2. Lightly edit the VMWare virtual machine configuration file (Ubuntu.vmx) to ensure that it has at least 1024MB of RAM. Start up the Linux VM and upgrade existing packages. This turns out to be a little more difficult than you might expect, because the Synaptic package manager (unlike Firefox) is unable to pass your Windows username and password as NTLM credentials to the network proxy server. Therefore you have to proceed as follows:

    1. Make sure that the VMWare player's network connection is set to "NAT"

    2. In System -> Preferences -> Network Proxy, set the manual HTTP proxy to the server and and port that your Windows web browser uses. If the Web browser's Internet settings nominate a proxy auto-configuration script, you may be able to use that instead. Don't forget to set your Windows username and password. Apply system-wide (you will have to supply your Linux user password twice).

    3. Open Firefox and navigate to http://sourceforge.net/projects/cntlm/ - download the .deb package and allow the package installer to install it. This is a local proxy server that logs onto the firewall on your behalf.

    4. Now configure the cntlm service as follows:
      • As superuser, edit /etc/cntlm.conf and set the following values (see the documentation):
        • Username (your Windows user ID)
        • Domain (your Windows domain)
        • Workstation (your PC name - not the full FQDN)
        • Proxy (proxy FQDN:proxy port)
        • Proxy (backup proxy FQDN:proxy port)
        • Listen 3128
      • Make sure that the permissions on cntlm.conf are -rw-r--r--
      • As a normal user, run the following command:
        cntlm -v -I -M http://test.com
      • Copy and paste the resulting profile (the lines between -------------- markers) into the cntlm.conf file - search for "#Auth" to find the insertion point
      • As the superuser, start up the cntlm service:
        /etc/init.d/cntlm start

    5. In System -> Preferences -> Network Proxy, set the manual HTTP and FTP proxy servers to localhost and port 3128. Username and password should still be your Windows credentials. Apply system-wide as before. Shutdown Firefox and open it again to check that the proxy settings are correct.

    6. Open System -> Administration -> Synaptic Package Manager (it will request your Linux user password). Go to Settings and configure a manual proxy just as in System Preferences.

    7. Still in the package manager, click Reload. You should see the package files being downloaded successfully. Click "Mark All Upgrades".

    8. Still in the package manager, type "subversion" into the search box. Mark both "subversion" and its dependencies and "python-subversion" for installation. Similarly for "mercurial". Finally, click "apply", click the Apply button, and sit back.

    9. Part way through the installation process, you may encounter a warning that the Linux system image cannot be safely installed without GRUB. However, the grub loader has been superseded by grub-pc, so you can safely ignore this (click the "go ahead without grub" checkbox).

  3. You can also upgrade the VMWare tools, which the player will claim are out of date. This is of unproven usefulness. It doesn't take terribly long, so you may as well. After downloading the upgrade, VMWare Player mounts it as a CD image under /media (run "df" to find out what it's called). The trick is to unzip from there into /tmp or your home directory, because the mounted volume is read-only. Then run "sudo perl <install-script>" to upgrade the software.

  4. Assuming all has gone well, you should now be able to run Mercurial and Subversion from the command line. Test it by running the following commands:
    hg help
    svn help
    svnadmin help

  5. You need to cache the subversion authentication parameters in order that Mercurial can invoke subversion non-interactively. To do this, make sure you are logged in as an ordinary user (not root) and run:
    svn log -l 2 REPOSITORY_URL
    where REPOSITORY_URL is the address of the repository from which you wish to migrate a branch.
    When prompted for the "user" password, just hit return. Now enter the username of a real user with access to the target repository, and hit RETURN. Next enter that user's password and hit RETURN. You should be rewarded with two lines of history from that repository.

  6. You must enable the convert extension by uncommenting the line containing "hgext.convert =" in /etc/mercurial/hgrc.d/hgext.rc.

  7. Now you're all set to do the export. Create a folder (e.g. "repo_migration") and change directory into it. Create a file-mapping file such as myproject_mapping.txt:

    include branches/Development/myproject
    rename branches/Development/myproject trunk

    See the hg convert documentation for details of the mapping format.

  8. The following command does the business. It will probably process around 100 revisions per minute on average, so you've got time to make lots of cups of tea:
    - hg convert -d svn --filemap myproject_mapping.txt -s svn REPOSITORY_URL NEW_REPOSITORY
    where REPOSITORY_URL is the address of the repository from which you are migrating, and NEW_REPOSITORY is the name of the local Subversion repository you want to create as the result.

  9. If the conversion starts to slow down, you can open a second terminal and, as superuser, set the nice level of the running hg process to -20, which should increase its priority to maximum. I'm not sure how much this helps, since the process seems mainly network-bound.

  10. Finally, copy the new local repository to a location from which your Subversion server can serve it, and Bob's your parent's brother.

It has just occurred to me that the reason this doesn't work under Windows is probably because we didn't have the python-subversion package installed. However, that is just a guess because the diagnostics given (exit code 1) provided no clue.

Ideally of course, you should just export the existing repository to Mercurial and let everyone use that instead of Subversion from now on. Unfortunately some tools we want to use, such as JIRA are not yet Mercurial-aware. Besides, we don't want to incur the overhead of training developers to work with a new SCM tool - it isn't that long since they even began to use Subversion.

Sunday 1 August 2010

Custom image folder for Joomla Contacts

Because my church web site has a large number of contacts, I wanted to put all the photos in their own folder. Joomla's Contacts component normally expects photos to be stored at the default image location, /images/stories. I have a folder named portraits under this folder, but Joomla doesn't recurse into it when you select an image in the contact form.

It turns out that you have to make only three small edits to get this working, but I spent a frustrating few hours investigating before I found out how. To save you the trouble, here's the recipe. All file paths are relative to the Joomla root.

  1. Edit /components/com_contact/views/contact/tmpl/default.php. On line 52, change
    <?php echo JHTML::_('image', 'images/stories' . '/'.$this->contact->image, ...
    to
    <?php echo JHTML::_('image', 'images/stories/portraits' . '/'.$this->contact->image, ...
  2. Edit /administrator/components/com_contact/admin.contact.php. On line 227, change
    $lists['image'] = JHTML::_('list.images', 'image', $row->image );
    to
    $lists['image'] = JHTML::_('list.images', 'image', $row->image, NULL, '/images/stories/portraits/' );
  3. Edit /administrator/components/com_contact/admin.contact.html.php. On line 437, change
    jsimg='../<?php echo $cparams->get('image_path'); ?>/' + ...
    to
    jsimg='../<?php echo $cparams->get('image_path'); ?>/portraits/' + ...

Afterwards, you will have to open each contact you've previously created and select the portrait anew. I did try doing a bulk edit by exporting the contacts table to CSV and globally replacing, but that was a dismal failure.

The Joomla maintainers are apparently part way through parameterising the location of the stories folder (see the call to get('image_path') above). Ideally the location of the portraits would be a separate parameter. I didn't have time to do this, but this investigation clearly reveals the places where the parameter would need to be used.

Friday 23 July 2010

Turn any program into a Windows service

Today, Oisin Mulvihill introduced me to his company's open source product, Service Station. It's very simple, very effective and stunningly useful. I am really surprised that it isn't much better known. With one short shell command, you can install any script or application (headless, command-line or GUI) as a Windows service that can be started, stopped and managed through the Services control panel. As a bonus, Service Station restarts programs that have crashed, cleans up all spawned processes when stopping a service, and provides comprehensive logging via the Windows Event Viewer - it really could not be simpler for the system administrator. It is suitable for every version of Windows from 2000 onwards.

Great Music Venue

Last night my friends and colleagues from the Zuhlke office and I, collectively known as Karma4Sale, enjoyed playing to a packed house at Peter Parker's Rock'n'Roll Club in Denmark Street, London.

Thanks to everyone who came along to listen and encourage us, to drummer Chris Sluman (sorry Chris, this is the best photo I have and you're completely invisible in it!), to the brilliant support act of young musicians from the Brit school, and not least to the incredible Mr Peter Parker, who made the evening possible in the first place and gave hands-on support throughout. I sincerely recommend his excellent club if you fancy a great night out either as a listener or as a performer.

Wednesday 19 May 2010

BCS SPA2010 conference

Final day of the conference - it's gone much too quickly as usual. My favourite session to date has been the brief whirlwind tour of agile practices given by Gwyn Morfey and Laurie Young of New Bamboo - "the Sword of Integration". This was a highly interactive session that involved everyone standing up and moving about enthusiastically, which despite the cramped room, meant that we all ended up remembering something instantly useful from the session.

By the way, the sword of integration itself is just one example of an instant solution to a pressing problem. The situation was that multiple developers checking in their changes would cause each other to have merge conflicts. The solution: a paper "sword" quickly assembled, which when held conferred on the holder the right to check in - and hit anyone who checked in when they shouldn't. The principle being illustrated is "just try it" - there is no need to get it absolutely right first time. If it doesn't work, we can change it later.

Sunday 2 May 2010

Real-time arrival information for London Buses

... should be arriving some time in 2011 - provided, of course, that there are no unforeseen traffic problems :-)

By the way, the link in that post has become obsolete. Try this one instead.

Meanwhile, some enterprising private individual has connected live running information from various feeds with Google's Map API to create Live Bus - but for the stops I am principally interested in, it has no information feeds as yet.

Thursday 28 January 2010

Ubuntu Desktop 9.10

For a project recently begun, I needed a Linux virtual machine and I downloaded the excellent Chrysaor VMWare appliance pre-built with Ubuntu Desktop 9.10 (karmic koala) and the VMWare Tools (which actually work as specified - you can cut/paste text and even entire files between your host OS and the VM).

NB I was only able to download this using a direct internet connection, because our company firewall blocks BitTorrent. There don't appear to be any FTP or HTTP mirrors of these VMWare images.

There were just a couple of problems with this virtual machine image, for which I have discovered workarounds.
  1. Timezone persistently defaults to EST, no matter what I do in the time/date control panel. It turns out that certain system files need editing by hand (change to reflect your actual location and language):
    • sudo rm /etc/localtime
    • sudo ln -s /usr/share/zoneinfo/Europe/London /etc/localtime
    • sudo vi /etc/timezone -- set it to Europe/London
    • sudo vi /etc/profile -- add the following line at the end:
      export TZ=/etc/localtime
    • sudo vi /etc/default/locale -- set it as follows:
      LANG="en_GB.UTF-8"
  2. Login crash: for a while I thought I had broken the system by installing some upgrade or enabling shared folders. The symptom was that when I logged in, the progress bar would be displayed for a few seconds but instead of displaying the desktop, Ubuntu would restart the windowing system and display the login prompt again. I eventually discovered by trial and error that this was due to starting up in full-screen on my second monitor. Provided I always login while the VM is running in a non-maximised window, it seems very reliable. If you're suffering similar problems, look in your vmware.log file for messages similar to the following:
    VMXVmdbGuestLaunchMenuCB: buf is null.
    VMXVmdbGuestLaunchMenuCB: failed: ret = VMDB failure
    GuestRpc: Channel 3 reinitialized.
I recommend that the first thing you do is create a new user ID for yourself with admin rights, and log in under that user to do everything else.

I upgraded Mercurial to 1.4.1 and TortoiseHg to 0.9.2 by downloading the following Debian packages and installing them manually using dpkg -i:
  1. mercurial_1.4.1-1_i386.deb and mercurial-common_1.4.1-1_all.deb from ftp://mirror.switch.ch/mirror/ubuntu/pool/universe/m/mercurial/
  2. tortoisehg_0.9.2-1_all.deb and tortoisehg-nautilus_0.9.2-1_all.deb from ftp://mirror.switch.ch/mirror/ubuntu/pool/universe/t/tortoisehg/
Note that Tortoise doesn't work unless you install the python-iniparse package. Version 0.3.1-1 works with Tortoise 0.9.2-1.

I was able to get Eclipse (Galileo) as a Ubuntu package and installed the excellent HgEclipse plugin from http://hge.javaforge.com/hgeclipse using the Eclipse "Install New Software" option.

Wednesday 13 January 2010

Craftsmanship for Teams

Very interesting discussion thread on Software Craftsmanship as a team exercise. In response to Cory Foy's posting, Steven Smith makes an interesting analogy with coaching a sports team and says how that is actually carried through in his practice.

Mercurial - managing a repository

Last Autumn, I set up a repository server with Mercurial. While not particularly difficult, it took a while to get all the configuration exactly correct, in particular, to leave no security loopholes. With some assistance from Zuhlke's IT centre, this was satisfactorily completed and the server is in production use.

It takes a little while to get used to the idea that you can't just access files directly from the repository server, as you can with CVS and Subversion. Instead, anyone wishing to get hold of the repository contents has to install a copy of the Mercurial software on their own machine (on Windows PCs, my choice would be TortoiseHg) and "clone" the repository. While this makes it a bit tedious for someone who just needs one or two files out of a big repository, it's great for teams of developers who can each check files in and out of their local workspace before merging a consistent set of changes with the central development branch.

I was quite surprised, however, by the lack of web-based admin facilities. There ought to be a control panel to allow suitably authorised people to create and delete collections of repositories with specified access permissions, set up and delete repository user accounts etc. Currently you have to use the shell commands and in some cases, this involves superuser privileges (e.g. to change ownership of new repositories to www-data so that the Apache FCGI scripts can access them).

Because I was unable to find such a facility either natively or under Webmin, I have embarked on developing some scripts to provide these functions. I have adapted the Python scripts and modules, HTML templates and stylesheets of Mercurial itself for the purpose. Anyone keen to help or to get access to this code is invited to contact me - immo.huneke _a_ zuhlke.com.

Unimaginative train operators

I was infuriated by the actions of the railway company SouthEastern and London Underground this morning, which added to the aggravation caused by the weather.

Due to an accident (I assume) Stafford Road between Wallington and the A23 was gridlocked, so my usual bus into Croydon was not likely to run any time soon.

I trudged through the snow to the nearest railway station and did manage to catch a train into London Bridge. It took nearly twice as long as normal, and when we arrived, London Bridge station was very congested. The Underground station entrance had been closed "due to overcrowding" (due to lack of trains I think). Everyone went to Platform 6 to try to get on a service to Charing Cross or Waterloo.

Several trains came and went, too full to take more than a few passengers. Then a half-empty one came in and stopped but the doors did not open. Eventually an announcement was made that this train was not due to stop at this station. But since it had stopped anyway at a red light, why didn't they open the doors and clear some of the backlog of frustrated passengers?

Wednesday 6 January 2010

Making Life Hell?

A friend, who I think is a Moslem, sent me the attached "quick and dirty" translation of an article in Milliyet by Semih I'diz. He asked for his name not to be published. My response is shown below.


They are Making Life Hell for Moslems

Semih I˙diz sidiz@milliyet.com.tr

Monday, 4 January 2010

First, a Nigerian madman called Ömer Faruk Abdulmuttalib is caught while trying to blow up an American passenger plane. After that, a young man from Somalia, whose name was not disclosed, tries to kill the danish cartoonist Kurt W Estergaard, who caused indignation in the moslem world with his Mohammed cartoons.

Then, life becomes unbearable again for the moslems in the west. There are increased signs that this is not a transitional but a permanent condition. The congregation leaders of millions of moslems living in the west remain spectators while their living space shrinks.

Basic reason is "cultural differences". The aforementioned congregation leaders cannot bring themselves to demonstrate the condemning stance that would satisfy the western public, although they live in the west and share the benefits of doing so.

They limit their reactions to a few "politically required" public statements. When they try to explain the situation by referring to Israel or to the US invasion of Iraq, they only increase the indignation felt against moslems.

We are talking about the psychology of the western society within which millions of moslems are living. The fact that people walking on the streets in the west are killed in the name of Islam cancels out the "explanatory arguments" of the moslem congregation leaders.

Under these circumstances, the "man on the street" in the west, who is not very sophisticated to start with, cannot see the difference between the few madmen misusing religion and the millions of moslems going after their business and who want to live without upsetting anyone.

To summarise, one cannot explain to a Dutch person the killling of Theo Van Gogh in the name of Islam, irrespective of what an unpleasant person he was. One cannot explain the knife-and-axe attack on Westergaard while he was sitting at home with his grandson, even if he was beginning to show extreme right-wing tendencies.

In the meanwhile, we can also see that the aggressive madmen we mentioned are providing the opportunity to the extreme Right in the west for developing its anti-moslem arguments. Indeed, after the attack on Westergaard, calls of "Throw out the moslems that do not adhere to our laws and customs" have immediately started in Denmark.

One also needs to see the minaret referendum in Switzerland in the same framework. Actually, Switzerland is one of the countries in Europe with the least of issues with its moslem minority. In spite of this, the terror being carried out in the world in the name of Islam has caused even the Swiss public to take an anti-moslem stance.

A study carried out recently by the "European Union Agency for Fundamental Rights" disclosed the negative developments in Europe regarding moslems. According to this study, if you are a moslem or you "look like a moslem", your chances of finding a job, moving to certain neighbourhoods or even moving freely within society is becoming increasingly difficult.

Some even see the plans for installing "naked body" scanners in European airports as a result of the attack of the Nigerian madman in the same framework. After a certain point, being naked is not such an issue for a western person. There will be many who will say "I accept it for security".

The situation among moslems is totally different due to the understanding of privacy as well as religious and cultural values. The covering up of women already shows the sensitivity there. According to some voices, once this equipment is in operation, moslems will give up travelling to the west or travelling outside the countries they live in the west.

Thus, the factors of security and control will increase. Those who do not accept these factors will "return to where they came from and remain there". This is the largest consequence from the deeds of the 9/11 criminals, the madmen from Somalia and Nigeria.

They do not contribute to improving the living conditions of moslems in the Gaza Strip or Iraq. Just the opposite, either consciously or intentionally, they play into the hands of the extreme-rightists in Europe, such as Geert Wilders.

To put it in a nutshell, they bring about the conditions for making life hell for moslems in the west.


Hi xxxxx,

Thanks for the article. Thought-provoking reading first thing in the morning! I will think carefully about forwarding it. I think that particularly the point about the full-body scanners at airports (and major rail stations as well, quite possibly - did you know about that?) is well made. I hadn't previously considered their likely impact on members of certain religions, though I don't feel particularly comfortable myself even being "patted down" and wouldn't relish the thought of effectively being inspected without my clothes on.

It isn't only Moslems who are being impacted by the extra "security" measures. They make life more expensive and less convenient for everyone, though possibly to a greater degree for members of certain races and religious minorities in the West. In some ways, this plays into the hands of the extremists. During WWII, the British public was determined not to let the enemy destroy their way of life, so beyond taking a few precautions such as a blackout against bombers, and evacuating children from the big cities, they cheerfully carried on regardless of the war. I think it's time to rediscover that resolve to live proudly according to the values we believe in, keep our eyes and ears open, and take a few risks for the sake of a well-balanced society.

Lutheran theologian Dietrich Bonhoeffer wisely warned (in 1934!) that safety and security are mutually exclusive. What he meant was that if nation states entered a race to build up their defences in the name of peace, they would endanger everyone by exposing them to a greater risk of war - and how right he was. We are facing a similar situation today. There is a global threat to civilised society from some sadly deluded religious fanatics. Those who think that the threat can be averted by military means are similarly deluded and similarly threaten civil liberties. Another way must be found. For almost 30 years, the United Kingdom faced fanatical Irish nationalists who were prepared to sacrifice innocent lives on a massive scale to pursue their political ends. It was only when we abandoned our entrenched positions and began to negotiate that an accommodation became possible. It even turned out that at least some of the nationalists were not such odious individuals after all.

Instead of expending prodigious resources in the "war on terror", we should all be pulling together to face common threats, such as climate change and mass extinction of rare species. Also see this story. Let's hope that the world can come to its senses in the nick of time, as it usually seems to manage. Is that a hopeful message with which to begin the year 2010?

Best regards,
Immo.

Software Engineering Best Practices

Tom Gilb recommends Capers Jones's latest book Software Engineering Best Practices. From my quick reading of the preface and introduction, it certainly looks like a thorough piece of work!