A Software Practitioner Still Practising

Ubuntu Linux desktop 11.10 proxy settings solved

2012-01-31T11:04:00.000Z

Watch out for system proxy settings in Ubuntu Desktop 11.10 (at least in its LXDE guise, which is what I am running in a VM). The control panel "Network" has disappeared and even if you install it using apt or Synaptic you'll find that it doesn't have any proxy options. gconf does, but seems to have no effect. Answer is to edit /usr/share/applications/chromium-browser.desktop and append

--proxy-pac-url="http://example.com/"

to the Exec line.

Fun with Robots, Arduino and Android

2011-09-24T10:17:00.000+01:00

Together with Mike Hogg from my firm, Zühlke Engineering, I recently developed a demonstration project over a three-day training exercise.We purchased a simple robot and added a set of three infra-red sensors capable of detecting a black line marked out on the ground with electrician's tape. We had to develop a line-sensing algorithm and control logic to follow the line and stop at stations (a short length of tape at right angles to the line) until instructed to move on.

We followed a test-driven development (TDD) approach for C++ using CppUTest, as described by James Grenning, to create a state machine to turn sensor readings into motor control instructions (with hindsight, we might have saved time by resorting to the Machine Objects library). To my surprise, once we loaded the finished firmware to the Arduino board, it worked very well with only a bit of tweaking of the motor speed settings corresponding to "hard left", "slight left" etc.

Then we linked this with an application we developed for an Android tablet computer (Motorola Xoom) that could be used to instruct the robot to go to any selected station, pick up/drop a "payload" (only conceptually) and return to base. The two components communicated by Bluetooth radio link.

Finally, the tablet app was linked via a RESTful Web Service interface to an "enterprise" system developed by other colleagues on the training course, that controlled the whole delivery network and knew which consignments had to be delivered to what stations. The tablet would send a message to the enterprise server telling it where the robot had arrived, and receive back an instruction containing the name of the next station to move to. The robot would then move off automatically after 5 seconds.

The whole thing was enormous fun (as well as very instructive) and was demonstrated successfully to the entire group (see photo).

Jason Gorman and Simon Peyton Jones, among others, have recently been at pains to point out the shortcomings of IT education in the UK. Thinking about the above exercise, it occurred to me that it is just the sort of project that could engender enthusiasm for the subject from a wide range of school students. It involves lots of varied tasks from assembling the hardware to designing the user interface, and of course the devising of suitable protocols between all the components.

Simon Peyton Jones of Microsoft Research will be presenting an interactive talk about Computing At School on 2nd November in London, and the following month (6 December) Mike and I plan to do a session on the Robot Shop exercise at the same location. Anyone interested will be most welcome.

Generate test data for Java development easily

2011-08-10T14:22:00.002+01:00

A colleague has just pointed out the release of Andy Gibson's DataFactory library as a Maven artifact. It generates arbitrarily long lists of random names and addresses, dates, date ranges (with arbitrary constraints), random selections from lists, numbers, strings, Internet addresses etc. etc.

Inspiring the next generation of developers

2011-06-19T15:19:00.006+01:00

Jason Gorman has published a thought-provoking opinion piece asking why schools are doing so little to promote IT as an enticing career option and to equip kids to take advantage of it. His analysis points to a lack of suitably qualified teachers (so what's new?) and a commensurate lack of ambition on the part of exam boards to make the syllabus industry-relevant.

Do get in touch with Jason to take part in a summit at Bletchley Park (where else?) on 25th August 2011 to identify practical measures that can be taken to improve this dire situation.

I think this is an issue the BCS should be addressing with all dispatch instead of tying itself in knots over some elusive ideal of making all IT practitioners professionally qualified. Some of the most proficient software practitioners I know have no professional qualifications at all.

Perils of using an elderly FitLibrary

2011-05-25T16:55:00.002+01:00

I learned about a really useful tool today while trying to hunt down a bug in a set of Fit automated acceptance tests.

Last September, my colleague Darren Bishop had set up the automated acceptance test infrastructure for a Maven/Java project I subsequently joined. He used version 20080812 of the Maven dependency (org.fitnesse.fitlibrary), probably because it was the latest available in the Maven Central repository. Thus the stage was set for my troubles last night and this morning!

While working on a small enhancement with a developer in the customer's product support unit, I noticed that one of the Fit tests was producing a blank report and not being counted in the summary totals. A comparison with other spreadsheets that did run properly revealed no significant differences.

Back at base, it quickly became apparent that I had to single-step through the program to find out where it was going wrong. But I could find no source jar for this version of fitlibrary - the latest dependency version available with source was from 2006. What to do?

After a futile attempt to desk-walkthrough the code of FolderRunner and SpreadsheetRunner, Keith Braithwaite suggested decompiling the class files. JD-Eclipse to the rescue! I installed it in my Eclipse and found it so insanely useful that I will never want to be without it again. Basically it creates beautifully formatted source code for you on demand (obviously without the programmer's original comments and documentation). This means that you can set breakpoints and single-step through library code even if you have no source attachment for your library. Installation was really easy. After that, if you single-step into a library method, the "source" is opened automatically. You can also open a .class file directly in the package explorer by browsing the libraries on your project's build path.

There's just one slight gotcha: the line numbers in the decompiled code rarely match the original, so if the class file contains line number annotations your instruction pointer and breakpoints may appear to be a few lines away from the location reported in the debugger's thread view. Ignore the cursor and read off the current line number from the comment at the start of each line. You can turn off the display of metadata and line numbers in the JD preferences, and I have done so - it needs a restart of Eclipse to take effect, but setting breakpoints is much easier afterwards.

So, what was the solution to the mystery?

In the FitLibrary class SpreadsheetRunner, the method collectTables() used an Iterator to return the rows of the test script one at a time and convert them into arrays of values that the runner could subsequently invoke to execute the test:

HSSFRow row = (HSSFRow)it.next();
HSSFCell[] cells = getCells(row);
String[] borderedCellValues = getBorderedCellValues(cells, workbook);

Unfortunately, getCells() dimensioned an array of HSSFCell objects using the value returned by a row:

private HSSFCell[] getCells(HSSFRow row) {
int maxCell = row.getLastCellNum();
HSSFCell[] cells = new HSSFCell[maxCell];
...
return cells;
}

When the row was "empty", getLastCellNum() returned -1, which caused the next line to throw a NegativeArraySizeException. This exception was caught by FolderRunner.runFile(), which attempted to report the problem to the standard output, but this didn't appear in the Maven test log.

I have not found out what makes POI decide that a row is "empty". There was only a single spreadsheet row in the entire suite of tests that was interpreted in this way. I achieved a workaround by pasting a copy of another blank row of cells onto this row, but the danger is that anyone could inadvertently reintroduce the error.

Lessons:

You can still use the Java debugger if you don't have the source. JD-Eclipse worked very well for me, even though it is only at version 0.1.3. (I wonder if it will decompile classes written in other languages such as JRuby?)

It's time someone made a proper Maven dependency, with sources and javadoc, of the latest Fit, Fitnesse and FitLibrary versions - the current version of FitLibrary, dated April 5th, 2011 in fact solves the exact problem I encountered!

So long then, Ken Olsen

2011-02-11T14:40:00.003Z

Ken Olsen, founder of DEC, passed away aged 84 on Sunday 7th February. Read a short obituary here. I used to consider DEC as the nimble young upstart snapping at IBM's heels and making computing affordable. My first real computing experience was on the undergraduate DEC-10 system (running TOPS10) at Leeds - and it shaped my subsequent career, though various PDP-8 and PDP-11 minis featured even before that. RIP Ken.

The oddness of socks

2011-01-29T16:01:00.001Z

In a moment of boredom, and prompted by a recent survey of wacky interview questions, I plotted out the probability graph of finding a matching pair of socks at random, based on the number of black ones among 25 grey ones. It turns out that you need only 3 black socks to drop the probability of a matching pair from 100% to 80%; 9 in 25 gets you to 60%. Anything over that makes very little difference up to 25, where the probability drops to a minimum as you would expect. After that it starts rising again slowly, but never reaches 100%. YOU NEVER HAVE A LESS THAN 48.9795918% CHANCE OF GETTING A MATCHING PAIR. I find that oddly comforting.

Oracle 10g problem under Windows Server 2003

2011-01-14T18:38:00.002Z

It cost me the best part of two days' detective work to fix a problem with the Oracle DB Control service. Symptoms include:

"CONFIG: Waiting for service 'OracleDBConsoleorcl' to fully start" in cfgtoollogs\dbca\orcl\cloneDBCreation.log
"The oracleDBconsoleorcl service could not be started"
"[Orion Launcher] ERROR app.ContextInitializer contextInitialized.272 - Integration Class not found: oracle.sysman.ias.ias.IASIntegration" in emoms.log

This started to happen after a failed import using the data pump (impdp) trashed the database and I rashly thought that I might clear up the problem by uninstalling and reinstalling Oracle 10.2.0.4.

Background: For a client project, we needed a local copy of the client's schema and stored procedures. I found it remarkably straightforward to install the Oracle product and a default database instance named orcl. However, attempting to import the client's exported schema and stored procs caused a rash of errors and then the SYSTEM user's password had been corrupted too. My suggestion: use the deprecated exp and imp tools instead. They're much more straightforward to use, AND the wonderful DDL Wizard is able to convert the exported file to DDL and PL/SQL that you can run in SQLPlus instead of overwriting your database contents willy-nilly.

Solution: I wish there was one. The problem appears to occur in the final phase of the database configuration assistant - after creating and configuring the database it tries to start up the console service for it and fails, for as-yet unfathomable reasons. Could it be to do with the fact that we also run MS SQL Server on the same box? Yet this was fine before I tried to reinstall Oracle. (As I eventually learned, I need not have redone the entire installation, because the database configuration assistant is quite good at deleting the database instance and creating a new one - in fact, this is what the installer invokes anyway).

Luckily the whole thing runs on a virtual machine, so I am going to get the previous week's VM backup image restored. Luckily we hadn't been doing a lot of work on that machine in the meantime.

My grateful thanks to the many bloggers who came up with possible solutions - none of which worked for me:

drop and reconfigure the repository
work around Windows's tendency to use different hostnames depending on network connectivity

Software: Craft or Trade? You decide

2011-01-12T10:50:00.001Z

Dan North has started a serious discussion with his insightful article about the Software Craftsmanship movement and where its manifesto possibly misses the point.

Breaking up the big rocks

2011-01-06T16:59:00.001Z

A colleague has circulated Richard Lawrence's nine patterns for dividing up user stories. These are going to be very useful. I wonder if someone will add more to make the magic Baker's Dozen?

New Ideas in Social Marketing

2011-01-06T14:36:00.001Z

Attended a stunning event last night at the BCS London conference rooms. The speakers were Thomas Power of Ecademy and Lee Bryant of Headshift. The event was jointly organised by Richard Tandoh, Dalim Basu and Sara Misell on behalf of the North London Branch, Elayne Coakes on behalf of the BCS Sociotechnical specialist group and me on behalf of BCS Software Practice Advancement.

I learned that my use of social media was probably at the very raw beginner level - and moreover, that all the companies I have ever worked for, including my current employer, are at or below that level. If you want some insights into your level of influence measured by the effectiveness of your online behaviour (and how it compares to others), try Klout or PeerIndex. These applications use people's Twitter usernames as the primary key, so if you aren't on Twitter, you don't even get to first base.

Thomas showed how the stream of information we receive from the Web is growing exponentially - his FriendFeed home page was updating with new events more than once a second. A potentially incredibly useful tool for cutting this deluge down to size is made by My6Sense. They make a reader for iPhone, iPod Touch and Android devices that watches your Twitter, Facebook, News and RSS streams and brings the most important items to your attention. Thomas believes it takes around 30 hours to train it. More importantly, this being a social media world, once something has caught your attention, the my6sense app lets you share it easily with your own network of contacts.

In the Q&A session, I found out about a recently launched online service that is attempting to cross-fertilise a crowd-sourced knowledge repository (think Wikipedia) with a personal reputation index (like eBranding Me). A fun way to find out more is to take the Quora programming challenge.

The discussion in the pub afterwards was most illuminating too, though it veered away from social media to discussing the disaster that is likely to befall the civilised world when, not if, the next massive Coronal Mass Ejection occurs. This event is expected to occur within 10-24 months as of this writing and is likely to knock out not only communications networks (particularly satellite-based ones) but all manner of computer systems, data networks and power grids. Since everything these days is dependent on computers, we're likely to suffer supply shortages of everything from drinking water to fuel. As for withdrawing money from your bank account, you will probably be well advised to forget it for a few months. The guy telling us this has inside knowledge as a result of working for a major data security company and claims that they have working solutions to sell that will handle the backup and disaster recovery requirements of computer users from home users to data centres (remember that from the first detection of a CME, the world has only 15 minutes to secure everything before the plasma storm hits). His advice is to print out your bank statements and keep them safe along with non-perishable emergency supplies; also to have a spare laptop computer somewhere wrapped in several layers of tin foil to protect it from EMP. Contact me if you want to know more!

We need to re-learn that it's OK to fail

2011-01-03T09:10:00.002Z

While trying to find the answer to a festive season quiz, I stumbled across this blog post. Check out the "more fialures" (sic) section - it makes reassuringly heartening reading...

Shocking - www.amazon.co.uk disappears

2010-12-05T22:00:00.003Z

This evening, I tried to buy a Christmas present for my wife that she had selected from Amazon's UK site. Imagine my surprise when I got a domain-parking page advertising naughty dating services!

Has Amazon allowed its domain registration to run out? It appears not - I checked the WHOIS entry. More probably, someone has hacked the DNS entry in "revenge" for last week's eviction of the WikiLeaks service from Amazon's cloud. Grow up, people!

In fact as I was trying to upload the above image, blogger.com also started to display the same kind of holding page. I had to hurriedly change to a different DNS. Who knows how long that one will last - we may be under some kind of sustained Internet hacking attack.

Trying to send an alert to anyone at Amazon by e-mail is, apparently, impossible. Nowhere on the amazon.com site is there a "contact us" link. The Amazon staff is better protected against any communication from their customers than EasyJet (see posts passim).

File Handles in the JVM

2010-11-13T16:37:00.003Z

I recently spent three or four fruitless hours (spread across several days) trying to get a bunch of Fit tests to run reliably. The setup fixture was used to clean out the folders in which the program under test was intended to find its input data and write its output data. Subsequent fixtures created the input files, executed the program under test, and analysed the output files.

So far so good, but as the number of tests gradually increased we started to get failures apparently at random. I suspected that the setup fixture was not deleting all the files out of the input and output folders, so increased the rigour of this method by invoking the FileUtils deleteDirectory method and throwing a runtime exception if it failed even after five retries.

It turned out that this was frequently failing because some process was keeping file handles open on files within the test folder hierarchy. Under Windows, this prevents the containing folder from being deleted or moved. We suspected virus checkers, search indexers and the like, but it turned out to be the Fit test harness itself. Even though we had taken great care to flush and close every file stream used by our Fit fixtures, the VM was not always releasing the corresponding file handles and so the next invocation of the setup fixture failed.

The answer turned out to be surprisingly simple: invoke the garbage collector and finalizer before retrying the call to deleteDirectory. Now it works every time!

Code sample:


...
import static org.apache.commons.io.FileUtils.deleteDirectory;
...
    public static void zapDirectory(String path) {
        ...
        File directory = new File(path);
        for (int i = RETRY_COUNT; directory.exists(); i--) {
            try {
                deleteDirectory(directory);
            } catch(IOException e) {
                if (i <= 0) {
                    throw new RuntimeException("Cannot delete folder \"
                        + path + "\" because: " + e.getMessage());
                }
                try {
                    System.gc();
                    System.runFinalization();
                    Thread.sleep(SLEEP_INTERVAL);
                } catch(InterruptedException e1) {
                    // don't care if interrupted
                }
            }
            if (directory.exists()) {
                throw new RuntimeException("Cannot delete folder \"
                    + path + "\" for an unknown reason");
            }
        }
    }

Migrating a Subversion repository

2010-08-02T20:10:00.005+01:00

In my current project, there are a number of development teams whose code repositories have grown, separately or together, in a somewhat haphazard fashion. As part of the initiative to introduce a more streamlined development process, therefore, I want to host all repositories on one server and reorganise them in a more uniform and logical way.

Using Subversion, it is relatively straightforward to move whole repositories, but taking one branch out of an existing one to add to a new repository (without losing the change history!) turns out to be surprisingly hard. After much experimentation, the best way turns out to be via Mercurial's excellent "convert" extension. Unfortunately, conversion from Hg to Subversion simply fails under Windows, so you need to use Linux.

This particular customer has a totally Windows-based infrastructure, as well as a particularly draconian firewall that uses Windows authentication exclusively. The following recipe therefore had to take that into account. Names have been blanked out where necessary to preserve client confidentiality.

Install the VMWare player and a Linux appliance, such as the Ubuntu 10.04 LTS image with VMware Tools.

Lightly edit the VMWare virtual machine configuration file (Ubuntu.vmx) to ensure that it has at least 1024MB of RAM. Start up the Linux VM and upgrade existing packages. This turns out to be a little more difficult than you might expect, because the Synaptic package manager (unlike Firefox) is unable to pass your Windows username and password as NTLM credentials to the network proxy server. Therefore you have to proceed as follows:

Make sure that the VMWare player's network connection is set to "NAT"

In System -> Preferences -> Network Proxy, set the manual HTTP proxy to the server and and port that your Windows web browser uses. If the Web browser's Internet settings nominate a proxy auto-configuration script, you may be able to use that instead. Don't forget to set your Windows username and password. Apply system-wide (you will have to supply your Linux user password twice).

Open Firefox and navigate to http://sourceforge.net/projects/cntlm/ - download the .deb package and allow the package installer to install it. This is a local proxy server that logs onto the firewall on your behalf.

Now configure the cntlm service as follows:
- As superuser, edit /etc/cntlm.conf and set the following values (see the documentation):
  - Username (your Windows user ID)
  - Domain (your Windows domain)
  - Workstation (your PC name - not the full FQDN)
  - Proxy (proxy FQDN:proxy port)
  - Proxy (backup proxy FQDN:proxy port)
  - Listen 3128
- Make sure that the permissions on cntlm.conf are -rw-r--r--
- As a normal user, run the following command:
  cntlm -v -I -M http://test.com
- Copy and paste the resulting profile (the lines between -------------- markers) into the cntlm.conf file - search for "#Auth" to find the insertion point
- As the superuser, start up the cntlm service:
  /etc/init.d/cntlm start

In System -> Preferences -> Network Proxy, set the manual HTTP and FTP proxy servers to localhost and port 3128. Username and password should still be your Windows credentials. Apply system-wide as before. Shutdown Firefox and open it again to check that the proxy settings are correct.

Open System -> Administration -> Synaptic Package Manager (it will request your Linux user password). Go to Settings and configure a manual proxy just as in System Preferences.

Still in the package manager, click Reload. You should see the package files being downloaded successfully. Click "Mark All Upgrades".

Still in the package manager, type "subversion" into the search box. Mark both "subversion" and its dependencies and "python-subversion" for installation. Similarly for "mercurial". Finally, click "apply", click the Apply button, and sit back.

Part way through the installation process, you may encounter a warning that the Linux system image cannot be safely installed without GRUB. However, the grub loader has been superseded by grub-pc, so you can safely ignore this (click the "go ahead without grub" checkbox).

You can also upgrade the VMWare tools, which the player will claim are out of date. This is of unproven usefulness. It doesn't take terribly long, so you may as well. After downloading the upgrade, VMWare Player mounts it as a CD image under /media (run "df" to find out what it's called). The trick is to unzip from there into /tmp or your home directory, because the mounted volume is read-only. Then run "sudo perl <install-script>" to upgrade the software.

Assuming all has gone well, you should now be able to run Mercurial and Subversion from the command line. Test it by running the following commands:
hg help
svn help
svnadmin help

You need to cache the subversion authentication parameters in order that Mercurial can invoke subversion non-interactively. To do this, make sure you are logged in as an ordinary user (not root) and run:
svn log -l 2 REPOSITORY_URL
where REPOSITORY_URL is the address of the repository from which you wish to migrate a branch.
When prompted for the "user" password, just hit return. Now enter the username of a real user with access to the target repository, and hit RETURN. Next enter that user's password and hit RETURN. You should be rewarded with two lines of history from that repository.

You must enable the convert extension by uncommenting the line containing "hgext.convert =" in /etc/mercurial/hgrc.d/hgext.rc.

Now you're all set to do the export. Create a folder (e.g. "repo_migration") and change directory into it. Create a file-mapping file such as myproject_mapping.txt:

include branches/Development/myproject
rename branches/Development/myproject trunk

See the hg convert documentation for details of the mapping format.

The following command does the business. It will probably process around 100 revisions per minute on average, so you've got time to make lots of cups of tea:
- hg convert -d svn --filemap myproject_mapping.txt -s svn REPOSITORY_URL NEW_REPOSITORY
where REPOSITORY_URL is the address of the repository from which you are migrating, and NEW_REPOSITORY is the name of the local Subversion repository you want to create as the result.

If the conversion starts to slow down, you can open a second terminal and, as superuser, set the nice level of the running hg process to -20, which should increase its priority to maximum. I'm not sure how much this helps, since the process seems mainly network-bound.

Finally, copy the new local repository to a location from which your Subversion server can serve it, and Bob's your parent's brother.

It has just occurred to me that the reason this doesn't work under Windows is probably because we didn't have the python-subversion package installed. However, that is just a guess because the diagnostics given (exit code 1) provided no clue.

Ideally of course, you should just export the existing repository to Mercurial and let everyone use that instead of Subversion from now on. Unfortunately some tools we want to use, such as JIRA are not yet Mercurial-aware. Besides, we don't want to incur the overhead of training developers to work with a new SCM tool - it isn't that long since they even began to use Subversion.

Custom image folder for Joomla Contacts

2010-08-01T15:56:00.005+01:00

Because my church web site has a large number of contacts, I wanted to put all the photos in their own folder. Joomla's Contacts component normally expects photos to be stored at the default image location, /images/stories. I have a folder named portraits under this folder, but Joomla doesn't recurse into it when you select an image in the contact form.

It turns out that you have to make only three small edits to get this working, but I spent a frustrating few hours investigating before I found out how. To save you the trouble, here's the recipe. All file paths are relative to the Joomla root.

Edit /components/com_contact/views/contact/tmpl/default.php. On line 52, change
<?php echo JHTML::_('image', 'images/stories' . '/'.$this->contact->image, ...
to
<?php echo JHTML::_('image', 'images/stories/portraits' . '/'.$this->contact->image, ...
Edit /administrator/components/com_contact/admin.contact.php. On line 227, change
$lists['image'] = JHTML::_('list.images', 'image', $row->image );
to
$lists['image'] = JHTML::_('list.images', 'image', $row->image, NULL, '/images/stories/portraits/' );
Edit /administrator/components/com_contact/admin.contact.html.php. On line 437, change
jsimg='../<?php echo $cparams->get('image_path'); ?>/' + ...
to
jsimg='../<?php echo $cparams->get('image_path'); ?>/portraits/' + ...

Afterwards, you will have to open each contact you've previously created and select the portrait anew. I did try doing a bulk edit by exporting the contacts table to CSV and globally replacing, but that was a dismal failure.

The Joomla maintainers are apparently part way through parameterising the location of the stories folder (see the call to get('image_path') above). Ideally the location of the portraits would be a separate parameter. I didn't have time to do this, but this investigation clearly reveals the places where the parameter would need to be used.

Turn any program into a Windows service

2010-07-23T22:54:00.001+01:00

Today, Oisin Mulvihill introduced me to his company's open source product, Service Station. It's very simple, very effective and stunningly useful. I am really surprised that it isn't much better known. With one short shell command, you can install any script or application (headless, command-line or GUI) as a Windows service that can be started, stopped and managed through the Services control panel. As a bonus, Service Station restarts programs that have crashed, cleans up all spawned processes when stopping a service, and provides comprehensive logging via the Windows Event Viewer - it really could not be simpler for the system administrator. It is suitable for every version of Windows from 2000 onwards.

Great Music Venue

2010-07-23T22:29:00.002+01:00

Last night my friends and colleagues from the Zuhlke office and I, collectively known as Karma4Sale, enjoyed playing to a packed house at Peter Parker's Rock'n'Roll Club in Denmark Street, London.

Thanks to everyone who came along to listen and encourage us, to drummer Chris Sluman (sorry Chris, this is the best photo I have and you're completely invisible in it!), to the brilliant support act of young musicians from the Brit school, and not least to the incredible Mr Peter Parker, who made the evening possible in the first place and gave hands-on support throughout. I sincerely recommend his excellent club if you fancy a great night out either as a listener or as a performer.

BCS SPA2010 conference

2010-05-19T10:28:00.002+01:00

Final day of the conference - it's gone much too quickly as usual. My favourite session to date has been the brief whirlwind tour of agile practices given by Gwyn Morfey and Laurie Young of New Bamboo - "the Sword of Integration". This was a highly interactive session that involved everyone standing up and moving about enthusiastically, which despite the cramped room, meant that we all ended up remembering something instantly useful from the session.

By the way, the sword of integration itself is just one example of an instant solution to a pressing problem. The situation was that multiple developers checking in their changes would cause each other to have merge conflicts. The solution: a paper "sword" quickly assembled, which when held conferred on the holder the right to check in - and hit anyone who checked in when they shouldn't. The principle being illustrated is "just try it" - there is no need to get it absolutely right first time. If it doesn't work, we can change it later.

Real-time arrival information for London Buses

2010-05-02T12:59:00.004+01:00

... should be arriving some time in 2011 - provided, of course, that there are no unforeseen traffic problems :-)

By the way, the link in that post has become obsolete. Try this one instead.

Meanwhile, some enterprising private individual has connected live running information from various feeds with Google's Map API to create Live Bus - but for the stops I am principally interested in, it has no information feeds as yet.

Ubuntu Desktop 9.10

2010-01-28T09:30:00.004Z

For a project recently begun, I needed a Linux virtual machine and I downloaded the excellent Chrysaor VMWare appliance pre-built with Ubuntu Desktop 9.10 (karmic koala) and the VMWare Tools (which actually work as specified - you can cut/paste text and even entire files between your host OS and the VM).

NB I was only able to download this using a direct internet connection, because our company firewall blocks BitTorrent. There don't appear to be any FTP or HTTP mirrors of these VMWare images.

There were just a couple of problems with this virtual machine image, for which I have discovered workarounds.

Timezone persistently defaults to EST, no matter what I do in the time/date control panel. It turns out that certain system files need editing by hand (change to reflect your actual location and language):
- sudo rm /etc/localtime
- sudo ln -s /usr/share/zoneinfo/Europe/London /etc/localtime
- sudo vi /etc/timezone -- set it to Europe/London
- sudo vi /etc/profile -- add the following line at the end:
  export TZ=/etc/localtime
- sudo vi /etc/default/locale -- set it as follows:
  LANG="en_GB.UTF-8"
Login crash: for a while I thought I had broken the system by installing some upgrade or enabling shared folders. The symptom was that when I logged in, the progress bar would be displayed for a few seconds but instead of displaying the desktop, Ubuntu would restart the windowing system and display the login prompt again. I eventually discovered by trial and error that this was due to starting up in full-screen on my second monitor. Provided I always login while the VM is running in a non-maximised window, it seems very reliable. If you're suffering similar problems, look in your vmware.log file for messages similar to the following:
```
VMXVmdbGuestLaunchMenuCB: buf is null.
VMXVmdbGuestLaunchMenuCB: failed: ret = VMDB failure
GuestRpc: Channel 3 reinitialized.
```

I recommend that the first thing you do is create a new user ID for yourself with admin rights, and log in under that user to do everything else.

I upgraded Mercurial to 1.4.1 and TortoiseHg to 0.9.2 by downloading the following Debian packages and installing them manually using dpkg -i:

mercurial_1.4.1-1_i386.deb and mercurial-common_1.4.1-1_all.deb from ftp://mirror.switch.ch/mirror/ubuntu/pool/universe/m/mercurial/
tortoisehg_0.9.2-1_all.deb and tortoisehg-nautilus_0.9.2-1_all.deb from ftp://mirror.switch.ch/mirror/ubuntu/pool/universe/t/tortoisehg/

Note that Tortoise doesn't work unless you install the python-iniparse package. Version 0.3.1-1 works with Tortoise 0.9.2-1.

I was able to get Eclipse (Galileo) as a Ubuntu package and installed the excellent HgEclipse plugin from http://hge.javaforge.com/hgeclipse using the Eclipse "Install New Software" option.

Craftsmanship for Teams

2010-01-13T11:30:00.002Z

Very interesting discussion thread on Software Craftsmanship as a team exercise. In response to Cory Foy's posting, Steven Smith makes an interesting analogy with coaching a sports team and says how that is actually carried through in his practice.

Mercurial - managing a repository

2010-01-13T10:25:00.002Z

Last Autumn, I set up a repository server with Mercurial. While not particularly difficult, it took a while to get all the configuration exactly correct, in particular, to leave no security loopholes. With some assistance from Zuhlke's IT centre, this was satisfactorily completed and the server is in production use.

It takes a little while to get used to the idea that you can't just access files directly from the repository server, as you can with CVS and Subversion. Instead, anyone wishing to get hold of the repository contents has to install a copy of the Mercurial software on their own machine (on Windows PCs, my choice would be TortoiseHg) and "clone" the repository. While this makes it a bit tedious for someone who just needs one or two files out of a big repository, it's great for teams of developers who can each check files in and out of their local workspace before merging a consistent set of changes with the central development branch.

I was quite surprised, however, by the lack of web-based admin facilities. There ought to be a control panel to allow suitably authorised people to create and delete collections of repositories with specified access permissions, set up and delete repository user accounts etc. Currently you have to use the shell commands and in some cases, this involves superuser privileges (e.g. to change ownership of new repositories to www-data so that the Apache FCGI scripts can access them).

Because I was unable to find such a facility either natively or under Webmin, I have embarked on developing some scripts to provide these functions. I have adapted the Python scripts and modules, HTML templates and stylesheets of Mercurial itself for the purpose. Anyone keen to help or to get access to this code is invited to contact me - immo.huneke _a_ zuhlke.com.

Unimaginative train operators

2010-01-13T10:23:00.001Z

I was infuriated by the actions of the railway company SouthEastern and London Underground this morning, which added to the aggravation caused by the weather.

Due to an accident (I assume) Stafford Road between Wallington and the A23 was gridlocked, so my usual bus into Croydon was not likely to run any time soon.

I trudged through the snow to the nearest railway station and did manage to catch a train into London Bridge. It took nearly twice as long as normal, and when we arrived, London Bridge station was very congested. The Underground station entrance had been closed "due to overcrowding" (due to lack of trains I think). Everyone went to Platform 6 to try to get on a service to Charing Cross or Waterloo.

Several trains came and went, too full to take more than a few passengers. Then a half-empty one came in and stopped but the doors did not open. Eventually an announcement was made that this train was not due to stop at this station. But since it had stopped anyway at a red light, why didn't they open the doors and clear some of the backlog of frustrated passengers?

Making Life Hell?

2010-01-06T20:34:00.004Z

A friend, who I think is a Moslem, sent me the attached "quick and dirty" translation of an article in Milliyet by Semih I'diz. He asked for his name not to be published. My response is shown below.

They are Making Life Hell for Moslems

Semih I˙diz sidiz@milliyet.com.tr

Monday, 4 January 2010

First, a Nigerian madman called Ömer Faruk Abdulmuttalib is caught while trying to blow up an American passenger plane. After that, a young man from Somalia, whose name was not disclosed, tries to kill the danish cartoonist Kurt W Estergaard, who caused indignation in the moslem world with his Mohammed cartoons.

Then, life becomes unbearable again for the moslems in the west. There are increased signs that this is not a transitional but a permanent condition. The congregation leaders of millions of moslems living in the west remain spectators while their living space shrinks.

Basic reason is "cultural differences". The aforementioned congregation leaders cannot bring themselves to demonstrate the condemning stance that would satisfy the western public, although they live in the west and share the benefits of doing so.

They limit their reactions to a few "politically required" public statements. When they try to explain the situation by referring to Israel or to the US invasion of Iraq, they only increase the indignation felt against moslems.

We are talking about the psychology of the western society within which millions of moslems are living. The fact that people walking on the streets in the west are killed in the name of Islam cancels out the "explanatory arguments" of the moslem congregation leaders.

Under these circumstances, the "man on the street" in the west, who is not very sophisticated to start with, cannot see the difference between the few madmen misusing religion and the millions of moslems going after their business and who want to live without upsetting anyone.

To summarise, one cannot explain to a Dutch person the killling of Theo Van Gogh in the name of Islam, irrespective of what an unpleasant person he was. One cannot explain the knife-and-axe attack on Westergaard while he was sitting at home with his grandson, even if he was beginning to show extreme right-wing tendencies.

In the meanwhile, we can also see that the aggressive madmen we mentioned are providing the opportunity to the extreme Right in the west for developing its anti-moslem arguments. Indeed, after the attack on Westergaard, calls of "Throw out the moslems that do not adhere to our laws and customs" have immediately started in Denmark.

One also needs to see the minaret referendum in Switzerland in the same framework. Actually, Switzerland is one of the countries in Europe with the least of issues with its moslem minority. In spite of this, the terror being carried out in the world in the name of Islam has caused even the Swiss public to take an anti-moslem stance.

A study carried out recently by the "European Union Agency for Fundamental Rights" disclosed the negative developments in Europe regarding moslems. According to this study, if you are a moslem or you "look like a moslem", your chances of finding a job, moving to certain neighbourhoods or even moving freely within society is becoming increasingly difficult.

Some even see the plans for installing "naked body" scanners in European airports as a result of the attack of the Nigerian madman in the same framework. After a certain point, being naked is not such an issue for a western person. There will be many who will say "I accept it for security".

The situation among moslems is totally different due to the understanding of privacy as well as religious and cultural values. The covering up of women already shows the sensitivity there. According to some voices, once this equipment is in operation, moslems will give up travelling to the west or travelling outside the countries they live in the west.

Thus, the factors of security and control will increase. Those who do not accept these factors will "return to where they came from and remain there". This is the largest consequence from the deeds of the 9/11 criminals, the madmen from Somalia and Nigeria.

They do not contribute to improving the living conditions of moslems in the Gaza Strip or Iraq. Just the opposite, either consciously or intentionally, they play into the hands of the extreme-rightists in Europe, such as Geert Wilders.

To put it in a nutshell, they bring about the conditions for making life hell for moslems in the west.

Hi xxxxx,

Thanks for the article. Thought-provoking reading first thing in the morning! I will think carefully about forwarding it. I think that particularly the point about the full-body scanners at airports (and major rail stations as well, quite possibly - did you know about that?) is well made. I hadn't previously considered their likely impact on members of certain religions, though I don't feel particularly comfortable myself even being "patted down" and wouldn't relish the thought of effectively being inspected without my clothes on.

It isn't only Moslems who are being impacted by the extra "security" measures. They make life more expensive and less convenient for everyone, though possibly to a greater degree for members of certain races and religious minorities in the West. In some ways, this plays into the hands of the extremists. During WWII, the British public was determined not to let the enemy destroy their way of life, so beyond taking a few precautions such as a blackout against bombers, and evacuating children from the big cities, they cheerfully carried on regardless of the war. I think it's time to rediscover that resolve to live proudly according to the values we believe in, keep our eyes and ears open, and take a few risks for the sake of a well-balanced society.

Lutheran theologian Dietrich Bonhoeffer wisely warned (in 1934!) that safety and security are mutually exclusive. What he meant was that if nation states entered a race to build up their defences in the name of peace, they would endanger everyone by exposing them to a greater risk of war - and how right he was. We are facing a similar situation today. There is a global threat to civilised society from some sadly deluded religious fanatics. Those who think that the threat can be averted by military means are similarly deluded and similarly threaten civil liberties. Another way must be found. For almost 30 years, the United Kingdom faced fanatical Irish nationalists who were prepared to sacrifice innocent lives on a massive scale to pursue their political ends. It was only when we abandoned our entrenched positions and began to negotiate that an accommodation became possible. It even turned out that at least some of the nationalists were not such odious individuals after all.

Instead of expending prodigious resources in the "war on terror", we should all be pulling together to face common threats, such as climate change and mass extinction of rare species. Also see this story. Let's hope that the world can come to its senses in the nick of time, as it usually seems to manage. Is that a hopeful message with which to begin the year 2010?

Best regards,
Immo.

Software Engineering Best Practices

2010-01-06T11:30:00.001Z

Tom Gilb recommends Capers Jones's latest book Software Engineering Best Practices. From my quick reading of the preface and introduction, it certainly looks like a thorough piece of work!

Another space tourist - so what?

2009-09-05T21:47:00.003+01:00

I've just read about Guy Laliberté's Poetic Social Mission for the One Drop Foundation. The seventh private space traveller is doing more with his once-in-a-lifetime opportunity than just enjoying himself - he plans to stage a worldwide music and arts event involving himself aboard the ISS, to draw attention to the worldwide need for fair and equitable access to water.

The guest list sounds brilliant (Peter Gabriel, U2, Shakira and many others) - the downside is that the two-hour spectacular is due to kick off at 8pm Eastern on October 9th, which makes it about 1am in our time zone. I think it's being broadcast via YouTube, so perhaps we'll be able to watch it later on? Details about this are a bit sketchy at the moment - the site just says "broadcast right here".

Save Bletchley Park with Boffoonery!

2009-08-05T21:58:00.003+01:00

The title says it all really.

Jason Gorman has managed to get some amazing events off the ground in the past couple of years and this one should be absolutely excellent. It's called Boffoonery! ("buffoonery for boffins") - a once-only comedy benefit gig to raise money for the extremely important wartime heritage site (Station X) and unique computing museum that is Bletchley Park near Milton Keynes.

Jason's partners in crime are comedy writer James Cary, whose credits include hit TV comedies like My Hero and My Family and who also writes the cult BBC Radio 4 sitcom Hut 33, which is set at Bletchley Park, and veteran TV and radio comedy producer David Tyler.

The show is still taking shape, but Jason is delighted to be able to confirm that Robert Llewellyn (Red Dwarf), Richard Herring (Fist Of Fun, This Morning With Richard Not Judy) and cerebral stand-up Robin Ince are on the bill. More big name comedians and comedy writers are TBA, so keep en eye out for updates.

The best way we can support Bletchley Park is to send a message to government and funding bodies about how much we care. If the show were to sell out, and sell out quickly, that would send a very clear message indeed.

And since you ask - yes, you can help. Tickets go on sale on Wednesday Aug 12th at 9am. Please help to spread the word and create a splash on Wednesday. Here are some of the ways you can help:

Buy your tickets on Aug 12th
Tell your friends and colleagues about Boffoonery!, and encourage them to buy their tickets on Aug 12th
Print off copies of the show's A4 poster, which you can download from www.boffoonery.com, and stick them up on your noticeboards at work
Blog about Boffoonery, and pass on these suggestions about how people can help save Bletchley Park
If you're a Twitter user, on the morning of Aug 12th send a tweet about #boffoonery and www.boffoonery.com, and RT any #boffoonery tweets that cross your path

It's going to be brilliant evening, and your kindness and generosity will ensure that generations to come will be able to see and experience the achievements of those brilliant boffins whose ingenuity saved 22 million lives and ushered in the information age.

You can find out more by visiting www.boffoonery.com, or following #boffoonery on Twitter or "boffoonery" on Facebook.

A pity there isn't a theatre big enough in Bletchley Park itself to host this event! That said, the Bloomsbury Theatre is easier for me to get to...

Don't forget: tickets go on sale on 12th August 2009 at 9am and the show takes place on Tuesday Nov 3rd.

PS other efforts to support the National Museum of Computing at Bletchley Park include this amazing URL Shortener called Station X. Keep up the good work, guys!

Gimme Shelter

2009-07-27T09:36:00.002+01:00

A friend has been made homeless - as a result of a bout of illness, he got behind with his rent payments. One morning his landlord just turned up with a heavy friend and dumped him and most of his belongings outside without a key.

While this is clearly an illegal eviction (the landlord should have taken the court route), my friend is unable to defend himself because he's still not well and besides, he's sure that no-one will believe his version of events. Well, it is fairly monstrous.

Today he rang Shelter to try to obtain help (having already had a pretty hopeless interview with the emergency housing office in his borough - they took the attitude that since he didn't have a court order evicting him, he must have made himself intentionally homeless). He reports that he got an answerphone message that began "Welcome to Shelter. Please note that we cannot provide accommodation..."

Well, what on Earth is Shelter for then?

I am also disappointed that The Salvation Army was unwilling to help him - I have been supporting it with a regular monthly donation for years as I was under the impression that it provided help to the homeless.

Passwords passé?

2009-07-20T13:48:00.004+01:00

I've just read a highly amusing and (for Twitter) embarrassing story about the loss of confidential information stored by a Twitter staffer under Google Apps.

The article closes by trotting out the usual advice about making usernames and passwords secure - make them long and cryptic, don't use the same credentials for more than one service, and don't write them down. However, I heard a piece on BBC Radio 4 not long ago in which Damian Grammaticus investigated computer security and hacking. It showed that any Windows password under 14 characters long could be cracked in seconds, regardless of how many punctuation characters and digits it contained. Since we're all forced to register with dozens, if not hundreds of different IT services on the Internet and our employers' networks, it simply isn't possible for standard-issue mortals to memorise that many cryptic usernames and passwords (particularly where IT policy demands that the password be changed regularly).

Common approaches to overcome these wetware limitations include

Writing usernames and passwords down - not recommended because anyone could find them and steal your precious secret information
Using a standard username and password for many different sites - again not recommended because if a thief finds out one valid combination, (s)he can access all your accounts
Generating usernames and passwords by means of some function, which you can compute mentally without too much effort and which preferably takes the current month and/or the URL of the service as parameters - this works OK in certain situations, but because the algorithm can't be terribly complicated, someone could reverse-engineer it and have access to all your accounts
Using a password manager application: something that you install on your laptop (say) and which stores all your usernames and passwords for you, encrypted of course, so that you can only gain access using a master password or even biometric authentication - disadvantages may include the risk of losing the device or the master access key, as well as the inconvenience of having to have the device to hand whenever you need one of your account credentials (not to mention the cost of the password manager itself)
Trusting an Internet service such as DataInherit to manage the keys for you, which has the advantage that should you die or be incapacitated, a nominated person or persons can automatically be given access to selected accounts in your portfolio - disadvantages include the price and the number of steps you have to go through to retrieve any given username and password

Surely the time is ripe for a security architecture that fully separates authentication from authorisation. Depending on your spending power and level of paranoia, you could procure authentication services from any reputable trust authority of your choosing, which could make use of a single very secure identifier and pass phrase, optionally combined with more stringent authentication factors such as digital certificates, RSA SecurID devices, or even biometrics. The identity thus established would be securely and trustworthily conveyed to whatever service you were trying to access - regardless of whether it is on a company intranet, the public Internet or your ATM network.

Unfortunately the Twitters and Googles of this world, not to mention the myriad online shopping sites, still want you to register with them directly and not with some third-party authentication provider. This is because they believe that there's value in knowing your real world (meatspace) name and address, age, marital status and what have you so that they can push targeted marketing at you. It doesn't seem to bother them that the majority of the data thus gathered is probably untrue, because most Internet users are paranoid about giving away such information.

I want single sign-on to be ubiquitous, effortless, virtually unhackable and foolproof (as well as heritable, as directed by me, when I leave this world, so that my digital assets are not locked away forever). By "ubiquitous", I mean that I should be able to use the same sign-on for my employer's network as for any client's network which wants to authorise me to access some selection of their IT services, for Internet services on which I register and preferably at the bank's ATM too. By "effortless", I mean that authenticating myself to all these services is necessary only once per computer per day and ideally doesn't require me to memorise anything, or at most one thing. A physical token in addition, or a biometric, would be acceptable.

The theoretical foundations already exist, underpinning such standards as Kerberos and WS-Trust. But to make it work in the real world of competing and highly disparate IT services, the community of service providers would need to discover some kind of shared interest in implementing it as standard. A grand computing challenge for the new millennium, perhaps?

Test-Driven Design is not testing

2009-06-15T17:40:00.001+01:00

I've recently worked with a team doing its first agile project (though one or two team members had been involved in an agile project before). The most difficult concept to get across was TDD - test driven design. I found that people really didn't grok the idea until I pair-programmed with them for a couple of hours. I wonder why that might be.

Dan North has suggested one possibility. He observed that newcomers to TDD often don't get the really big payback because they continue to think that TDD is mainly about testing - even if they will admit that writing the tests before the code leads to better quality code. They never transition to treating TDD as a design process, letting them discover the API to a component they're writing, nor to the realisation that TDD is about defining the behaviour of their component and its interactions with other components of the system.

Keith Braithwaite has put forward another consideration. In physical engineering disciplines, practitioners speed up their work process by using gauges. There are many kinds, from the simple spark plug gap gauge, which is simply a sliver of metal to slide between the electrodes, to electronic vernier calliper gauges that can be pre-set to a precise dimension with tolerances above and below. Each workpiece is tested at each stage of the process by checking its dimensions with the appropriate gauge(s). Workpieces that are out of tolerance are sent back for rework or scrapped. Our unit tests are a bit like that - they provide a safeguard that the software component we're working on still meets all its requirements following any engineering we've done.

It occurred to me today that unit and acceptance tests, particularly if automated, perform another valuable function in the context of an agile (especially a lean) development process. Whereas the waterfall processes familiar to most developers are characterised by "quality gates" at key stages, every single artifact in an agile development has its own little quality gate, manifested in the appropriate tests. This theoretically frees the development process from the usual bottlenecks that the quality gates tend to become.

I say "theoretically", because in many instances agile development projects have to take place within a quality system that doesn't take advantage of incremental delivery. Instead, continued approval and in many cases funding for the project tends to be contingent on passing the traditional quality gates following requirements analysis, functional specification, high-level design, low-level design, coding, integration, system test and acceptance. Project managers are therefore forced to conjure up some kind of spurious linkage between the milestones laid down in the rigid quality system and some arbitrary points along their product release plan. This can hamper their freedom to adjust the release plan in response to changing circumstances and emerging technical insights.

This could be avoided if the quality system could recognise that properly written tests represent every work product of a software development project apart from the code itself. It should therefore simply insist on a verification at each iteration (or at each release, perhaps) that the tests comprehensively and comprehensibly represent the requirements of the business on the system under development and that the required set of tests pass repeatably. I say "the required set" because there's always the possibility that some tests will intentionally fail - e.g. where they have been written to test features that are not yet in the current release.

In other words, TDD can be used to eliminate the quality-gate bottlenecks of quality systems that assume waterfall development processes.

24th March 2009: Theory of Constraints Challenged

2009-05-15T11:59:00.004+01:00

My sincere thanks to Kevin Rutherford and Allan Kelly for co-presenting a fascinating session about lean software development to the BCS Kingston & Croydon branch on 24th March this year, entitled "Lean, Constraints, Action!". The audience was excellent too and helped us re-create a famous experiment related by Eliyahu Goldratt in "The Goal".

(Click images to see a larger version)

I had participated in this game previously at the London XP Day 2008 (facilitated by Karl Scotland in an Open Space session). It is designed to demonstrate an intuitively paradoxical finding: that a lean, pull-oriented flow substantially reduces the amount of inventory or work in progress (WIP), while improving throughput.

However, I had a sneaky feeling that the experiment was biased, because in the first "push" simulation, the assembly line was not pre-loaded with WIP, while in the second "pull" simulation, the line was pre-loaded with workpieces at each "workstation's" input buffer up to either the maximum limit or to 50% of that limit. Therefore in a simulation of 10 rounds (equivalent to ten working days - approximately equal to the average cycle time in a six-workstation setup) the push simulation will only start to produce output towards the very end of the simulation, while the pull simulation will produce something from the very first day.

So I got Allan and Kevin to agree to vary the rules a little bit, to try to get closer to a "steady state" from the first "day". Before each of the two simulations, our teams placed three Lego blocks on each of the coasters representing the input buffers of the second through sixth workstations (the first workstation of course has the whole of the product backlog as its input hopper). In fact, as it turned out, four workpieces would have been closer to the true steady state in the pull simulation, even more in the push simulation.

Off our teams went and played the production line for ten rounds each. In the push simulation, the die was passed in order from workstation 1 to workstation 6 during each round and the number of workpieces transferred to the next input buffer was the number thrown, up to the number of pieces available in the workstation's input buffer. Instances of starvation were rare under this system, but did occur sometimes. At the end we counted up the number of pieces that had come off the end of the line and the number currently in progress (i.e. on any of the five input buffers for workstations 2 to 6).

In the pull simulation, the die was passed in the opposite direction and the input buffers were constrained to a maximum of six workpieces. So if the next input buffer had three pieces already in it and the player threw anything over a 3, they could only pass along 3 more workpieces (subject to their own input buffer holding at least 3, of course). Once again, the results after 10 rounds were compared.

The results didn't surprise me particularly, but I think some of the others were a little taken aback:

As you can see, the constraint resulted in both lower WIP and lower throughput. This makes sense when you consider that there were far more occasions during the pull game than during the push game when players were unable to process the full number of workpieces indicated by the die.

Looking back at the game notes, it is noted that if the simulation is run for much longer than 10 days, the pull (or Kanban) system "will rarely produce as much as the traditional push". This may have escaped the attention of some readers (or perhaps it's a more recent edit - I don't know).

My conclusion is that you get nothing for free. The cost of reducing WIP is reduced throughput, which is perfectly acceptable as long as you're aware of it. Software development projects are not production lines in any case, so it is very unlikely that any developer will sit around kicking her or his heels if the work runs out on a given day. There are always low priority tasks such as fettling the build system, cleaning up the documentation on the project Wiki, answering user support requests etc. - or just take the next item off the product backlog and raise the kanban limit temporarily.

Distributed bug-tracking in Haskell

2009-05-05T16:56:00.002+01:00

At the recent SPA 2009 conference, there was a lot of talk about functional programming, Haskell in particular (a couple of years ago, the flavour of the month had been Erlang). Just to prove that Haskell is no longer "just a research language", along comes DisTract, a distributed issue-tracking system that runs in Firefox browsers. If you're already using Git, Darcs, Mercurial or Monotone as your distributed software configuration management solution, the author reasoned, why shouldn't you be able to close bugs while you're off-line at the same time as you check in your fix? Caveat: I have not tried this yet, but it sounds like a really neat idea. Does anyone know of a user forum?

SPA2009 - first impressions

2009-04-09T14:02:00.002+01:00

Maybe I'm biased, of course, but for me the SPA2009 conference felt even better than last year's. I think it had the right mix of technical and non-technical sessions, mostly of very high quality, and a few really interesting BOF (birds of a feather) sessions - which for once, didn't feel as if they were just squeezed in at the last minute.

Functional programming (in particular, Haskell) was a major theme this year, as was testing. My eyes were opened to the possibility of doing test-driven development (TDD) in Haskell - in fact, functional languages are better at this than imperative (stateful) languages. It was also good to meet up with a lot of familiar faces at the joint XtC meeting on the Tuesday night.

My session notes are on my other laptop. I will try to post them on the SPA SG site over the Easter weekend, other duties permitting.

Automating testing of web sites with multiple browsers

2009-04-04T09:29:00.003+01:00

After several days of trying, I've just abandoned an attempt to automate web site tests with multiple browsers (Firefox, Internet Explorer, Safari and Opera under various operating systems). My findings are written up here. Briefly, I think that Selenium RC has the right architecture, but currently the linkage between the RC server and the browser only works 100% for Firefox. This isn't enough.

It seems from the documentation that Watir should support the functionality needed. Can anyone confirm that it's possible to test pages such as this one in at least three different browser types with it?

Update (9 April): I discovered that Selenium RC has a start-up parameter "-proxyInjectionMode", which is equivalent to choosing the "*pi..." launchers where available. Unfortunately this only works with "*firefox", so you might just as well use "*pifirefox". It isn't clear to me why it causes tests to fail when they launch a browser with "*mock".

I was also reminded at the SPA2009 conference that it might be worth trying out Cucumber, rSpec, Concordion or other tools, so long as they will drive more than one browser.

Why people think it's OK to cheat / steal

2009-03-19T10:51:00.002Z

Dan Ariely has given a brilliant talk about this and how it impacts the global economic crisis. Watch this on Ted - it'll be 16 minutes well spent. Good insights into human psychology and behaviour.

Tools to support agile methods

2009-03-17T15:19:00.011Z

The whole point of managing a project using an agile method is to use lots of coloured cards and post-it notes, but sometimes you have to fall back on a boring old software tool. The thorny question keeps arising: which one do I recommend?

Many different project management tools exist, at different price-points and levels of functionality. A long list is shown below. It is important to be clear about a project's requirements for a tool before selecting. Avoid the temptation to extend any tool you select or to spend much effort integrating it with the rest of your project environment – this is a sure way to lock yourself into a single supplier.

The following list of tools is not exhaustive, though I have kept adding to it as I came across new tools:

]project-open[

Achievo

ActiveCollab

Agilebuddy - reported by a comment on this post to be a full-featured agile project management software tool, which is easy to use and great for Scrum teams. Offered on a subscription model for US$9.95 per user per month

Agilefant

Agilo – based on the Trac issue-management tool and said to support only a single project at a time

airTODO – PMBOK rather than agile, but minimalist (an Eclipse plug-in)

AxoSoft's OnTime - available as a hosted cloud solution, a Windows native application or as a web application, one comment under this post has reported that it "great for agile / scrum development". Comparable in scope to Trac, but looks a lot more snazzy

Banana Scrum - according to one comment below, this is a nice, web based tool that helps without getting in team's way

Extreme Planner

Greenhopper – a JIRA plugin favoured by some Scrum teams

IceScrum

Mingle - provides a shared workspace for agile teams, supporting XP, Scrum and custom hybrid approaches. Includes a virtual card wall, wiki, charts, reports and more. Integrated with issue tracking and continuous build. Priced US$995 per user with substantial discounts for multiple licences, academic institutions etc.

Pivotal Tracker

ProjectCards – mixed approach that uses both physical cards and software

ProjectKoach

Rally

Rational Team Concert - integrates work item, continuous integration builds and software configuration management (SCM) on the collaborative infrastructure of the Jazz Team Server

redMine

Scrum Edge - someone commented below that it was much simpler to use than most other Scrum tools

Scrumworks - quite popular free open source package, heavily forms-based, can integrate with JIRA

Silver Catalyst

StuffPlanner – Zühlke's inhouse-developed browser-based tool now at version 0.1 and with an Eclipse Mylyn connector available; access can be provided by arrangement

TargetProcess - designed to support Scrum, XP and custom Agile processes particularly in the .NET environment. Includes a comprehensive set of tools and features, including defect-tracking, test management and customer helpdesk portal. Community edition free for up to 5 users

teamwork

tinyPM

VersionOne - very full-featured, supports Scrum, DSDM, XP and AUP across multiple projects. Priced from US$348 per user per year, there is a free "team edition" for one team of up to 10

XP Story Studio – no development since 2004

XPlanner

xProcess - free open source project management and process improvement tool, focused particularly on agile and priority-driven approaches. Preconfigured processes include Scrum, FDD, Prince2, Unified and others; can be tailored. Gantt/Burndowns/target status continuously updated. See also xProcess Europe

XPWeb

A number of web sites compare small subsets of the available software tools, including

http://www.agile-tools.net/

http://brunomiranda.com/past/2008/8/19/agile_project_management_tools/

http://agilesoftwaredevelopment.com/blog/mattgrommes

http://www.life-cycle.de/journal/

Latest unwarranted privacy violation

2009-03-17T10:40:00.002Z

The Daily Telegraph reports that the travel plans and personal details of everyone leaving Britain are to be tracked by the Government. Under the "e-borders" scheme, they intend to make it a criminal offence not to submit, in advance of every trip, personal information such as passport and credit card details, home and email addresses and exact travel plans. This is already starting to happen.

The kind of thing that e-borders enables is illustrated by this story, which shows that you could be stopped from leaving the country if you have unpaid parking fines or indeed any other mark against your name. What are the chances that before the year is out, we'll be reading stories about innocent people trying to travel abroad who get arrested at the airport or sea port thanks to some mix-up?

With developments like these, it's hard to believe that we are not living in a police state.

CodeCity

2009-03-17T10:33:00.003Z

This looks really intriguing: CodeCity gives a really nice way to visualise properties of Java code bases. An example is shown below (an analysis of Azureus v2.5.00):

It's a free download, so give it a try on your own code!

My Favourite Keyboard Shortcuts

2009-02-26T22:22:00.002Z

That was the title of a session I proposed for today's first-ever Software Craftsmanship Conference, at the BBC's attractive Media Centre in London's White City. To my surprise, the session was accepted. It worked quite well despite attracting only about half a dozen participants. I certainly learned a good deal from it!

The output of the session has been posted on the conference Wiki. However, you'll have to be logged in to see it, as the wiki is private to conference participants. I don't know if Jason Gorman plans to change this now the conference is over.

The top tips (as a result of a quick spot vote at the end) were:

The three-step refactoring tango: Jason's recipe for method extraction where one or more parameters are themselves method calls. In Eclipse, it goes ALT-SHIFT-L (extract local variable), ALT-SHIFT-M (extract method), then ALT-SHIFT-I (inline local variable as method parameter).

Expand/shrink current selection: In Eclipse, ALT-SHIFT-up/down arrow or in IDEA, CTRL-w/W. With the cursor on a given word, the word is selected. Repeated applications of the same keyboard combo selects progressively larger portions of the source file (up to and including the entire file). Using the opposite key combination reduces the size of the selection all the way back to 0.

Test-Driven Web Development on Spring and Hibernate

2009-02-18T09:57:00.002Z

Adam Shimali and I will be piloting our TDD course at SPA2009 on the Sunday afternoon. Three two-hour samples of a full two-day course for free - and you get a 10% discount off the cost of the full course for yourself or a colleague.

The course is very hands-on. If you bring a laptop (Windows or MacOS) we can provide you with the complete Eclipse development environment installation on a CD or memory stick - you'll be good to go in about five minutes. It will help if you've done a bit of Java programming before, but even this is not essential as you'll be paired with someone experienced.

There's still time to book for SPA2009, but don't delay!

Agile and Lean - complementary or conflicting?

2009-02-17T10:53:00.003Z

Dave West has contributed an article entitled A Marriage Made in Heaven?. I found it very instructive to read that as well as the comments attached to it. To my mind, there are things that the software community can learn (and has learned) from lean manufacturing, but in many respects software development is much more of a joint creative act. As Dave says, Peter Naur as long ago as 1985 equated programming with collaborative theory-building - in other words, it has much in common with research at the forefront of physics or mathematics, where results are difficult to predict and effort is almost impossible to forecast.

BALPA refuses to co-operate with ID card scheme

2009-02-16T23:46:00.002Z

Well done, the British Airline Pilots Association. It has refused to co-operate with the odious National ID register, as reported in The Times. I especially enjoyed reading BALPA's clear views on the insidious methods the Government has chosen to bring in the ID cards:

"It is clear that the Government's staged introduction of biometric identity cards first to overseas students, then to migrant workers and then for aviation workers, represents a way of picking off what are seen as easy targets."

It'll be interesting to see what the Home Office chooses to do next. They can't afford to be seen to back down in the face of such a challenge to their "authority", yet they can't force airlines to sack valuable staff. So I expect that a technical pretext will be found for not requiring the airline pilots to carry the cards after all, and the department will go after another softer target first.

It is beyond me why the Government keeps on trying to introduce this scheme. The Tories and LibDems have both pledged to scrap the whole thing as soon as Labour is voted out of office, which only gives them a couple more years at most to push it through. Sadly, the stubborn and unimaginative lot currently in power are likely to waste hundreds of millions of pounds of our money on this folly before it is finally sunk.

Predictably Irrational: the international financial crisis

2009-01-26T16:34:00.002Z

My eternal gratitude to Dan Ariely and his Predictably/Irrational blog for pointing out the existence of a radio programme and its transcript in PDF, which for the first time properly explained to me why we're in such a mess. I wonder what collective idiocy is next up.

It also provides a sobering reminder that it's usually a bad idea to throw money at a problem. In this particular instance, Gordon Brown's fantastic idea to "save the world" by hosing all our cash and then some into the banks' pool of money supply was such a bad idea that it must stand as one of the most monumentally stupid decisions of all time. Read the transcript yourself to find out why.

Anyone without the requisite quarter of an hour to hand can refer to this very much more succinct, but necessarily less thorough, tale. It seems prescient, having been posted in 2005!

How to write conference session proposals

2009-01-23T17:15:00.002Z

Most of you probably already know how to do this - after all, you get your papers accepted at conferences all over the world all the time, right?

If (like me) you find your brilliant ideas trampled underfoot as often as not, read Kent Beck's advice. It may have been written over 15 years ago, but I'm certain it applies equally well today. It's also charmingly self-referential.

Mock external components using AOP

2009-01-06T18:11:00.002Z

Mark Nadelson has provided a well-written tutorial for a novel (to me, at any rate) use of Aspect-Oriented Programming in Java. Read the "print" version to get the whole article in one lump without the commercials.

When you write jUnit tests, you often encounter a need to create mock objects (e.g. using JMock or EasyMock) that can simulate some external dependency of the software under test. Mark has extended this idea to any external component that has an interface. In order to determine whether the software under test uses a mock component or the real one, he uses dynamic aspect-weaving by means of Spring AOP with AspectJ annotations.

I've got to have a play with this approach to see whether it can be used flexibly enough to support TDD fully. One problem I could foresee is that the pointcut specification is not in the test script but in the mock object (i.e. the class that implements the aspect). So it might be less than straightforward to link the mock object to specific test cases/steps. By contrast, when you use JMock in a jUnit test, you specify both the mock object and its expectations in one place.

XPath made simple

2009-01-05T14:34:00.003Z

I know I'm a bit behind the curve on this one, but I've only just stumbled across a really useful tool:

XPather can

generate XPaths while browsing or inspecting HTML/XML/*ML documents
evaluate your XPaths and inspect the results
extract the content of *ML documents

The latest version 1.4.1 was released on June 18 2008 (Firefox 3.0 compatible).

XPather is a simple Firefox extension that integrates both with the browser and its DOMInspector. It is designed to be lightweight and cross-platform. It is valuable mainly as a web/XML-app development and debugging tool. In combination with the DOMInspector, which I can also highly recommend, it is particularly useful for developing WebTest scripts: right-click in any element of a page that you want your script to compare or set, and the option "Show in XPather" pops up a new window with the precise XPath to select that element.

Return of the Thick Client?

2008-12-16T09:22:00.003Z

Web 2.0 is taking all sorts of unexpected directions. The latest to come to my attention are two initiatives aimed at offloading processor load from server to client.

First up, Google announced a new developer project named Native Client. With Native Client, Google aims to "give web developers access to the full power of the client's CPU while maintaining the browser neutrality, OS portability and safety that people expect from web applications." As far as I can see, it's what Java Applets were originally designed to do - and look how successful they've been. At least outside the enterprise environment (where desktop configurations tend to be strictly controlled), applets suffered from the "write once, debug everywhere" syndrome and have largely fallen out of use. It'll be interesting to see whether an Internet-savvy company like Google makes a success of this new approach.

Next, a start-up named Good OS has announced Cloud, "a browser operating system". I am trying hard to get my head around that concept. They claim that it's an environment for enhancing the user's Internet experience and that it can be co-hosted on Windows, Linux or other operating systems. If I have read the scant information available correctly, the idea is that your machine will start up already logged on to your favourite portal (Yahoo, Google, Windows Live...) and that core applications such as Skype, which are accessed from a MacOS-like object dock, run within their own dedicated browser tab on top of Cloud's integrated compressed Linux kernel. Whenever you need to, you can switch to the native OS with a single key press.

Unfortunately the "Why Cloud?" page on Good OS's web site is unfinished, so it is difficult to comprehend the vision behind the product. A clue might be that it is being bundled with the Gigabyte touch-screen Netbook models and so looks like a lower-cost alternative to Windows CE. Perhaps the company is aiming for an "Internet Appliance" niche. It made a name for itself by supplying the gOS Linux operating system for the ultra-low-cost Wal-Mart PCs.

Testing web applications? Cucumber is the cool new kid on the block

2008-12-15T17:21:00.002Z

I've had quite complimentary things to say about Canoo WebTest, but one thing it doesn't provide out of the box is a representation of the tests that your average customer is able to read and understand intuitively - that is to say, a tabular format like Fit or a narrative of the "Given... when... then... " form.

Cucumber to the rescue! I heard several people mention this at the recent XP Day in London. I hope to investigate it in more depth soon.

I would be interested to know whether it can be used to test anything other than Ruby code and whether it can drive a web browser to interact with Rich Internet Applications (RIAs). Stuart Ervine's and Nat Pryce's GWT development demo at XP Day was truly impressive. Nat has combined WebDriver with WindowLicker to create a clean interface between a test script and synchronous or asynchronous RIAs. But telling a jUnit test to move the mouse cursor into a particular widget and click the button was nothing if not verbose!

Sending SMS texts from the PC desktop

2008-12-04T18:09:00.003Z

Ever left your mobile at home and needed to send a text to someone? And even to receive that person's reply without access to your phone? Or just found that ten-finger typing is quicker than predictive TXTing with two thumbs?

I've discovered Vodafone Text Centre does the job (see feature summary in the attached message). You can install it for free, but note that the cost of SMS messages you send using Text Centre goes on to your normal Vodafone mobile account.

There's a very comprehensive manual and online help. Make sure your Outlook isn't running before you start installing (use Task Manager to kill it if necessary).

In a test here in the office, I found that my first outgoing message was delivered in just a few seconds, whereas the reply took around 10 minutes to reach my Outlook inbox (you can choose to have the reply routed to your mobile instead, of course, which is quicker but means that you don't have access to Outlook's convenience features when forwarding or filing the reply).

If you're in the habit of leaving Outlook running when you leave the office, you can also get Text Centre to alert you by SMS when you receive an e-mail (or only for high-priority e-mails) or to send you an SMS alert of upcoming appointments.

Overall, I am impressed. With Vodafone Text Centre you can:

Send text messages to individuals and groups from your PC, using your existing contacts and distribution lists.
Choose to have replies sent to your mobile phone or to your e-mail inbox.
Receive appointment reminders via text messages.
Choose to receive text messages to notify you of urgent e-mails.

Vodafone says that Text Centre will be upgraded on a regular basis, so suggests that you visit the website from time to time to ensure you have the most up-to-date version of the software.

Munging PDF files and pages

2008-11-19T17:17:00.000Z

Having scanned over a dozen handwritten pages from my notebook into a PDF file, I had to invert every other page to make it possible to read the whole thing. It took me a while to find the answer, but YOU can go straight to the solution!

The command line is


pdftk.exe scannedfile.pdf cat 1 2S 3 4S 5 6S 7 8S 9 10S 11 12S output readablefile.pdf

Ramaze - first impressions continued

2008-11-10T17:09:00.003Z

My earlier posting was getting too long, so I'm continuing in this new one. Meanwhile, the tutorial's author has kindly left a comment on the original posting, indicating that the tutorial is due to be updated pretty soon. I look forward to that!

Chapter 11 of the tutorial can be used pretty much as-is, but I found that after an error, the redirection specified by the error method would be overwritten by the helper aspect. After some research, I discovered that the latest version of Ramaze supports a redirection status operator, redirected?. Using this, the helper aspect can be written very simply:

helper :aspect
after( :create, :delete, :open, :close ) {
  redirect Rs() unless redirected?
}

Chapter 12 of the tutorial needs no changes, except that you don't have to add the flash section to the page template because we did it earlier.

Now I wanted to convert the application to something more suited to enterprise deployment (and also more suited to shared hosting deployment - typically such environments provide an Apache server and MySQL database). So the first thing to do was to move from YAML to SQL - I chose MySQL.

If you haven't yet installed MySQL, do so before the next step. Take care to ensure that your PATH environment variable contains the MySQL binary as well as its libraries (hold down the Windows key and press PAUSE to bring up the system properties, then choose Environment Variables in the Advanced tab). For example, on my system the System PATH begins:

C:\ruby\bin;
C:\Program Files\MySQL\MySQL Server 5.0\bin;
C:\Program Files\MySQL\MySQL Server 5.0\lib\opt;
C:\Program Files\MySQL\MySQL Server 5.0\lib\debug;
...

Add a new database for the application and create a user account named "ramaze" with password "TodoList" for both localhost and remote-host access:

mysql -u root -p
******
CREATE DATABASE IF NOT EXISTS todolist_db;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE,
  DROP, RELOAD, PROCESS, FILE, REFERENCES, INDEX,
  ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
  LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW,
  CREATE ROUTINE, ALTER ROUTINE ON *.*
  TO 'ramaze'@'localhost'
  IDENTIFIED BY 'TodoList' WITH GRANT OPTION;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE,
  DROP, RELOAD, PROCESS, FILE, REFERENCES, INDEX,
  ALTER, SHOW DATABASES, CREATE TEMPORARY TABLES,
  LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW,
  CREATE ROUTINE, ALTER ROUTINE ON *.*
  TO 'ramaze'@'%'
  IDENTIFIED BY 'TodoList' WITH GRANT OPTION;
quit

I decided to use Sequel as my database access layer. You probably also have to install the gems mysql and sequel before the code will work - I am not sure about the mysql gem, as it was one of the things I installed before I eventually discovered the need to set up the PATH correctly.

gem install sequel
gem install mysql

I tried choosing the 'ruby' option, but this was unable to generate the native code on my machine. So I uninstalled it and tried again, choosing the 'mswin32' option, which worked fine.

Now take a safety copy of your model file, todolist.rb, and start modifying it to use MySQL instead of YAML. Replace the following lines:

require 'ramaze/store/default'
TodoList = Ramaze::Store::Default.new('todolist.yaml')

with the following lines:

require 'rubygems'
require 'sequel'

DB = Sequel.mysql('todolist_db',
  :user => 'ramaze',
  :password => 'TodoList',
  :host => 'localhost')

class TodoList < Sequel::Model(:tasks)
  set_schema do
    primary_key :id
    varchar :title, unique => true, :null => false
    boolean :done
  end
end

To begin with, I tried using the title as the primary key. But I soon found that not only was it necessary to define the title field first and then separately to name it as the primary key, the user-supplied title was not always suitable as a key value due to the presence of shell metacharacters and so on. So I decided to go with the Sequel flow and use a system-generated ID as the primary key.

Next I found that the model didn't support the method original(), which the original simple model supports. So I supplied one myself (subsequently found to be unnecessary after refactoring main.rb):

  # Copy all records into a list
  def self.original
    tasks = []
    self.dataset.each {|r| tasks.push [r[:title], {:done => r[:done]}]}
    return tasks
  end

More important was to add some initialisation code after the class was defined:

unless TodoList.table_exists?
  DB.transaction do
    puts "Creating table 'tasks'\n"
    TodoList.create_table
  end
end

Now try running the application. It seems to work, but nothing gets stored in the database. At this point we have to bite the bullet and refactor the main module to support a more relational view of the underlying data model.

In the index() method, we'll start using the ID of each task to identify it instead of the title. Instead of requesting an array of key-value pairs from the original() method, we get an array of task objects from the dataset() method. That of course implies that we have to extract the title and the ID from the task object and use them as appropriate:

  def index
    @title = ["To-Do List"]
    @tasks = []
    TodoList.dataset.each do |task|
      id = task[:id]
      title = task[:title]
      if task[:done]
        status = 'done'
        toggle = A('Open Task', :href => Rs(:open, id))
      else
        status = 'not done'
        toggle = A('Close Task', :href => Rs(:close, id))
      end
      delete = A('Delete', :href => Rs(:delete, id))
      @tasks << [title, status, toggle, delete]
    end
    @tasks.sort!
  end

The methods open(), close(), task_status() and delete() have to change because they all now take an id as parameter:

  def delete id
    unless TodoList.delete id
      failed "Cannot delete task no.: #{id}"
    end
  end

  def open id
    task_status id, false
  end
  
  def close id
    task_status id, true
  end

  def task_status id, status
    unless task = TodoList[id]
      failed "No such task no.: #{id}"
      redirect_referer
    end
    task[:done] = status
    TodoList[id] = task
  end

I decided to override the []= method of the model, so that tasks would actually be written to the database. It was an easy step from there to create a new row in the tasks table whenever the ID parameter to this method was absent or nil. So the create() method becomes:

  def create
    if title = request['title']
      title.strip!
      if title.empty?
        failed("Please enter a title")
        redirect '/new'
      end
      if TodoList.find(:title => title)
        failed("Task '#{title}' already exists")
      else
        TodoList[nil] = {:title => title, :done => false}
      end
    end
  end

Note the check for duplicates, which is easy to do now that we can look up titles in the dataset.

Turning to the file todolist.rb, here are the methods I had to add to allow tasks to be added and deleted in the database:

  def self.delete(id)
    puts "Attempting to delete '#{id}'\n"
    DB.transaction do
      if task = TodoList.find(:id => id)
        task.destroy()
      else
        puts "Not found\n"
        return false
      end
    end
  end

  # Assignment should update the underlying database
  def self.[]=(id, values)
    DB.transaction do
      if (id == nil || !(task = TodoList.find(:id => id)))
        task = TodoList.new
      end
      task.title = values[:title]
      task.done = values[:done]
      task.save
    end
  end

That's pretty much it. But before I stopped, I wanted to prettify the user interface a bit. I didn't like the fact that the column widths tended to change whenever the sole "not done" item was added to or deleted from the list, and in any case I preferred clickable icons to text links. So I designed some icons:

not done

done

delete

Feel free to copy the icon images. To let the Mongrel server access these, they have to be placed in the folder "public" of your project. Then I redesigned the index() method and the corresponding index.html file very slightly to use these (first declaring some constants):

  DELETE_ICON  = '<img src="delete_sml.gif">'
  NOTDONE_ICON = '<img src="notdone_sml.gif">'
  DONE_ICON    = '<img src="done_sml.gif">'
  
  # the index action is called automatically when no other action is specified
  def index
    @title = ["To-Do List"]
    @tasks = []
    TodoList.dataset.each do |task|
      id = task[:id]
      title = task[:title]
      if task[:done]
        toggle = A(DONE_ICON, :href => Rs(:open, id))
      else
        toggle = A(NOTDONE_ICON, :href => Rs(:close, id))
      end
      delete = A(DELETE_ICON, :href => Rs(:delete, id))
      @tasks << [title, toggle, delete]
    end
    @tasks.sort!
  end

    <p><a href="/new">New Task</a></p>
    <?r if @tasks.empty? ?>
      <p>No Tasks</p>
    <?r else ?>
      <table>
  <?r @tasks.each do |title, toggle, delete| ?>
    <tr>
      <td class="title" > #{title}  </td>
      <td class="toggle"> #{toggle} </td>
      <td class="delete"> #{delete} </td>
    </tr>
  <?r end ?>
      </table>
    <?r end ?>

The full source code is attached in the comments to this post.

Headlights

2008-11-08T16:12:00.000Z

While I'm on the subject of motoring, why is it that some people insist on driving along well-lit roads at night with headlights on full-power dipped beam or even high beam? It doesn't allow them to see any better, but significantly impairs the ability of oncoming drivers to see what's near their car. Many's the time when I have been uncertain of the amount of space between a dazzling pair of headlamps on the opposite side of the road and parked vehicles on my side, so that I have had to stop. Once I nearly knocked down a guy crossing the road behind a car coming towards me - he was wearing dark clothes and was simply hidden by the glare from the headlamps.

The only situation in which I will agree that the use of something stronger than sidelights is useful is when I'm approaching a corner. Traffic coming from another direction announces itself by the headlight beam cast ahead. Yet even this is sometimes misleading, as there are plenty of idiots who will park at the side of the road leaving their headlamps switched on. As we don't need this kind of advance warning during the daytime, why should it be required at night?

I always choose sidelights or dim-dip headlights when motoring in areas with good street lighting, out of consideration to other drivers - yet they frequently flash me, assuming that I have forgotten to switch on my lights properly. Other sins include the use of fog lights front and rear when there is no fog. This is particularly dangerous when the road is wet, because they are mounted low down and the reflections off the water can almost blind other road users.

Rules 113 and 114 of the Highway Code sum up the requirements of the law, which seem perfectly common-sense:

You MUST

ensure all sidelights and rear registration plate lights are lit between sunset and sunrise
use headlights at night, except on a road which has lit street lighting. These roads are generally restricted to a speed limit of 30 mph (48 km/h) unless otherwise specified
use headlights when visibility is seriously reduced (see Rule 226)

Night (the hours of darkness) is defined as the period between half an hour after sunset and half an hour before sunrise.

You MUST NOT

use any lights in a way which would dazzle or cause discomfort to other road users, including pedestrians, cyclists and horse riders
use front or rear fog lights unless visibility is seriously reduced. You MUST switch them off when visibility improves to avoid dazzling other road users (see Rule 226)

In stationary queues of traffic, drivers should apply the parking brake and, once the following traffic has stopped, take their foot off the footbrake to deactivate the vehicle brake lights. This will minimise glare to road users behind until the traffic moves again.

Driving and Parking in London

2008-11-08T15:49:00.000Z

If you want to keep your sanity, don't own a car in London. Occasionally however, you need one. For those occasions, I'm very much in favour of car-club schemes like CityCarClub, ZipCar and StreetCar, but none of them has a depot anywhere near my house yet. So I still own a car, but I don't use it very much. At certain times of day, I have found that it actually took me longer to drive somewhere than it would have taken to walk. And you don't have to pay a congestion charge for walking.

However, I'm currently on a project that involves flying to a customer site in Europe several times a month, leaving London City Airport very early in the morning - so early in fact that I would miss the flight if I tried to go by public transport. In the past, I have used a minicab instead, but fares have recently gone up so much that I resorted to driving to the airport myself. If you stay at least two days, you can book a space at the airport car park on the Internet and save a fair bit of money, but not for a single day - and the parking costs almost as much as the minicab.

To the rescue came a new service called ParkAtMyHouse. Such a simple yet elegant idea - anyone with a garage or a bit of parking space can advertise it on-line and you simply haggle about the price. The web site handles the reservation and communication between owner and renter of the space. There's a feedback mechanism similar to eBay. The service levies a fee of 10% of the rent paid for its services, which seems reasonable as you would not have any other way of discovering these cheap parking spaces. You can rent by the hour, day, week, month or longer. I didn't find anything very close to the airport, but only two stops away on the DLR was someone willing to rent me a space for only £5 a day - job done!

Ramaze - first impressions

2008-11-05T23:49:00.000Z

I've spent the past couple of days evaluating Ramaze. I started with the tutorial To-Do-List example and tried to move beyond that to something resembling a real web application backed with a proper database (MySQL). In general, I feel that this framework is more intuitive to use and therefore gets results faster than e.g. Ruby on Rails, if you're having to learn both from scratch. I also found it very easy to refactor the code, because it encourages a fairly aspect-oriented style and all changes are loaded immediately without having to re-start the server. Last but not least, the designers of Ramaze have considered enterprise-level deployment requirements from the very start, so your Ramaze application can run on anything from a shared virtual host rented for a few dollars per year to an enterprise server farm or on-demand compute-cloud configurations.

Ramaze also leverages the power of Ruby in very elegant ways. For example, here's a description of a basic content-management system based on Ramaze in just a single page of code.

I'm going to walk you through the to-do list tutorial, which dates back to 31st January 2008. Ramaze has evolved a fair bit since then and I have had to find out what the changes are to adapt the tutorial to the current version. Hopefully my observations will be incorporated in a future version of the tutorial. All of my work was done on Windows XP (blame my employer, not me :-) so the installation instructions etc. are related to that.

First of all, if you don't yet have Ruby and Ramaze installed, here's how you do it:

Grab the latest Ruby installer
Install with default options
Open a command shell and make sure that your Ruby bin directory is on the PATH
gem install ramaze

You may find yourself being asked to approve the installation of various dependencies. Just agree to each one. If gem offers you options (ruby, java, mswin32 etc.), choose "ruby" or "mswin32" (assuming you're on a Windows 32-bit machine).

My next big problem came with Eclipse. I wanted to use Eclipse as my IDE for Ruby and Ramaze. Trying to use the Help -> Software Updates feature to install Ruby support, I discovered by much trial and error that it was possible to do so, provided you configure the update site http://update.aptana.com/update/rails/3.2/ and select "Uncategorized -> Ruby Development Tools" and nothing else. Using this plug-in, you get a language-sensitive editor and the ability to launch Ruby applications, not much else, but it's a whole lot better than using just a text editor. The first time you try to open the Ruby perspective, the Hierarchy pane displays an error message ("Could not create the view: org/eclipse/search/internal/ui/text/ResourceTransferDragAdapter"). Just close it and use the Navigator pane instead. I'd be interested to know if there is a better Ruby plug-in for Eclipse.

While I was about it, I also installed the Web Page Editor from one of the Eclipse update sites. This came in useful when I was working on template pages later.

Then I started working my way through the tutorial. The first discrepancy came in chapter 3 - the View. Ramaze has evolved so that you supply just partial pages under the view folder. So the view/index.xhtml file in the tutorial should contain only the part of the page that will be supplied to the main page template in the variable @content. It looks like this:

  <ul>
   <?r
     TodoList.each do |title, value|
     status = value[:done] ? 'done' : 'not done'
   ?>
     <li>#{title}: #{status}</li>
   <?r end ?>
 </ul>

Of course, at this stage you won't have assigned a value to @title, so your page will be rendered without a page title or first heading.

Also on this page, ignore the instruction to "run ramaze" from the command line. That won't work under Windows. Instead, Eclipse comes to the rescue. Just right-click the start.rb file in the Navigator pane and select "Run As... -> Ruby Application". When you want to terminate the server, click the red stop button in the console pane. Subsequently you can re-run the server by selecting start.rb from the pull-down menu under Eclipse's green Run button.

In chapter 4 of the tutorial, again please omit the HTML scaffolding and just write

  <?r if @tasks.empty? ?>
   No Tasks
 <?r else ?>
   <ul>
     <?r @tasks.each do |title, status| ?>
       <li>#{title}: #{status}</li>
     <?r end ?>
   </ul>
 <?r end ?>

Now add the following line at the very top of the index method:

  @title = ["To-Do List"]

The standard page template inserts this value both in the page title and the top-level heading. You may have noticed that the MainController contains the line

  layout '/page'

This indicates that the file view/page.xhtml will be used as the template, not src/element/page.rb as described in the tutorial.

In chapter 5 of the tutorial, obviously you should not add the level-one heading as shown. Just put the link at the very top of the file view/index.xhtml.

Next, the new file view/new.xhtml is also much simplified:

  <a href="/">Back to TodoList</a>
 <form method="POST" action="create">
   Task: <input type="text" name="title" /><br />
   <input type="submit" />
 </form>

Before creating an action for create, as described in the tutorial, you will also need an action for new. This is the only way that I have found to generate a title for the new.xhtml page. Insert the following in main.rb:

def new
@title = ["Create a new To-Do List item"]
# See view/new.xhtml
end

Chapter 6 of the tutorial remains pretty much unchanged. Just remember not to delete the @title initialisation at the very top of the index method.

In chapter 7 of the tutorial, I found that I was able to add and delete tasks to my heart's content until I tried deleting a task such as "Lunch with Bob?". Ramaze has a very simple mapping scheme from URL path to action argument, which stops parsing the path_info as soon as it encounters a "?" or ";". I spent a considerable amount of time trying to think of ways to circumvent this and pass the offending characters as url-encoded escape sequences, but in the end it was all to no avail. The lesson is that you would be well advised to never use user-supplied values as keys into your data. Instead, treat the user-supplied string as a data field and generate your own system ID for each item. By the way, this lesson applies equally well in other frameworks. Following my completion of the tutorial I will show how I refactored the code to achieve greater robustness. For now, just avoid using metacharacters in your task titles!

The solution to the problem of stuck tasks was of course to edit the YAML database file to delete the offending characters.

Chapter 8 of the tutorial seems to me now to be more or less redundant. All I had to do was to modify the file view/page.xhtml slightly to incorporate the error message from the flash object. Here's the resulting body section:

   <h1>#{@title}</h1>
  <div id="error">
    <p>#{flash[:error]}</p>
  </div>
  <div id="content">
    #@content
  </div>

Note that there is no need for any logic to decide whether an error message is present. If it's there, it gets rendered as part of the page. Otherwise the renderer omits that element.

The prettification step in chapter 9 of the tutorial still applies as it stands, except that the stylesheet entries are inserted into the file view/page.xhtml instead of the Page element. I also added a style for the error message, where present. The resulting HTML header now contains the following entries:

   <style type="text/css">
    body      { margin:      2em; font-family:Verdana }
    #content  { margin-left: 2em;               }
    #error    { margin-left: 2em; color:red;    }
  </style>
  <style type="text/css">
    table     { width:       80%;               }
    tr        { background:  #efe; width:100%;  }
    tr:hover  { background:  #dfd;              }
    td.title  { font-weight: bold; width: 60%;  }
    td.status { margin:      1em;               }
    a         { color:       #3a3;              }
  </style>

I had real fun in chapter 10 of the tutorial. Firstly, I had to install the mongrel adapter:

gem install mongrel

For both mongrel itself and its fastthread dependency, I chose the "ruby" option.
When I tried to start Ramaze again, it complained that some other process was already listening on port 80. After some detective work with process explorer, I discovered that this was a little peer-to-peer application called kservice from Kontiki Inc. I had probably unwittingly installed it with some Internet media player. After following the uninstall instructions kindly blogged by "Geoff" I was finally able to start my server on port 80.

(to be continued)

SOA adoption reported to be accelerating

2008-10-31T11:34:00.000Z

I was interested to read this interview with Jason English, the VP of Corporate Marketing for iTKO. It shows that Service Oriented Architecture is coming of age when vendors are offering viable testing and virtualisation solutions.

It also previews the virtual conference SOA in Action, coming on November 19th, 2008, for which it looks as if it is free to sign up. "SOA in Action will feature keynotes by Gartner's Yefim Natis and Forrester's Randy Heffner, as well as a panel discussion led by ebizQ's Joe McKendrick and featuring Phil Wainewright of ZDNet and Dr. Chris Harding of The Open Group."

Java Native Access

2008-10-29T05:03:00.000Z

Every Java programmer knows about JNI, but what about JNA?

I'm pretty impressed with this. The documentation seems pretty comprehensive and the examples are well constructed. The software seems very stable. Obviously you can kill the VM if you dereference invalid pointers etc., but after a short period of familiarisation, I found that I could knock together a Java API for a subset of an existing C/C++ DLL (kernel32), using just the published API documentation to guide me, in very little time. I wrote a little test program that gets the compressed and uncompressed size of arbitrary files, and prints proper error messages for files it cannot open.

I make no claim that the example code is particularly elegant, but it shows what you can do. One great advantage is that you don't need to install a separate C/C++ development environment if a DLL already exists that gives you the needed native functionality.

Note that JNI is built for raw speed - some applications use it for example to perform graphics manipulations thousands of times a second. Using JNA, because it uses inspection and all kinds of clever tricks, you need to budget about ten times as much execution time per transition from Java to native machine code and back again. For many applications, that's irrelevant because native code is only called infrequently (up to a few hundred times a second).

Myths and Realities about Virtualisation

2008-10-14T19:04:00.000+01:00

I just came across an excellent article by Amrit Williams entitled Myths, Misconceptions, Half-Truths and Lies about Virtualization. Read it especially for the discussion thread that follows it.

New Test-Driven Development training course

2008-10-14T17:49:00.000+01:00

I have just completed work on a training course in test-driven development, intended for delivery over two days or self-study, which uses the development of a dating service as a worked example. Along the way, students will learn to use data access objects (DAO), Object-Relational Mapping (ORM) using Hibernate, Spring, jMock and Canoo WebTest. These tools will, I hope, provide them with good insights into the way that test-driven approaches alter the way in which software architectures grow, leading to more robust yet flexible designs.

My thanks to Adam Shimali, who developed the material initially. He and I aim to present a five-hour "taster" of this course on the Sunday afternoon at SPA2009.

Improving team dynamics

2008-10-14T17:40:00.000+01:00

Interested in making your team more effective? I found the following list of useful links just now after reconnecting to a former colleague on Plaxo Pulse.

And of course, my own company Zühlke Engineering offers tools, coaching and training to agile teams.

Importing free-format address lists into vCard (.vcf) and Excel

2008-08-05T18:15:00.000+01:00

A colleague had the following problem: he had a Word document containing information about many different companies he dealt with. They were more or less in a standardised form, with the name of the company on the first line of each "record" and the address on the second (or on the same line in some cases), but after that it was pretty haphazard - sometimes telephone numbers were prefixed with TEL:, sometimes not, sometimes the URLs were listed beneath a heading of "Websites:", sometimes not, and so on.

His desire was to export this information in a form that it could be treated much more like a proper database - spreadsheet format would be a start. Exporting the Word document to text was easy, but where to go from there? I started toying around with a Perl script to try to clean up the data, and found that there were some convenient modules around for capturing input to a vCard standard format.

The attached script uses the Text::vCard module (which you'll have to install manually from CPAN as it isn't available using PPM) with its Addressbook and Node packages to build up a whole address book from the input file. Although I have made no attempt at elegance, it has a good stab at parsing UK-style and US-style company names and addresses into their constituent fields, provided they are separated by commas. NB there are no contact names, as the original data file didn't have any - this is left as an exercise for the reader!

#!/usr/bin/perl -w
#
# convert a file to address records in VCF (vCard) format
# reads standard input and writes to standard output
#
if ( $#ARGV + 1 != 0 ) {
  print STDERR "usage: parse_addresses <inputfile.txt >outputfile.vcf\n";
  exit;
}

use Text::vCard::Addressbook;

my $addressbook = new Text::vCard::Addressbook;
# For testing / debugging: load a pre-existing address book
#my $addressbook =
#  Text::vCard::Addressbook->new(
#  { 'source_file' => 'C:/temp/Text-vCard-2.03/rfc2426.vcf', } );

while ( !eof STDIN ) {
  parseEntry($addressbook);
}

print $addressbook->export();

sub parseEntry {
  my ($addressbook) = @_;

  my $line = <STDIN>;
  chomp($line);

  # First line should contain organisation name and optionally its address
  my ( $org, $addr );
  if ( $line =~ m'^\s*([0-9A-Za-z][^:]*)\:?\s*$' ) {
    $org = $1;
    chomp( $line = <STDIN> );
    $addr = $line;
  }
  elsif ( $line =~ m'^\s*([0-9A-Za-z][^:]*)\:\s*(\S.*)$' ) {
    $org  = $1;
    $addr = $2;
  }
  else {
    # Not recognised
    print STDERR "Not recognised start of entry: $line\n";
    while ( ( defined $line ) && ( $line !~ '^\s*$' ) ) {
      chomp( $line = <STDIN> );
      print STDERR "Discarding: $line\n";
    }
    return undef;
  }

  # print STDERR "Organisation: $org -- Address: $addr\n";
  my $vCard = $addressbook->add_vcard();
  $vCard->version('3.0');
  $vCard->add_node({ 'node_type' => 'ORG' })->name($org);
  my $adr = $vCard->add_node( { 'node_type' => 'ADR' } );
  my @unsorted = ();
  $addr =~ s/\W*$//; # Remove trailing spaces and full-stops
  foreach my $adrField ( split( ',', $addr ) ) {
    if ( $adrField =~ m'^\s*([A-Z][A-Za-z ]+\s)?([-0-9]{4,11})\s*([A-Z][A-Za-z]+)?\s*$' ) {
      # US state and zip-code
      my $state = $1;
      chop( $state ) if defined $state;
      my $zip_code = $2;
      my $country = $3;
      if ( defined $state ) { $adr->region($state); }
      $adr->post_code($zip_code);
      if ( defined $country ) { $adr->country($country); }
    } elsif ( $adrField =~ m'^\s*([A-Z][A-Za-z ]+\s)?([A-Z]{1,2}[0-9O]{1,2}[A-Z]? [0-9O]{1,2}[A-Z]{2})\s*([A-Z][A-Za-z ]+)?\s*$' ) {
      # UK county and post-code
      my $county = $1;
      chop ($county) if defined $county;
      my $post_code = $2;
      my $country = $3;
      if ( defined $county ) { $adr->region($county); }
      $post_code =~ s/O([0-9][A-Z]?) ([0-9])/0$1 $2/;
      $post_code =~ s/O([A-Z])? ([0-9])/0$1 $2/;
      $post_code =~ s/ O([0-9])/ O$1/;
      $post_code =~ s/ ([0-9])O([A-Z]{2})/ $1O$2/;
      $adr->post_code($post_code);
      if ( defined $country ) { $adr->country($country); }
    } elsif ( $adrField =~ m'P.*BOX\s*(\d+)'i ) {
      # Post Office Box
      my $po_box = $1;
      $adr->po_box($po_box);
    } elsif ( $adrField =~ m'^\s*(\d+\s+[A-Za-z0-9 ]+$)' ) {
      # House number and street
      my $street = $1;
      $adr->street($street);
    } elsif ( $adrField =~ m'^\s*([A-Z][A-Za-z0-9 ]+$)' ) {
      push @unsorted, $1;
    }
  }

  # Retrieve unsorted items in reverse order
  if (!defined $adr->country() && $#unsorted > 2) {
    my $country = pop @unsorted;
    if (defined $country && $country =~ m'[A-Z][A-Za-z ]+') {
      $adr->country( $country );
    } else {
      push @unsorted, $country;
    }
  }
  if (!defined $adr->region() && $#unsorted > 1) {
    my $region = pop @unsorted;
    if (defined $region && $region =~ m'[A-Z][A-Za-z ]+') {
      $adr->region( $region );
    } else {
      push @unsorted, $region;
    }
  }
  if (!defined $adr->city()) {
    my $city = pop @unsorted;
    if (defined $city && $city =~ m'[A-Z][A-Za-z ]+') {
      $adr->city( $city );
    } else {
      push @unsorted, $city;
    }
  }
  if (!defined $adr->street()) {
    $adr->street( pop @unsorted );
  }
  if ($#unsorted >= 0) {
    $adr->extended ( join (', ', @unsorted) );
  }

  chomp( $line = <STDIN> );
  while ( ( defined $line ) && ( $line !~ '^\s*$' ) ) {
    if ( $line =~ m'^\s*(https?\://\S+)'i ) {
      $vCard->url($1);
    } elsif ( $line =~ m'(www\.\S+)'i ) {
      $vCard->url($1);
    } elsif ( $line =~ m'(\S+@\S+)\s*$' ) {
      my $email = $1;
      my $node  = $vCard->add_node( { 'node_type' => 'EMAIL' } );
      my @types = qw (work internet);
      $node->add_types( \@types );
      $node->value($email);
    } elsif ( $line =~ m'^\s*(\+?[0-9() ]+)$' ) {
      my $tel   = $1;
      my $node  = $vCard->add_node( { 'node_type' => 'TEL' } );
      my @types = qw (work voice);
      $node->add_types( \@types );
      $node->value($tel);
    } elsif ( $line =~ m'^\s*tel.*\:\s*(\S.*)$'i ) {
      my $tel   = $1;
      my $node  = $vCard->add_node( { 'node_type' => 'TEL' } );
      my @types = qw (work voice);
      $node->add_types( \@types );
      $node->value($tel);
    } elsif ( $line =~ m'^\s*fax.*\:\s*(\S.*)$'i ) {
      my $tel   = $1;
      my $node  = $vCard->add_node( { 'node_type' => 'TEL' } );
      my @types = qw (work fax);
      $node->add_types( \@types );
      $node->value($tel);
    } elsif ( $line =~ m'^[^:]*name[^:]*\:\s*(\S.*)$'i ) {
      $vCard->fn($1);
    } elsif ( $line =~ m'^\s*([^:]+)\:\s*$' ) {
      my $noteHead = $1;
      chomp( $line = <STDIN> );
      if ( $line =~ m'^\s*(https?\://\S+)'i ) {
        $vCard->url($1);
      } elsif ( $line =~ m'(www\.\S+)'i ) {
        $vCard->url($1);
      } elsif ( $line =~ m'(\S+@\S+)\s*$' ) {
        my $email = $1;
        my $node  = $vCard->add_node( { 'node_type' => 'EMAIL' } );
        my @types = qw (work internet);
        $node->add_types( \@types );
        $node->value($email);
      } elsif ( $line =~ m'^\s*(\+?[0-9() ]+)$' ) {
        my $tel   = $1;
        my $node  = $vCard->add_node( { 'node_type' => 'TEL' } );
        my @types = qw (work voice);
        $node->add_types( \@types );
        $node->value($tel);
      } else {
        $vCard->note("$noteHead: $line");
      }
    } elsif ( $line =~ m'^\s*([^:]+)\:\s*(\S.*)$' ) {
      my $noteHead = $1;
      my $note = $2;
      $vCard->note("$noteHead: $note");
    } else {
      print STDERR "Cannot understand entry detail: $line\n";
    }
    chomp( $line = <STDIN> );
  }
}

1;    # End.

(Look out for cut-off long lines in the above - copy/paste works far better in Firefox than in Internet Explorer. If you have problems, ask me to e-mail you a copy of the script). The script spits out the resulting vCard file to the standard output. Anything it can't parse is echoed to standard error.

I found a really nifty converter from VCF to CSV at http://labs.brotherli.ch/vcfconvert/ - once you've got a pile of addresses in spreadsheet format, you can do anything with it.

Dayfindr - a suitable case for viral marketing

2008-07-24T18:25:00.000+01:00

I frequently recommend my colleague Ben Nortier's demonstration application Dayfindr. This is a simple calendar co-ordinator, allowing an arbitrarily large group of people to find a date or dates when everyone will be available to undertake some kind of joint activity, such as a meeting, theatre visit etc. Although it doesn't deal with subtleties like specific times of day, the comment facility is there to let you fine-tune your entries.

Briefly, Ben put this together as an exercise in Erlang / OTP programming and it's getting an ever-increasing amount of use simply through word-of-mouth recommendation. So far it has easily coped with everything that's been thrown at it. Every time I show it to someone, it knocks them sideways with its straightforward simplicity, robustness and speed, even though it is hosted on a virtual server with very frugal memory and processor allocations. Give it a try and see what you think!

Software Practice - The State of the Art

2008-07-16T05:44:00.000+01:00

The British Computer Society's specialist group for Software Practice Advancement is staging a series of talks entitled The State of the Art. This features five talks from six of the leading experts in their respective fields, ranging from architecture to team dynamics.

It's excellent value at just £90 plus VAT - discounted to just £75 plus VAT for members of the specialist group or of the BCS generally. Numbers are limited, so book soon!

Maven multi-module builds and Eclipse PDE

2008-07-08T00:02:00.000+01:00

I have not blogged about Maven and Eclipse for a while, but I have learned a couple of important new lessons recently.

1. Even though OSGI gives you the capability of supporting multiple versions of the same bundle simultaneously, the same isn't true of Eclipse features that contain those bundles (a.k.a. Eclipse plug-ins). So I discovered that when feature A depends on feature B and C, where B depends on feature D version 1.1.0 and C depends on feature D version 1.1.2, you will get two versions of feature D in your Eclipse target platform, resulting in a cryptic error message from the PDE build of the form "unable to find feature: D". What it means is that B wants version 1.1.0 of D, but this has been masked by the presence of version 1.1.2. Our likely approach to this will be to avoid putting version dependencies (at least, versions of included features) into feature.xml files at all. There is no need, after all: the precise versions of everything needed should be specified in the accompanying POM, which assembles the RCP target platform for the build.

2. If you're using multi-module builds, there is no point in using goals such as "test" or "package". Unless you at least install the artifact created during the build of each module into your local Maven repository, the next module along will still get the previous version as a dependency, and the build will fail.

3. If you're using multi-module builds, don't run the install or deploy goal in the same invocation as the site or site-deploy goal. Maven tries to generate the site documentation for the top level before it has built the subordinate levels, so the dependencies required to create the reports are not present and the build will fail. We now routinely run "mvn clean install" or "mvn clean deploy" followed by "mvn site" or "mvn site-deploy" respectively.

Government security experts express concerns about ID Card scheme

2008-05-12T05:15:00.000+01:00

In today’s Observer, it is revealed that the Government’s own security experts are worried about Labour’s plans to introduce mandatory ID cards. Here are some choice quotes:

"In a potentially damaging revelation, which undermines claims that the scheme will enhance national security, the group has concluded that [ID cards] will be prone to corruption...

"The Isap report goes on to warn that the scheme may not be embraced by government departments, suggesting the cards are not being well received in some Whitehall departments.

"The panel also warns the initiative is struggling to fulfil its remit. It states that the scheme lacks a ‘robust and transparent operational data governance regime and clear data architecture’, suggesting there is confusion over its roll-out."

Eclipse and Maven - a quick update

2008-04-24T17:15:00.000+01:00

I have not posted anything about the progress of my researches into Eclipse and Maven integration for a while. That's because for a long time, we didn't get significantly further along in our research. But just recently we've started making significant progress again.

A very important lesson was that the pde-maven-plugin is really not worth persevering with - at least in our situation. We decided that we wanted to build every Eclipse plug-in and every Eclipse feature using its own POM, so that we could for example generate code coverage reports for each component individually and also configuration-manage them at that level of granularity. It turned out to be not all that difficult to invoke the PDE headless build for these circumstances from Maven using about two dozen lines of Ant script.

After nine months of Eclipse 3.3, the pde-maven-plugin has still not been officially updated to handle the changes from Eclipse 3.2 (we had to use the Alpha 2 snapshot version downloaded from Codehaus) and so we're concerned that we might have similar upward compatibility problems when Eclipse 3.4 comes out in a few months.

When it's all been finalised, I hope to publish a longer explanation of how our build system works here. Don't hold your breath though; at the moment we still have to solve a few issues - for example, we need to figure out how to produce the source plug-ins alongside the executable ones.

Having problems with string matching / replacing in Java?

2008-04-03T22:47:00.000+01:00

I find it almost impossible to write Java regular expressions correctly first time. They are pretty hard to get your head around (unless your head is a peculiar shape :-)

A trial-and-error approach is much eased by FileFormat's Regular Expression Test Page. At the bottom of the page you'll find a number of links to useful references and tutorials, which may help you eliminate some of the error from this approach.

The site is worth bookmarking anyway for its range of format conversion tools and information about file extensions.

Taking the pain out of Eclipse RCP builds and exports

2008-04-01T23:40:00.000+01:00

It is well worth reading the manual, as the old saying goes - but sometimes the manual can be misleading, especially if the software being documented has evolved since it was written.

Eclipse Help

If you're using Eclipse 3.3, the advice in the 2006 edition of The Book about enabling Help in your Eclipse RCP applications is no longer quite accurate. Section 13.2 ("Getting the Help Plug-ins") tells you to install lots of things, while my experimentation has shown that all you need are the following plug-ins in your target platform and any launch configuration, plus their dependencies:

org.eclipse.help.ui
org.eclipse.help.webapp

You will also find this information in the built-in help provided with Eclipse RCP Developer under Platform Plug-in Developer Guide -> User assistance support -> Help -> Configuration/setup -> Rich Client Platform (RCP) help. On discovering it, I felt a little like Arthur Dent finding the planning application for the bypass through his house (look for the telling phrase "Beware of the Leopard")! Interestingly, the on-line version of the same help page disagrees - listing a lot of superfluous plug-ins - which confused me for quite a while.

The dependencies, all of which you can copy from the plugins folder of your Eclipse installation to your target platform, are as follows:

org.eclipse.help.base
org.apache.jasper
org.apache.lucene
org.apache.lucene.analysis
org.eclipse.equinox.http.jetty
org.eclipse.equinox.http.registry
org.eclipse.equinox.http.servlet
org.eclipse.equinox.jsp.jasper
org.eclipse.equinox.jsp.jasper.registry
org.eclipse.osgi.services
javax.servlet
javax.servlet.jsp
org.mortbay.jetty
org.apache.commons.logging
org.apache.commons.el

Also, to use Eclipse Help successfully under Windows (and that includes your own help for exported / launched applications), you have to have your Internet Explorer preferences set up to use no proxy server when accessing localhost / 127.0.0.1. One way to do this in Internet Explorer's Tools -> Internet Options -> Connections -> LAN Settings is to turn off all three main check boxes. You could also try setting the "auto detect settings" checkbox or the manual proxy server configuration (provided you also check the "bypass proxy server for local addresses" checkbox).

Editing and Debugging

For some things, Eclipse needs source plug-ins to be present in the target platform location, not just in the installation location.I have not yet established all the circumstances under which this may be the case, but I have found that if you are trying to use the special-purpose editor to work on a plugin.xml file, unless the source plug-ins are present in the target platform location, when you right-click on an extension point in the Extensions tab you won't see the correct extension types in the context menu (just "Generic").

To support the most common types of UI extension, such as menu entries, copy the source plug-ins for org.eclipse.platform and org.eclipse.rcp from your Eclipse plugins folder into the target platform (hint: they are folders!). It is not even necessary to select them when specifying the target platform in Eclipse Preferences, but does no harm.

Source organisation

It turns out that the line of least resistance is to separate your plug-ins and features into two groups:

All the stuff you have access to in source form directly from its repository
Everything else - basically the Eclipse and third-party stuff

The second category forms your Eclipse RCP target platform, which should be a minimal Eclipse RCP installation plus all the features and plug-ins and their dependencies that you need in your application, plus source plug-ins where available. I created a target platform initially by downloading the RCP Runtime Platform plus Delta Pack (version 3.3.1.1, which matches the version of Eclipse RCP Developer I was using), but then deleted a lot of things that were not relevant to my intended deployment architecture before adding the plug-ins and features I really needed (see above).

For the first category, I recommend setting up your source folder structure to match the output in the build directory.

product root

features

a.b.c.d.feature
etc.

plugins

a.b.c.d
a.b.c.d.product
a.b.c.d.update-site
etc.

It helps if you append "feature", "product" or "update-site" as the last component of the folder name, as appropriate, where a folder contains something other than a plug-in. Once each of these has been turned into an Eclipse project in your workspace, they will simply appear as a linear alphabetically ordered list in the package explorer, so this helps you to navigate.

Ignore the features supplied with the Eclipse SDK, RCP Runtime and Delta Pack. Instead, define your own features to include just the plug-ins on which your application depends. It helps to factorise this into a half-dozen features that collect Eclipse plug-ins for various capabilities - e.g. the feature com.example.myproject.help.feature could be used to list all the plug-ins shown above that are needed to provide application help. Then if you want to include help with an application, all it has to do is list this feature in its application feature.xml as an included feature (of course you still have to write the help texts and create a help plug-in to contain them!).

Version numbering

In product.xml and feature.xml files, it is a good idea to set the version numbers of included plug-ins and features to 0.0.0 (or a range, where applicable) to ensure that the latest version is always used during the build - the exported feature.xml files will end up containing the resolved version numbers.

BCS survey underlines laxity of Government data protection

2008-03-15T16:12:00.000Z

As if more evidence were needed for the undesirability of a Government-run national identity register, Channel 4 News reported last night that a British Computer Society survey had demonstrated a total absence of data accuracy audits or data correction budgets in 14 out of 14 UK Government departments.

Other models for identity registration

2008-03-10T18:12:00.000Z

A story appeared on Slashdot today comparing the UK and US identity database schemes with the Jukinet system that's been quietly running in Japan since 1992. Some of the responses are quite insightful and informative. People describe systems that exist in Bulgaria, Japan, Sweden, Norway and other places, each of which have some things to recommend them.

For me, I think the main lesson is that yes, my personal data is already held in many different databases, both of government agencies such as the tax authorities and of private companies such as my credit card issuer. However, data protection legislation exists explicitly to prevent anyone, government or private, misusing this data by combining disparate databases to build a "profile" of me as an individual and to use that to my advantage or disadvantage. I fundamentally object to paying a huge amount of money so that this government can ride roughshod over those rights of the citizen. It isn't so much a question of privacy, more of protecting the individual against the might of the state.

I quite like the idea of an identity service as something that people can subscribe to if they wish - much like the trust providers used in a Public Key Infrastructure (PKI). There should be a free market in personal authentication, just as there is on the Internet. This would drive down prices and encourage the development of value-added services.

Latest ID Register scam

2008-03-07T17:20:00.000Z

In a desperate bid to ram through its controversial plans for a national identity register and the associated ID cards, the British government has announced yet another defenceless section of society to be targeted: students.

Not content with making university students pay over the odds for what are frequently sub-standard educational opportunities, the government now plans to "encourage" them to supply personal details including biometric data "voluntarily" in order to "help" them access educational services. Home Secretary Jacqui Smith claims that "young people who register for an ID card will find it easier to enrol on a course, apply for a student loan or open a bank account". The implication is that if they do not agree to register, they will find it hard or impossible to enrol on a course, get a student loan or open a bank account.

If that doesn't amount to coercion, I don't know what does. As a parent of two students currently at university, I feel very strongly about this. This is very reminiscent of the kind of vindictive tactics employed by the morally bankrupt East German government in the communist era. Moreover, the government expects to add me (an EU national resident and working in the UK) to its database automatically in a few years time "unless I opt out", which I certainly intend to do.

A series of high-profile security breaches recently proved that governments generally, and this one in particular, cannot be trusted to handle personal information securely or to refrain from using it as a means of coercion. The sausage-slicing approach it has adopted for introducing the scheme is evidence that it knows it faces a massive revolt if it were to apply the same rules to the whole population in one go. By bringing in the scheme in this sneaky, insinuating way, the government makes me more convinced than ever that its motivation for the identity register is anything but the publicly stated one of making us "confident that other people are who they say they are" - security experts have already shown that an ID card of the current design will be relatively easy to fake, so that pretext doesn't persuade me at all.

I don't normally subscribe to conspiracy theories but what else can you believe in the face of this government's announcements and actions?

More on the Maven POM for PDE headless builds

2008-03-06T19:23:00.000Z

It turns out that you can override almost any property in the build.properties file using an equivalent plugin configuration item within the buildProperties section.

Moreover, by observing certain conventions the job is made much easier. One of these is that the PDE build takes place within a "build directory" that contains subdirectories named "features" and "plugins", and that the product's project directory (where pom.xml resides) is one of the subdirectories of "plugins". Therefore, in the Maven POM, it would be logical to define buildDirectory as "../..". Here's a possible approach:

  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>pde-maven-plugin</artifactId>
        <version>1.0-alpha-2-SNAPSHOT</version>
        <extensions>true</extensions>
        <!-- Custom lifecycle configuration -->
        <configuration>
          <eclipseInstall>${env.M2_HOME}/../eclipse</eclipseInstall>
          <pdeProductFilename>prototyp.product</pdeProductFilename>
          <pdeBuildVersion>3.3.2.R331_v20071019</pdeBuildVersion>
          <buildProperties>
             <base>${maven.work.dir}</base>
             <baseLocation>${maven.work.dir}/eclipse</baseLocation>
             <buildDirectory>${basedir}/../..</buildDirectory>
          </buildProperties>
        </configuration>
        <!-- Also bind to mvn clean -->
        <executions>
          <execution>
            <id>clean-pde</id>
            <phase>clean</phase>
            <goals>
              <goal>clean</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

Please note: we use a convention that the Maven installation is alongside the Eclipse IDE installation, hence the definition of eclipseInstall above. We also define the maven.work.dir property in the settings.xml file - this is where all the stuff goes that Maven needs, e.g. the local repository and the Eclipse PDE target platform.

Currently I am working on a Maven plugin that will assemble the Eclipse PDE target platform automatically from project dependencies. I'll document this here once it's working.

Business Process Transaction Monitoring (BPTM)

2008-03-02T06:30:00.000Z

I went along to a meeting of the BCS Kingston & Croydon branch last Tuesday, at which a group of people from BT's Design group, who specialise in Systems and Application Monitoring and Management tools, revealed some astonishing achievements in a very low-key way, as if they had no idea how important they were.

These people have distilled their dozens of years of experience of managing increasingly complex distributed systems with few staff and fewer tools into a powerful yet spare vocabulary (or ontology, to use a fancy term) that efficiently describes the universe of discourse. It includes such concepts as server, virtual machine, date, time, business process, transaction, event-type and (very important) end-to-end correlation key, which precisely locates a reported event in a specific application component. All this, logically enough, is aligned with the ITIL standard for service delivery.

But not only that, they've defined binary, textual and graphical representations of log entries or event notifications that capture all this information, and a set of libraries that implement all of this and are accessed via a very simple API, which has been implemented by a standard code library (I understand that a Java implementation is available, but there may be support for other languages too). Not least, there is a defined process for integrating an application into the service monitoring and management framework.

Most applications already generate copious log information, and most commercial monitoring tools work by scanning the log files for interesting events. You have to configure patterns that allow the monitoring software to recognise different events. As a result, all large-scale monitoring infrastructures are permanently out of date with respect to the log formats and events generated by the applications, which are continually evolving. Moreover, the sheer volumes of log information generated mean that monitoring products that take this approach tend to be overwhelmed by the deluge of data and can find it difficult to react in a timely manner to real problem situations when they arise.

BT's BPTM takes a different approach: the application is "instrumented" by wrapping existing calls to the system logging facility, when it's much easier to identify the meaning of the logged information in terms of the underlying data model and to add any missing properties (such as system identifier, timestamp and e2e correlation key). As a result, team boss Ian Johnston claims that an average application can be instrumented in one day (preceded by a one-day workshop to identify the requirements of managing that application, and followed by another day to roll out and test the instrumented version of the code).

The BPTM library takes a "reporting by exception" approach to cut down on the amount of communication required. For example, events that are expected and that duly occur are merely logged locally by the application. This measure alone reduces the management data traffic by a factor of 20:1 on average. Then there are event correlation rules that can recognise typical failure scenarios and offer scripted diagnostic and remediation advice to support staff, many of whom are offshore.

By using this combination of approaches, the design group has already equipped over 80 separate applications in the "BT Matrix" or Service Oriented Architecture to be centrally monitored and managed. Newly instrumented applications are auto-discovered by the BPTM infrastructure - they simply hook themselves into the reporting network and pop up on the monitoring console (which is of course a rich Internet application).

Operators are alerted to emergency situations, such as service bottlenecks, via a variety of mechanisms. The primary user interface is a mimic diagram, which shows the flow of messages that make up an end-to-end business transaction through a series of components. The user can drill in to see both more detail and historical trend information, so that e.g. new server capacity can be brought on-stream before a bottleneck becomes critical.

It's obviously in BT's interest to publicise the BPTM standard so that more suppliers will start using it and building it into their products from the outset. But I don't think that Ian and his team are going about this in the right way yet. To build up momentum, it is not enough to hold occasional talks to BCS branches, where you reach at most 20 interested individuals at a time. You need to convince the solution architects and other decision makers that this is the right way to go. The first thing to do is to publish the standard, and simultaneously or not long afterwards, make the libraries that implement it Open Source. This should create a community of interest across the industry. After all, large service-oriented architectures are becoming increasingly common, in all market sectors, not just in telecoms, so the management headache is shared by all projects. Then some judiciously targeted white papers and articles should appear in the appropriate journals, and the trade press needs to be made aware.

If publicised in the right way, I can't see how this technology can fail to make waves.

A British company providing first-class products

2008-03-02T05:30:00.000Z

I've been very happy with my DNUK Linux server, which I purchased in September 2001 to provide filing services, web hosting etc. to the whole family. It has been doing duty in the cupboard under the stairs day in, day out without complaint. With the exception of some disk errors that cropped up soon after I bought the machine, and which DNUK sorted out for me very satisfactorily very quickly under the warranty, yesterday was the first time ever it went wrong.

The power supply failed, basically. I managed to find a repair man locally who was able to fit a replacement in about 30 minutes. The old one had some kind of loose bits rattling around in it - probably glass from a blown fuse. Now that a new PSU has been fitted, it's back to work as usual. I fully expect to get another four years or more of use out of it.

The repair man was very complimentary about the build quality of the machine and the tank-like solidity of the chassis. I too was impressed, but then I am not really a judge of these things. All I remember was the pleasure of dealing with a company that was small enough to treat its customers like real people, yet had all the snazzy web-based product selection and customisation capabilities you would expect of a major supplier. And it was excellent value for money, too.

What is "droplifting"?

2008-02-28T23:16:00.000Z

Read the highly entertaining story of how a late friend's unpublished novel got to be sold in a high-street chain of bookstores.

London Mayoral Candidates and the National Identity Register

2008-02-27T01:16:00.000Z

In late January or early February, I canvassed the candidates of major parties for the May 2008 election of the London Mayor, asking them to come out publicly with a statement that they would not co-operate with any national identity register (ID Cards) scheme if elected. I wrote:

I would like to know where the [your party here] candidate in the 2008 London Mayoral election stands regarding the national identity register. Will (s)he resist pressure to withhold services from people who do not have an identity card? The government intends to make the "voluntary" scheme effectively compulsory by making it impossible to get a new passport, driving licence etc. without an ID card. If London refuses to play the game, it will be doing its citizens a great service.

The Green party candidate, Siân Berry, responded with alacrity, saying that she had already signed the No2ID pledge in 2004 and was well known for her stand against the scheme (see http://camden.greenparty.org.uk/news/newsidcards.html and other sites). She goes on:

"I would not allow any services for Londoners to depend on the scheme, and would urge Londoners to resist the ID system individually as well.

You may also be interested in our Census Alert campaign to stop Lockheed Martin being given the contract to run the UK Census in 2011. This is a campaign I set up last year after finding out they were on the final shortlist, and there are now politicians from a wide range of parties supporting the cause. See www.censusalert.org.uk for more details."

So that's a result then.

The Liberal Democrat party's candidate, Brian Paddick, sent me a personal letter by snail mail, which arrived today, in which he states that he is totally opposed to the ID cards. From a former police officer, this comes as a nice surprise (except to those who have been following his campaign closely) and is in line with the national party's line as represented by Nick Clegg (see http://www.youtube.com/user/LibDem). Nick Clegg has publicly pledged to refuse to be registered, even if it means being taken to court.

Sadly, Brian Paddick did not go so far as to pledge that London would refuse to participate in any coercion by the Government to force people to join the scheme.

I didn't get any response from either the Conservative or (New?) Labour candidates or their representatives.

However, Conservative candidate Boris Johnson is on record as opposing the scheme - see http://www.youtube.com/watch?v=kZAAzSzleWk.

Only Labour candidate Ken Livingstone is really letting the side down. According to an Andrew Marr interview mentioned on the NO2ID site, he actually supports the ID card scheme - see http://boardreader.com/t/Articles_Publications_150279/Andrew_Marr_Show_Ken_Livingstone_Intervi_19360.html.

I hope this helps you when it comes to making the decision in May!

Further background information about Maven PDE builds

2008-02-22T18:43:00.000Z

I was asked in a comment to publish the settings.xml file for my Maven installation, because the correspondent was not able to build the example project successfully.

Our project makes use of a local Maven proxy server to mirror the combined contents of the Maven Central repository, various snapshot repositories and the project's own built artifacts. In the Maven conf directory, we have therefore installed the following settings.xml file, which reflects this arrangement:

<settings>
  <servers>
    <server>
      <id>project-repo</id>
      <username>...</username>
      <password>...</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>main-profile</id>
      <properties>
        <!-- settings.localRepository is set by the USER settings.xml -->
        <!-- Maven working directory -->
        <maven.work.dir>${settings.localRepository}/..</maven.work.dir>
        <!-- Clover working directory: clover license is expected in ${clover.work.dir}/license -->
        <clover.work.dir>${maven.work.dir}/clover-work</clover.work.dir>
        <!-- Additional plugin provider path for Eclipse target platform -->
        <plugin.target.dir>${maven.work.dir}/plugin_target_dir</plugin.target.dir>
        <!-- project sites will be deployed to ${site.dir} with mvn site:deploy -->
        <site.dir>${maven.work.dir}/maven_site</site.dir>
      </properties>
      <repositories>
        <repository>
          <id>central</id>
          <name>Internal Mirror of Central Repositories</name>
          <url>http://....:9999/repository/</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <name>Internal Mirror of Central Plugins Repository</name>
          <url>http://....:9999/repository/</url>
        </pluginRepository>
      </pluginRepositories>  
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>main-profile</activeProfile> 
  </activeProfiles>
</settings>

In the .m2 directory under the user's home directory, we install a minimal settings.xml, which simply indicates where to find the local Maven repo on your hard disk. However, I have customised my copy so that I can do builds when not connected to the company network but with access to the Internet. This shows you where you can obtain things like the PDE Maven plugin snapshot.

<settings>
  <localRepository>C:\data\...\maven_repo</localRepository>
  <profiles>
    <profile>
      <id>publicRepos</id>
      <activation>
        <property>
     <name>searchPublicRepos</name>
     <value>true</value>
   </property>
      </activation>
      <repositories>
        <repository>
          <id>codehausSnapshots</id>
          <name>Codehaus Snapshots</name>
          <releases>
            <enabled>false</enabled>
            <updatePolicy>always</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <url>http://snapshots.maven.codehaus.org/maven2</url>
          <layout>default</layout>
        </repository>
        <repository>
          <id>mavenCentral</id>
          <name>Maven Central</name>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>false</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </snapshots>
          <url>http://repo1.maven.org/maven2/</url>
          <layout>default</layout>
   </repository>
        <repository>
          <id>mavenSnapshots</id>
          <name>Maven Snapshots</name>
          <releases>
            <enabled>false</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </snapshots>
          <url>http://people.apache.org/repo/m2-snapshot-repository/</url>
          <layout>default</layout>
   </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>codehausSnapshots</id>
          <name>Codehaus Snapshots</name>
          <releases>
            <enabled>false</enabled>
            <updatePolicy>always</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>fail</checksumPolicy>
          </snapshots>
          <url>http://snapshots.maven.codehaus.org/maven2</url>
          <layout>default</layout>
        </pluginRepository>
        <pluginRepository>
          <id>mavenCentral</id>
          <name>Maven Central</name>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>false</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </snapshots>
          <url>http://repo1.maven.org/maven2/</url>
          <layout>default</layout>
   </pluginRepository>
        <pluginRepository>
          <id>mavenSnapshots</id>
          <name>Maven Snapshots</name>
          <releases>
            <enabled>false</enabled>
            <updatePolicy>never</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>daily</updatePolicy>
            <checksumPolicy>warn</checksumPolicy>
          </snapshots>
          <url>http://people.apache.org/repo/m2-snapshot-repository/</url>
          <layout>default</layout>
   </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
</settings>

As I mentioned in passing in my original post, you also have to make sure that you (a) correct the version number of two dependencies in the pde-maven-plugin's POM, and (b) upload all the Eclipse plugins that the pde-maven-plugin needs to your local repository or to your project repository, using the eclipse:to-maven goal.

I have actually set up a Maven project to do upload Eclipse plugins repeatably and reliably in case we need to clean up and rebuild the project repository (e.g. after an upgrade of Eclipse). Here's the POM:

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>....</groupId>
  <artifactId>devclipse-to-maven</artifactId>
  <version>1.0-SNAPSHOT</version>
  <packaging>pom</packaging>
  <name>Eclipse plugins to Maven Artifacts converter</name>
  <url>....</url>
  <description>....</description>

  <!-- ******************************************************************
    Inherit from project grandparent pom
    ****************************************************************** -->
  <parent>
    <groupId>....</groupId>
    <artifactId>configuration</artifactId>
    <version>1.8</version>
  </parent>

  <!-- ******************************************************************
    Environment Information
    ****************************************************************** -->
  <scm>
    <connection>
      scm:svn:http://..../devclipse_to_maven/
    </connection>
    <developerConnection>
      scm:svn:http://..../devclipse_to_maven/
    </developerConnection>
    <tag>HEAD</tag>
    <url>
      http://..../devclipse_to_maven/
    </url>
  </scm>

  <!-- ******************************************************************
    Build configuration
    ****************************************************************** -->
  <properties>
    <eclipse.source.dir>${env.M2_HOME}/../eclipse</eclipse.source.dir>
    <eclipse.ext.dir>${env.M2_HOME}/../ext/eclipse</eclipse.ext.dir>
    <eclipse.usr.dir>${env.M2_HOME}/../usr/eclipse</eclipse.usr.dir>
    <eclipse.target.dir>${project.build.directory}/eclipse</eclipse.target.dir>
  </properties>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-antrun-plugin</artifactId>
        <version>1.1</version>
        <executions>
          <execution>
            <id>copy-plugins-and-features</id>
            <phase>process-resources</phase>
            <goals>
              <goal>run</goal>
            </goals>
            <configuration>
              <tasks description="Grab the features and plugins from
               the Eclipse SDK that have been found empirically to
               be required as part of the target platform. We will
               upload them to the Maven repository in the next phase.">
                <mkdir dir="${eclipse.target.dir}/plugins"/>
                <copy todir="${eclipse.target.dir}/plugins">
                  <fileset dir="${eclipse.source.dir}/plugins">
                    <include name="com.ibm.icu_*.jar"/>
                    <include name="org.eclipse.ant.core_*.jar"/>
                    <include name="org.eclipse.compare_*.jar"/>
                    <include name="org.eclipse.core.commands_*.jar"/>
                    <include name="org.eclipse.core.contenttype_*.jar"/>
                    <include name="org.eclipse.core.databinding_*.jar"/>
                    <include name="org.eclipse.core.expressions_*.jar"/>
                    <include name="org.eclipse.core.filebuffers_*.jar"/>
                    <include name="org.eclipse.core.filesystem_*.jar"/>
                    <include name="org.eclipse.core.jobs_*.jar"/>
                    <include name="org.eclipse.core.net_*.jar"/>
                    <include name="org.eclipse.core.resources_*.jar"/>
                    <include name="org.eclipse.core.runtime.compatibility.auth_*.jar"/>
                    <include name="org.eclipse.core.runtime_*.jar"/>
                    <include name="org.eclipse.core.variables_*.jar"/>
                    <include name="org.eclipse.debug.core_*.jar"/>
                    <include name="org.eclipse.equinox.app_*.jar"/>
                    <include name="org.eclipse.equinox.common_*.jar"/>
                    <include name="org.eclipse.equinox.preferences_*.jar"/>
                    <include name="org.eclipse.equinox.registry_*.jar"/>
                    <include name="org.eclipse.help_*.jar"/>
                    <include name="org.eclipse.jdt.core_*.jar"/>
                    <include name="org.eclipse.jdt.debug_*/**/*"/>
                    <include name="org.eclipse.jdt.launching_*.jar"/>
                    <include name="org.eclipse.jface.databinding_*.jar"/>
                    <include name="org.eclipse.jface.text_*.jar"/>
                    <include name="org.eclipse.jface_*.jar"/>
                    <include name="org.eclipse.osgi_*.jar"/>
                    <include name="org.eclipse.rcp.source.win32.win32.x86_*/**/*"/>
                    <include name="org.eclipse.swt.win32.win32.x86_*.jar"/>
                    <include name="org.eclipse.swt_*.jar"/>
                    <include name="org.eclipse.team.core_*.jar"/>
                    <include name="org.eclipse.team.ui_*.jar"/>
                    <include name="org.eclipse.text_*.jar"/>
                    <include name="org.eclipse.ui_*.jar"/>
                    <include name="org.eclipse.ui.console_*.jar"/>
                    <include name="org.eclipse.ui.editors_*.jar"/>
                    <include name="org.eclipse.ui.forms_*.jar"/>
                    <include name="org.eclipse.ui.ide_*.jar"/>
                    <include name="org.eclipse.ui.navigator_*.jar"/>
                    <include name="org.eclipse.ui.navigator.resources_*.jar"/>
                    <include name="org.eclipse.ui.views_*.jar"/>
                    <include name="org.eclipse.ui.views.properties.tabbed_*.jar"/>
                    <include name="org.eclipse.ui.workbench_*.jar"/>
                    <include name="org.eclipse.ui.workbench.texteditor_*.jar"/>
                    <include name="org.eclipse.update.configurator_*.jar"/>
                    <include name="org.eclipse.update.core_*.jar"/>
                    <include name="org.eclipse.update.ui_*.jar"/>
                  </fileset>
                </copy>
                <mkdir dir="${eclipse.target.dir}/features"/>
                <copy todir="${eclipse.target.dir}/features">
                  <fileset dir="${eclipse.source.dir}/features">
                    <include name="none_*"/>
                  </fileset>
                </copy>
                <copy todir="${eclipse.target.dir}/plugins"
                  failonerror="false">
                  <fileset dir="${eclipse.ext.dir}/plugins">
                    <include name="org.polarion.team.svn.client.javahl.win32_*/**/*"/>
                    <include name="org.polarion.team.svn*.jar"/>
                    <include name="org.sf.easyexplore_*.jar"/>
                  </fileset>
                </copy>
                <copy todir="${eclipse.target.dir}/features"
                  failonerror="false">
                  <fileset dir="${eclipse.ext.dir}/features">
                    <include name="none_*"/>
                  </fileset>
                </copy>
                <copy todir="${eclipse.target.dir}/plugins"
                  failonerror="false">
                  <fileset dir="${eclipse.usr.dir}/plugins">
                    <include name="none_*"/>
                  </fileset>
                </copy>
                <copy todir="${eclipse.usr.dir}/features"
                  failonerror="false">
                  <fileset dir="${eclipse.ext.dir}/features">
                    <include name="none_*"/>
                  </fileset>
                </copy>
              </tasks>
            </configuration>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-eclipse-plugin</artifactId>
        <version>2.4</version>
        <executions>
          <execution>
            <id>installation</id>
            <phase>install</phase>
            <configuration>
              <eclipseDir>${eclipse.target.dir}</eclipseDir>
            </configuration>
            <goals>
              <goal>to-maven</goal>
            </goals>
          </execution>
          <execution>
            <id>deployment</id>
            <phase>deploy</phase>
            <configuration>
              <eclipseDir>${eclipse.target.dir}</eclipseDir>
              <deployTo>${distributionManagement.repository.id}::default::${distributionManagement.repository.url}</deployTo>
            </configuration>
            <goals>
              <goal>to-maven</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

  <!-- ******************************************************************
    Project Information
    ****************************************************************** -->
  <organization>
    <name>....</name>
    <url>http://..../</url>
  </organization>

  <developers>
    <developer>
      <id>ihu</id>
      <name>Immo H&uuml;neke</name>
      <email>ihu@zuhlke.com</email>
      <url>http://aspsp.blogspot.com/</url>
      <organization>Z&uuml;hlke Engineering</organization>
      <organizationUrl>http://www.zuhlke.com/</organizationUrl>
      <roles>
        <role>architect</role>
        <role>developer</role>
      </roles>
      <timezone>0</timezone>
      <properties>
        <picUrl>http://www.spaconference.org/cgi-bin/wiki.pl/?mugimmohuneke.jpg</picUrl>
      </properties>
    </developer>
  </developers>

  <inceptionYear>2008</inceptionYear>

</project>

Hope this helps. If you still have problems, please e-mail me your Maven output - I may be able to guess where it is going wrong.

A Maven Eclipse RCP Product build at last!

2008-02-16T05:02:00.000Z

My sincere thanks to René Gröschke and Patrick Paulin for putting me on the right track. These notes were developed using Eclipse RCP Developer (version 3.3.1.1 "Europa").

Below, I have documented the required steps in some detail, based largely on René's tutorial. Some gotchas to watch out for include:

Contrary to popular belief, it is not necessary to install the Maven 2 Integration feature in Eclipse - in fact we found it confusing and removed it again, preferring to invoke various Maven goals through External Tool configurations under the Run menu
The build.properties file is very literal in its interpretation of properties you specify. In particular, when specifying a pathname, you must use forward slashes even under Windows, you must not have spaces in the pathname and you cannot enclose it in quotation marks. And most particularly, you cannot use symbolic references to any other property or environment variable (although the template supplied with the Eclipse PDE build plug-in misleads you here by suggesting that you can use ${user.home} as part of the buildDirectory property) - otherwise you will see the dreaded error message: Content Provider not set for Site: "{0}".
You must specify a target platform in the base location that has had the Eclipse RCP Delta Pack installed, otherwise you may be able to build using PDE but you won't generate an executable.

It is recommended to download both the RCP Runtime for your expected target platform architecture(s) and the RCP Delta Pack. Install each target platform RCP runtime into its own location - e.g. if you have installed Eclipse in C:\tools\eclipse, I would suggest unzipping the RCP Runtime for 32-bit Windows on Intel x86 processors to C:\tools\eclipse\win32.win32.x86\eclipse. Unzip the delta pack into each of the targets. Let it overwrite all clashing filenames. Afterwards, if you are short of space, you can go through each Eclipse target platform and safely delete any plugins or features that don't correspond to the selected target architecture.

Having installed the target platform(s), it is probably a good idea to set the default Eclipse workspace target platform to the one for the type of workstation you are running on - e.g. if you're using a 32-bit Windows / Intel x86 computer, select Window -> Preferences -> Plugin Development -> Target Platform and browse to C:\tools\eclipse\win32.win32.x86\eclipse (vary this path depending on the location of your Eclipse target platform). This ensures that native Eclipse builds and product exports will correspond as closely as possible to those built by the PDE in "headless" mode.

Another thing you need to do is to doctor the POM of the pde-maven-plugin (snapshot) so that it doesn't depend on outdated versions of org.eclipse.pde.core and org.apache.maven.shared.maven-osgi. Here is one way to do it:

    <dependency>
      <groupId>org.eclipse.pde</groupId>
      <artifactId>core</artifactId>
      <version>[3.3.0,4.0.0)</version>
    </dependency>
    <dependency>
      <groupId>org.apache.maven.shared</groupId>
      <artifactId>maven-osgi</artifactId>
      <version>[0.1.9,)</version>
    </dependency>

Make sure you catch all copies of the POM. You can do this in your local Maven repository after a first attempt to build, or more permanently on your project repository. You can use the eclipse:to-maven goal to get Eclipse plugins from the target platform folder into your local and/or project repository.

Let's work through a complete though simple example of creating an Eclipse RCP application that can be built from the command line using Maven. Broadly, this has the following stages:

Create the RCP application
Create a product configuration file
Create a build.properties file
Make a Maven POM
Run it and test the result

Steps A-C are identical to what you would do if you wanted to create a project capable of supporting headless PDE builds. The only concessions to Maven are the layout of the source and output directories and the fact that we do not specify an eclipseLocation property in the build.properties file - this is instead configured in the Maven POM as the eclipseInstall parameter.

Now, are you sitting comfortably in front of your Eclipse IDE? Then follow the bouncing ball...

A. Create the RCP application

Start a new project: File -> New -> Project -> Plug-In Project
Set project name = "com.breskeby.tutorial.rcp.withMaven"
Uncheck "Default location"
Specify a location outside the Eclipse workspace, e.g. C:\data\eclipseProjects\plugins\com.breskeby. tutorial.rcp.withMaven
Set Source folder = src/main/java (following the Maven convention)
Set Output folder = target/classes
Next
Check "Generate an Activator" and "This plugin will contribute to the UI"
Check "Create an RCP Application"
Next
Select the template "Hello RCP"
Next
Change the application window title to "Hello RCP with Maven"
Finish
If asked, decide whether to open the Plugin Developer perspective (it probably is not worth it at this stage)
Test the new plugin by clicking "Launch an Eclipse Application"

B. Create a product configuration file

File -> New -> Other -> Product Configuration
Parent folder = com.breskeby.tutorial.rcp.withMaven
Filename = withmaven.product
Check "Create a configuration file"
Finish - the product overview is displayed
Set Product Name = "RCP with Maven Example"
Product ID: click New...
Accept default values except Product Application
Set Application = "com.breskeby.tutorial.rcp.withMaven"
Finish
Check "Product Configuration based on plug-ins"
Click on the Configuration tab and click "Add..."
Select com.breskeby.tutorial.rcp.withMaven and click OK
Click "Add required plug-ins"
Save the product configuration
Test by right-clicking on the withmaven.product file in the package explorer and selecting "run as..." -> "Eclipse Application"

C. Create a build-properties file

Right-click the project folder com.breskeby.tutorial.rcp.withMaven and select New -> Folder: name the folder "buildConfiguration"
In Windows Explorer or equivalent, find your Eclipse installation and navigate down to eclipse/plugins/org.eclipse.pde.build_*/template/headless-build. Copy build.properties from this file to the clipboard. Paste it into your new buildConfiguration folder in Eclipse. The asterisk in the above path stands for the version number of the Eclipse PDE builder. Make a note of it, as you'll need it in step D
Open the build.properties copy in an Eclipse text editor and click on the build.properties tab. Change the following properties:
1. product=withmaven.product
2. archivePrefix=withMaven
3. configs=win32, win32, x86 or change depending on your desired target(s) - but René has noted that only one target architecture can be built at a time
4. buildDirectory=[project directory]/target/eclipse.build (substitute the pathname of your project directory - see step A4 above - but use forward slashes)
5. buildId=withMaven
6. base=C:/tools/eclipse/win32.win32.x86 (depends on where you installed the target platform and which configuration you specified in configs - item c above). The base directory must contain a folder called "eclipse", which contains folders named "features" and "plugins"
7. pluginPath=.
The plugin path is the list of places to be searched for features and plugins needed during the build. In this case, the current working directory (i.e. the project directory) is fine, since the product build first creates the plugin in a plugins subdirectory.

D. Make a Maven Project Object Model (POM)

Right-click the project folder com.breskeby.tutorial.rcp.withMaven and select New -> File: name the file "pom.xml"

Into the new file, copy the template XML below

<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
  http://maven.apache.org/maven-v4_0_0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>com.breskeby.tutorials.rcp</groupId>
  <artifactId>withMaven</artifactId>
  <packaging>zip</packaging>
  <name>RCP With Maven Example</name>
  <version>1.0-SNAPSHOT</version>
  <description>A Simple Product PDE Example</description>
  <build>
    <plugins>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>pde-maven-plugin</artifactId>
        <version>1.0-alpha-2-SNAPSHOT</version>
        <extensions>true</extensions>
        <configuration>
          <eclipseInstall>${env.M2_HOME}/../eclipse</eclipseInstall>
          <pdeProductFilename>withmaven.product</pdeProductFilename>
          <pdeBuildVersion>3.3.2.R331_v20071019</pdeBuildVersion>
        </configuration>
      </plugin>
    </plugins>
  </build>
</project>

Modify the configuration section as follows:
1. eclipseInstall - this is the path to your Eclipse RCP Developer installation (e.g. C:\tools\eclipse in the above example). Because my current project follows a convention about installing all tools in one place, I was able to use a property for this as shown above - your mileage may vary
2. pdeBuildVersion - insert the version number copied from the pathname of the folder whose name begins "org.eclipse.pde.build_" (see step C2 above)

E. Run the build and test the result

Make sure all edited files have been saved
Ensure that your settings.xml is configured such that the required Maven snapshot repository is searched for plugins
Select the pom.xml file in the package explorer
Set up and use an external tool configuration to invoke Maven or run it from a command-line in the project directory: mvn clean install

I hope that you have found this useful. If you run into snags, please check that you have followed all the above steps to the letter before contacting me (if you leave a comment on this blog I should be notified by e-mail).

The art of designing APIs

2008-02-12T17:40:00.000Z

In the restaurant last night, I was discussing with two colleagues why open-source software efforts such as Maven sometimes go awry. One reason is that APIs get changed, which really hacks off some of their users. I mentioned that I had seen Joshua Bloch give a really useful talk about the design of APIs about four years ago. If his guidelines were followed more widely, perhaps it wouldn't be necessary for APIs or their semantics to change so often.

You can still get hold of Joshua's presentation slides and my notes of his talk at the SPA Conference web site. Use the local link [u1] to access the slides.

Joshua Bloch contributes to the Official Google Research Blog.

Maven dependency version ranges are broken

2008-02-04T23:15:00.000Z

I've been tearing my hair out trying to get Maven to build Eclipse RCP products. A lot of the problems I have had, as it turns out, are to do with Maven POMs that express dependencies on Eclipse artifacts with a range of versions.

Because you can't always update all your Eclipse plugins at once, it might seem to make sense to express dependencies on them not as a fixed version number but as a range. The following is a typical example (/org/eclipse/jdt/launching/3.3.0-v20070510/launching-3.3.0-v20070510.pom):

<project>
  ...
  <groupid>org.eclipse.jdt</groupid>
  <artifactid>launching</artifactid>
  <name>Java Development Tools Launching Support</name>
  <version>3.3.0-v20070510</version>
  ...
  <dependencies>
    <dependency>
      <groupid>org.eclipse.core</groupid>
      <artifactid>resources</artifactid>
      <version>[3.3.0,4.0.0)</version>
    </dependency>
    ...
  </dependencies>
  ...
</project>

Unfortunately this only works where the version number of the actual artifact in the repository conforms to one of two patterns:

A.B.C-D

A.B.C-qualifier

(it is unclear whether the qualifier itself is subject to any formatting rules). In all other cases, Maven compares the artifact version number to the upper and lower bound of the range as a simple string, which actually means that version 3.3.0-v20070604 sorts below 3.3.0 and thus does not match the expression in the above example.

I may be able to fix this for individual POMs within our project repository and get the example build running, but that is not a general-purpose solution. Before long, someone on our project will bring in a dependency on some Eclipse plugin, which is obtainable from the Maven repository at Maven Central but only with a POM that contains a version range in one of its transitive dependencies.

Unfortunately, the majority of Eclipse plugins are likely to be in the Maven repository with this kind of version range dependency. The reason is that they get put there using the maven-eclipse-plugin (to-maven goal), which does not resolve OSGi Require-bundle version ranges when converting them to Maven dependencies. The latest version is meant to honour a property "resolveVersionRanges" (default is false), but I have not been able to get this to work.

And here's why - Carlos has removed the line of code that invoked the resolveVersionRanges routine.

Building Eclipse RCP applications using Maven

2008-01-31T01:48:00.000Z

Still experiencing some problems following the Codehaus tutorial. Downloaded the examples and unpacked them without problems into the correct directory structure in my workspace, but then found that Eclipse doesn't recognise the three examples as Java projects even after running mvn eclipse:eclipse on them. Moreover, trying to build the plugin using mvn clean install site produces the sad error message:

...\target\eclipse\startup.jar not found.
Have you set up your -DeclipseInstall?

My first thought was that my initial assumption must have been false (that it would be unnecessary to download the RCP Runtime and Delta Pack as detailed in the Codehaus tutorial, because we had started with the Eclipse RCP Developer edition). So I downloaded and unzipped them to a nice little target/eclipse folder of their own, and adjusted the three example POMs accordingly. While this is undoubtedly a Good Thing, because it masssively reduces the size of the exported product, it didn't magically cause a startup.jar to appear.

Luckily, René Gröschke has posted a commentary on the Codehaus tutorial, identifying where it leads the student astray. Luckily I can read German. I'll try to post here what I find out.

Where's the system bundle jar file? (cont.)

2008-01-25T23:55:00.000Z

Well, you probably won't be surprised to learn that the result is no different using the latest maven-eclipse-plugin version 1.5-SNAPSHOT, dated 30th November 2007. I had to create an entire Maven project to invoke the to-maven goal on the maven-eclipse-plugin, just so that I could specify the version I wanted to run. But this should be quite a useful asset in the long run anyway: I can now type

mvn install

mvn deploy

to invoke the corresponding actions. At the moment, however, there is no way to get beyond any plugin whose MANIFEST.MF specifies system.bundle as a required bundle.

The OSGI Bundle Repository Browser confirms that system.bundle does in fact refer to the org.eclipse.osgi jar file. I therefore decided to supply the missing artifact, by using file:deploy-file to create it from org.eclipse.osgi...jar. I added this to the POM of my eclipse-to-maven project and linked it to the initialize phase.

It still doesn't work. For some reason, because the Require-Bundle instruction in MANIFEST.MF does not specify a version number, the maven-eclipse-plugin cannot find the artifact I created. It tells me that it was unable to resolve the version range for version [0,).

So as an experiment, I altered the MANIFEST.MF of the Xerces plugin to explicitly require the actual version number of the org.eclipse.osgi bundle - and hey presto, this was resolved successfully. But we really can't go through all manifests of all Eclipse plugins every time we want to run this plugin and update the system.bundle requirement!

This must be a bug in the maven-eclipse-plugin (caused by a documented incompatibility between Eclipse 3.2 and 3.3, which has led a number of bundle committers to add "Require-bundle: system.bundle" to their manifests), so I have submitted a bug report about it.

In fact, on closer investigation it transpired that there are currently only two Eclipse plugins that fall into this category: org.eclipse.xerces and org.eclipse.xml.resolver. In fact it wasn't even necessary to create a system.bundle artifact - I just altered the two manifests temporarily to express a dependency on the exact version of the org.eclipse.osgi plugin supplied with the Eclipse installation in question. With this modification, the eclipse:to-maven goal was able to run successfully to completion.

On to the next stage! But first, there is a weekend to take care of...

Where's the system bundle jar file?

2008-01-25T01:43:00.000Z

Quick update to my last posting: a number of Eclipse plugins have recently sprouted this RequireBundle attribute thanks to a published incompatibility between Eclipse 3.2 and 3.3 in the area of Boot Delegation.

It looks as if the System Bundle is either mapped to org.eclipse.osgi or doesn't really exist as a separate jar file in its own right at all. Maybe there's a newer version of the Maven Eclipse plugin that understands this?

Creating Maven artifacts from Eclipse plugins

2008-01-25T01:03:00.000Z

Along the long and tortuous road of creating Maven builds for Eclipse RCP applications, one of the first steps is to take the Eclipse plugins that make up the target platform and convert them into Maven artifacts. This ensures that the Maven POM is able to refer to them by its holy triad of group ID, artifact ID and version.

In the Maven Eclipse Plugin, a goal eclipse:to-maven is provided that is meant to satisfy exactly this need. Given an Eclipse installation directory, it trawls through all the plugins it can find and installs them in your local Maven repository, even generating POMs that correctly show interdependencies between plugins. If you optionally specify a deployTo parameter, it will even deploy the resulting artifacts to a remote repository for everyone on your project to use.

When I tried this for my Eclipse RCP installation, however, it failed. There were two versions of the plugin org.apache.xerces in the plugins folder, and Maven could not resolve which of them to turn into a Maven artifact. This might not have been a problem but for the fact that the version numbers of both were the same (2.0.8) apart from the timestamp - one was a folder, the other a self-contained jar file.

The same problem has also been encountered by others - e.g. when trying to use the slightly older but equivalent eclipse:make-artifacts goal.

Having checked in the Eclipse Plugin View that there was nothing dependent on it, I removed the older of the two Xerces plugins. The to-maven goal was now able to run - but the build still didn't complete (see below).

I consider this to be a bug in the Eclipse RCP distribution version 3.3.1.1. Despite a lengthy discussion in the bug tracker five years ago, nothing appears to have been done. I have accordingly lodged a new bug report and we'll see whether this gets fixed in a future release.

The new problem now is the generated POM for org.apache.xerces. For some reason the Maven eclipse plugin is unable to determine the version number of some dependency (the error message doesn't make a lot of sense):

[INFO] Unable to resolve version range for dependency
Dependency {groupId=system, artifactId=bundle,
version=[0,), type=jar} in project org.apache.xerces

Inspection of the MANIFEST.MF in the Xerces plugin jar file reveals:

Bundle-Version: 2.8.0.v200705301630
Require-Bundle: system.bundle,org.apache.xml.resolver;
visibility:=reexport

So that is why Maven can't be sure what version is wanted. Luckily, this particular artifact is already in the Maven central repository, so to short-circuit all this I removed the Xerces plugin jar temporarily as well and re-ran the Maven command.

To my surprise, the build failed this time with the following message:

[INFO] Unable to resolve version range for dependency
Dependency {groupId=org.apache, artifactId=xerces,
version=[2.8.0,3.0.0), type=jar}
in project org.eclipse.wst.xml.core

This makes no sense, since version 2.8.0 of org.apache.xerces clearly does exist in the central repository. Maybe Maven doesn't consider 2.8.0-v200705301630 to be in the above range?

So I tried another tack: restore the older Xerces plugin to the Eclipse plugins folder. This time the Xerces plugin is successfully installed in the local repository, and the plugin org.eclipse.wst.xml.core likewise. Instead, the build fails a bit later with:

[INFO] Unable to resolve version range for dependency
Dependency {groupId=system, artifactId=bundle,
version=[0,), type=jar}
in project org.apache.xml.resolver

This is exactly the problem that the Xerces plugin failed on in the first instance. So I suspect that I'm going to have to find out how to create a Maven artifact called system.bundle. Watch this space.

Build Eclipse RCP products using Maven 2 - how hard can it be?

2008-01-23T20:40:00.000Z

I am working as part of a project team developing a suite of applications that use Eclipse RCP as their platform. We are also using CruiseControl for continuous integration and testing. We currently build Eclipse RCP products, features and fragments using Eclipse PDE, partly because of its OSGI support. We would like to move to using Maven 2 in order to match the way we build all our Eclipse plugins and Java libraries. This will make it straightforward to generate comprehensive documentation on the Web, stay in control of dependencies, link into our bug tracking and progress tracking system and so on.

A parallel initiative is my attempt to set up the Eclipse development environment for each team member so that Maven builds and Eclipse builds work seamlessly together on the workstation. Initially I thought that this would require the use of the Maven 2 Integration for Eclipse plugin. However, it turned out that even this is not nearly as closely integrated into Eclipse as Ralf Quebbemann's equivalent Maven 2 Integration is integrated into IntelliJ IDEA, for example. We found it easiest to configure various Maven goals as "external tools" in Eclipse's Run menu and to use the Maven eclipse plugin to generate or update the Eclipse classpath and project files (mvn eclipse:eclipse).

[Footnote: mvn eclipse:configure-workspace works during installation, but only if the configured Maven repositories are online, which rather defeated my objective of creating a standalone developer kit installer. I found that this goal simply inserts a definition of the M2_REPO classpath variable into one of Eclipse's metadata files, so I made the installer do that directly instead. It now runs much faster too.]

Despite the existence of the book Better Builds with Maven, a common complaint about Maven 2 is that it is not sufficiently well documented for heavy-duty use. The book was last updated in August 2007, since when two further releases of Maven have come out. The book doesn't mention Eclipse RCP or PDE. In trying to marry the use of Maven with Eclipse, we are very much having to hoe our own furrow.

I'm using this blog to record the approaches we take and how we fare. Hopefully after a few weeks (not too many!) it will be possible to summarise all the findings into a "how to" for future reference.

My current understanding is that the most promising approach would be using a combination of the maven-eclipse-plugin (with the PDE property set to true) and the pde-maven-plugin. These allow us to create the Eclipse project and classpath files slightly differently in order to permit "headless" PDE builds to take place. As I understand it, the PDE Maven plugin generates an Ant build file that can then be used to run the headless PDE build. There are a few wrinkles; for example we may have to obtain a more recent snapshot of the pde-maven-plugin because the last official release doesn't appear to work with Europa.

One project which does appear to have been successful in building RCP products using Maven is jLibrary, an open source document management / content management system. Martin Perez and Ryszard Petrowski have documented their build approach for jLibrary 1.2. I intend to study this and learn from it. More anon!