Building Successful Communities of Practice

Just got back from a stimulating evening meeting of the BCS SPA specialist group. The presenter was Emily Webber on Building Successful Communities of Practice. If I understood correctly, a community of practice is a cross-functional, self-selected group of practitioners doing roughly the same tasks in different teams, business units or even organisations. Spotify has identified almost the identical concept in its "guilds".

The big takeaway for me (apart from the fact that Emily in real life doesn't look nearly as similar to Sarah Millican as she does in her photo) was that the mutual support provided by such a group gives people more autonomy and more mastery of their craft, both of which are strong motivating factors and promote happiness, which in turn results in demonstrably higher productivity and lower staff turnover (a study by Warwick University showed the productivity of happy employees to be 10-12% higher than average, while that of unhappy ones was 10% or more below). This would seem to suggest that allowing as much as one day a week to employees to exchange ideas and experiment with ways to improve their technique could be a worthwhile investment.

BDD for JSP in Eclipse with Groovy Webtest

Pak Wing and I needed to put together a rapid prototype of a document generator that delivers web pages using custom templates together with various back-end sources of information. The first thing we did, of course, was to sketch out a test plan to identify the capabilities of our system. A simple first test was to see that the correct page title would be generated in response to a GET with varying parameters in the query string, including an invalid case.

The next question was how to execute the tests. My previous experience with Canoo Webtest had been favourable, so I was keen on this approach - but not on the traditional XML notation that Webtest uses. Since it has become feasible to use Groovy as the test scripting language, we decided to use this as it can support the behaviour-driven approach (where test cases are expressed in the "given, when, then" form), is a more compact notation than XML and easier for our stakeholders to read.

However, Canoo's manuals are a little light on detail about how to get all this to work in an Eclipse environment, so I'm recording my findings here as I go along.

Software to Install
Packages:

• Java SE JDK
• Eclipse
• Groovy
• Canoo Webtest
• An application server - in our case, we wanted to use JSP, so Tomcat version 7 was the obvious choice. As we're developing under Windows 7, we chose the Windows Service installer version (but note that the service should not be started - Eclipse will run the server when you're ready to deploy the application)

Eclipse plug-ins:

• Eclipse Web Developer Tools
• Eclipse XML Editors and Tools
• Groovy-Eclipse Feature (optionally plus sources)
• Web Page Editor (optional)
• Don't forget integration with your favourite Software Configuration Management system

I hope that was everything - please drop me a comment if you find you're missing some vital component that I've forgotten to mention.

Starting your Project
After a number of experiments, we found that the most straightforward approach was to create a new Dynamic Web Project and to convert it to a Groovy project afterwards, using the Groovy entry in the context menu. You can also convert to a Maven project subsequently, if that is your preferred build system and you have the M2E (Eclipse Maven Project) plugin installed.

I like Maven's organisation of source folders into src/main and src/test. Put your Java files (e.g. servlets) in src/main/java and test scripts in src/test/groovy. HTML and JSP files go into src/main/resources. Set up the Java Build Path appropriately.

It is standard practice to name the package for a JUnit test script identical to the package of the class under test. We followed this practice also for Groovy test scripts. A Groovy Webtest script is in fact a thinly disguised Ant build script, but you run it as a JUnit test (while avoiding all the tedium of writing JUnit test cases in Java). The Ant libraries are supplied with Webtest.

Running Your First Test
Here's my version of the simple test script shown on the Canoo home page. Note the addition of firewall configuration:



If you paste this into your own Eclipse project, you will immediately be informed that the import "com.canoo.webtest.WebtestCase" cannot be resolved. To fix this, open the Java Build Path configuration, click on the Libraries tab and add a new User Library. I called it WEBTEST_LIBS. To define this User Library, click "Add JARs", navigate to the lib folder under your Webtest installation directory, highlight all jar files you find there and click OK.

Obviously you should amend the firewall configuration in the above constructor function TestGroovyWebTest() as appropriate.

To run the test script, simply right-click the Groovy file, select "Run as..." and click "JUnit Test". You should get both the standard JUnit output of a green bar (hooray!) and a Webtest monitor window that shows the test in progress, followed by a result screen in your web browser.

Create and Deploy a Dummy Application
For the next stage, we wanted the Groovy test script to actually interact with the application we were developing.

First configure your server. In Eclipse, go to the View menu, select "Show View", expand the "Server" section and select "Servers". If this is not available, you probably have not installed the Eclipse Web Developer Tools feature correctly.

Right-click in the Servers panel and select "New". Add the server (you should find it in a pull-down list). Configure the correct run-time environment for the application server you have installed, and name it (probably "localhost"). Click "Finish" (don't configure any apps at this stage).

Create a very simple index.jsp page under src/main/resources:



Now you can right-click the JSP file, select "Run as..." and click "Run on server". The output ought to appear in a new edit panel.

You can now add a test case to invoke your new app and check that it has produced the right page:



Gotchas
There are a few problems to look out for:

• If you collaborate with someone else on your project, and their Eclipse and other configurations do not match yours precisely, you may find that some of the build path configuration has to be changed after they import your project. To minimise this, make sure that for example you configure "workspace default" as the JRE for the project.
• In one case, we found that Webtests could not be run more than once. The reason was that each run produces a report under the same folder into which the Java and Groovy compilers place class files. For some reason, with certain installations of the Tomcat server, the synchronisation mechanism meant that the test reports were being deployed along with the application, which meant that Webtest could not delete them before running the next test. In these cases, we found that the only remedy was to restart Eclipse and then start Tomcat again - too tedious in the long run. It might be less hassle just to run the Groovy test script outside Eclipse, but we have not tried this.

Fun with Robots, Arduino and Android

Together with Mike Hogg from my firm, Zühlke Engineering, I recently developed a demonstration project over a three-day training exercise. We purchased a simple robot and added a set of three infra-red sensors capable of detecting a black line marked out on the ground with electrician's tape. We had to develop a line-sensing algorithm and control logic to follow the line and stop at stations (a short length of tape at right angles to the line) until instructed to move on.

We followed a test-driven development (TDD) approach for C++ using CppUTest, as described by James Grenning, to create a state machine to turn sensor readings into motor control instructions (with hindsight, we might have saved time by resorting to the Machine Objects library). To my surprise, once we loaded the finished firmware to the Arduino board, it worked very well with only a bit of tweaking of the motor speed settings corresponding to "hard left", "slight left" etc.

Then we linked this with an application we developed for an Android tablet computer (Motorola Xoom) that could be used to instruct the robot to go to any selected station, pick up/drop a "payload" (only conceptually) and return to base. The two components communicated by Bluetooth radio link.

Finally, the tablet app was linked via a RESTful Web Service interface to an "enterprise" system developed by other colleagues on the training course, that controlled the whole delivery network and knew which consignments had to be delivered to what stations. The tablet would send a message to the enterprise server telling it where the robot had arrived, and receive back an instruction containing the name of the next station to move to. The robot would then move off automatically after 5 seconds.

The whole thing was enormous fun (as well as very instructive) and was demonstrated successfully to the entire group (see photo).

Jason Gorman and Simon Peyton Jones, among others, have recently been at pains to point out the shortcomings of IT education in the UK. Thinking about the above exercise, it occurred to me that it is just the sort of project that could engender enthusiasm for the subject from a wide range of school students. It involves lots of varied tasks from assembling the hardware to designing the user interface, and of course the devising of suitable protocols between all the components.

Simon Peyton Jones of Microsoft Research will be presenting an interactive talk about Computing At School on 2nd November in London, and the following month (6 December) Mike and I plan to do a session on the Robot Shop exercise at the same location. Anyone interested will be most welcome.

Breaking up the big rocks

A colleague has circulated Richard Lawrence's nine patterns for dividing up user stories. These are going to be very useful. I wonder if someone will add more to make the magic Baker's Dozen?

BCS SPA2010 conference

Final day of the conference - it's gone much too quickly as usual. My favourite session to date has been the brief whirlwind tour of agile practices given by Gwyn Morfey and Laurie Young of New Bamboo - "the Sword of Integration". This was a highly interactive session that involved everyone standing up and moving about enthusiastically, which despite the cramped room, meant that we all ended up remembering something instantly useful from the session.

By the way, the sword of integration itself is just one example of an instant solution to a pressing problem. The situation was that multiple developers checking in their changes would cause each other to have merge conflicts. The solution: a paper "sword" quickly assembled, which when held conferred on the holder the right to check in - and hit anyone who checked in when they shouldn't. The principle being illustrated is "just try it" - there is no need to get it absolutely right first time. If it doesn't work, we can change it later.

Craftsmanship for Teams

Very interesting discussion thread on Software Craftsmanship as a team exercise. In response to Cory Foy's posting, Steven Smith makes an interesting analogy with coaching a sports team and says how that is actually carried through in his practice.

Test-Driven Design is not testing

I've recently worked with a team doing its first agile project (though one or two team members had been involved in an agile project before). The most difficult concept to get across was TDD - test driven design. I found that people really didn't grok the idea until I pair-programmed with them for a couple of hours. I wonder why that might be.

Dan North has suggested one possibility. He observed that newcomers to TDD often don't get the really big payback because they continue to think that TDD is mainly about testing - even if they will admit that writing the tests before the code leads to better quality code. They never transition to treating TDD as a design process, letting them discover the API to a component they're writing, nor to the realisation that TDD is about defining the behaviour of their component and its interactions with other components of the system.

Keith Braithwaite has put forward another consideration. In physical engineering disciplines, practitioners speed up their work process by using gauges. There are many kinds, from the simple spark plug gap gauge, which is simply a sliver of metal to slide between the electrodes, to electronic vernier calliper gauges that can be pre-set to a precise dimension with tolerances above and below. Each workpiece is tested at each stage of the process by checking its dimensions with the appropriate gauge(s). Workpieces that are out of tolerance are sent back for rework or scrapped. Our unit tests are a bit like that - they provide a safeguard that the software component we're working on still meets all its requirements following any engineering we've done.

It occurred to me today that unit and acceptance tests, particularly if automated, perform another valuable function in the context of an agile (especially a lean) development process. Whereas the waterfall processes familiar to most developers are characterised by "quality gates" at key stages, every single artifact in an agile development has its own little quality gate, manifested in the appropriate tests. This theoretically frees the development process from the usual bottlenecks that the quality gates tend to become.

I say "theoretically", because in many instances agile development projects have to take place within a quality system that doesn't take advantage of incremental delivery. Instead, continued approval and in many cases funding for the project tends to be contingent on passing the traditional quality gates following requirements analysis, functional specification, high-level design, low-level design, coding, integration, system test and acceptance. Project managers are therefore forced to conjure up some kind of spurious linkage between the milestones laid down in the rigid quality system and some arbitrary points along their product release plan. This can hamper their freedom to adjust the release plan in response to changing circumstances and emerging technical insights.

This could be avoided if the quality system could recognise that properly written tests represent every work product of a software development project apart from the code itself. It should therefore simply insist on a verification at each iteration (or at each release, perhaps) that the tests comprehensively and comprehensibly represent the requirements of the business on the system under development and that the required set of tests pass repeatably. I say "the required set" because there's always the possibility that some tests will intentionally fail - e.g. where they have been written to test features that are not yet in the current release.

In other words, TDD can be used to eliminate the quality-gate bottlenecks of quality systems that assume waterfall development processes.

24th March 2009: Theory of Constraints Challenged

My sincere thanks to Kevin Rutherford and Allan Kelly for co-presenting a fascinating session about lean software development to the BCS Kingston & Croydon branch on 24th March this year, entitled "Lean, Constraints, Action!". The audience was excellent too and helped us re-create a famous experiment related by Eliyahu Goldratt in "The Goal".



(Click images to see a larger version)

I had participated in this game previously at the London XP Day 2008 (facilitated by Karl Scotland in an Open Space session). It is designed to demonstrate an intuitively paradoxical finding: that a lean, pull-oriented flow substantially reduces the amount of inventory or work in progress (WIP), while improving throughput.

However, I had a sneaky feeling that the experiment was biased, because in the first "push" simulation, the assembly line was not pre-loaded with WIP, while in the second "pull" simulation, the line was pre-loaded with workpieces at each "workstation's" input buffer up to either the maximum limit or to 50% of that limit. Therefore in a simulation of 10 rounds (equivalent to ten working days - approximately equal to the average cycle time in a six-workstation setup) the push simulation will only start to produce output towards the very end of the simulation, while the pull simulation will produce something from the very first day.



So I got Allan and Kevin to agree to vary the rules a little bit, to try to get closer to a "steady state" from the first "day". Before each of the two simulations, our teams placed three Lego blocks on each of the coasters representing the input buffers of the second through sixth workstations (the first workstation of course has the whole of the product backlog as its input hopper). In fact, as it turned out, four workpieces would have been closer to the true steady state in the pull simulation, even more in the push simulation.

Off our teams went and played the production line for ten rounds each. In the push simulation, the die was passed in order from workstation 1 to workstation 6 during each round and the number of workpieces transferred to the next input buffer was the number thrown, up to the number of pieces available in the workstation's input buffer. Instances of starvation were rare under this system, but did occur sometimes. At the end we counted up the number of pieces that had come off the end of the line and the number currently in progress (i.e. on any of the five input buffers for workstations 2 to 6).

In the pull simulation, the die was passed in the opposite direction and the input buffers were constrained to a maximum of six workpieces. So if the next input buffer had three pieces already in it and the player threw anything over a 3, they could only pass along 3 more workpieces (subject to their own input buffer holding at least 3, of course). Once again, the results after 10 rounds were compared.

The results didn't surprise me particularly, but I think some of the others were a little taken aback:



As you can see, the constraint resulted in both lower WIP and lower throughput. This makes sense when you consider that there were far more occasions during the pull game than during the push game when players were unable to process the full number of workpieces indicated by the die.

Looking back at the game notes, it is noted that if the simulation is run for much longer than 10 days, the pull (or Kanban) system "will rarely produce as much as the traditional push". This may have escaped the attention of some readers (or perhaps it's a more recent edit - I don't know).



My conclusion is that you get nothing for free. The cost of reducing WIP is reduced throughput, which is perfectly acceptable as long as you're aware of it. Software development projects are not production lines in any case, so it is very unlikely that any developer will sit around kicking her or his heels if the work runs out on a given day. There are always low priority tasks such as fettling the build system, cleaning up the documentation on the project Wiki, answering user support requests etc. - or just take the next item off the product backlog and raise the kanban limit temporarily.

Distributed bug-tracking in Haskell

At the recent SPA 2009 conference, there was a lot of talk about functional programming, Haskell in particular (a couple of years ago, the flavour of the month had been Erlang). Just to prove that Haskell is no longer "just a research language", along comes DisTract, a distributed issue-tracking system that runs in Firefox browsers. If you're already using Git, Darcs, Mercurial or Monotone as your distributed software configuration management solution, the author reasoned, why shouldn't you be able to close bugs while you're off-line at the same time as you check in your fix? Caveat: I have not tried this yet, but it sounds like a really neat idea. Does anyone know of a user forum?

SPA2009 - first impressions

Maybe I'm biased, of course, but for me the SPA2009 conference felt even better than last year's. I think it had the right mix of technical and non-technical sessions, mostly of very high quality, and a few really interesting BOF (birds of a feather) sessions - which for once, didn't feel as if they were just squeezed in at the last minute.

Functional programming (in particular, Haskell) was a major theme this year, as was testing. My eyes were opened to the possibility of doing test-driven development (TDD) in Haskell - in fact, functional languages are better at this than imperative (stateful) languages. It was also good to meet up with a lot of familiar faces at the joint XtC meeting on the Tuesday night.

My session notes are on my other laptop. I will try to post them on the SPA SG site over the Easter weekend, other duties permitting.

Tools to support agile methods

The whole point of managing a project using an agile method is to use lots of coloured cards and post-it notes, but sometimes you have to fall back on a boring old software tool. The thorny question keeps arising: which one do I recommend?

Many different project management tools exist, at different price-points and levels of functionality. A long list is shown below. It is important to be clear about a project's requirements for a tool before selecting. Avoid the temptation to extend any tool you select or to spend much effort integrating it with the rest of your project environment – this is a sure way to lock yourself into a single supplier.

The following list of tools is not exhaustive, though I have kept adding to it as I came across new tools:

• ]project-open[


• Achievo


• ActiveCollab


• Agilebuddy - reported by a comment on this post to be a full-featured agile project management software tool, which is easy to use and great for Scrum teams. Offered on a subscription model for US$9.95 per user per month


• Agilefant


• Agilo – based on the Trac issue-management tool and said to support only a single project at a time


• airTODO – PMBOK rather than agile, but minimalist (an Eclipse plug-in)


• AxoSoft's OnTime - available as a hosted cloud solution, a Windows native application or as a web application, one comment under this post has reported that it "great for agile / scrum development". Comparable in scope to Trac, but looks a lot more snazzy


• Banana Scrum - according to one comment below, this is a nice, web based tool that helps without getting in team's way


• Extreme Planner


• Greenhopper – a JIRA plugin favoured by some Scrum teams


• IceScrum


• Mingle - provides a shared workspace for agile teams, supporting XP, Scrum and custom hybrid approaches. Includes a virtual card wall, wiki, charts, reports and more. Integrated with issue tracking and continuous build. Priced US$995 per user with substantial discounts for multiple licences, academic institutions etc.


• Pivotal Tracker


• ProjectCards – mixed approach that uses both physical cards and software


• ProjectKoach


• Rally


• Rational Team Concert - integrates work item, continuous integration builds and software configuration management (SCM) on the collaborative infrastructure of the Jazz Team Server


• redMine


• Scrum Edge - someone commented below that it was much simpler to use than most other Scrum tools


• Scrumworks - quite popular free open source package, heavily forms-based, can integrate with JIRA


• Silver Catalyst


• StuffPlanner – Zühlke's inhouse-developed browser-based tool now at version 0.1 and with an Eclipse Mylyn connector available; access can be provided by arrangement


• TargetProcess - designed to support Scrum, XP and custom Agile processes particularly in the .NET environment. Includes a comprehensive set of tools and features, including defect-tracking, test management and customer helpdesk portal. Community edition free for up to 5 users


• teamwork


• tinyPM


• VersionOne - very full-featured, supports Scrum, DSDM, XP and AUP across multiple projects. Priced from US$348 per user per year, there is a free "team edition" for one team of up to 10


• XP Story Studio – no development since 2004


• XPlanner
- a popular free open source planning and tracking tool for XP. Quite simple in keeping with the low-ceremony agile approach

• xProcess - free open source project management and process improvement tool, focused particularly on agile and priority-driven approaches. Preconfigured processes include Scrum, FDD, Prince2, Unified and others; can be tailored. Gantt/Burndowns/target status continuously updated. See also xProcess Europe


• XPWeb

A number of web sites compare small subsets of the available software tools, including

• http://www.agile-tools.net/


• http://brunomiranda.com/past/2008/8/19/agile_project_management_tools/


• http://agilesoftwaredevelopment.com/blog/mattgrommes


• http://www.life-cycle.de/journal/

Test-Driven Web Development on Spring and Hibernate

Adam Shimali and I will be piloting our TDD course at SPA2009 on the Sunday afternoon. Three two-hour samples of a full two-day course for free - and you get a 10% discount off the cost of the full course for yourself or a colleague.

The course is very hands-on. If you bring a laptop (Windows or MacOS) we can provide you with the complete Eclipse development environment installation on a CD or memory stick - you'll be good to go in about five minutes. It will help if you've done a bit of Java programming before, but even this is not essential as you'll be paired with someone experienced.

There's still time to book for SPA2009, but don't delay!

Agile and Lean - complementary or conflicting?

Dave West has contributed an article entitled A Marriage Made in Heaven?. I found it very instructive to read that as well as the comments attached to it. To my mind, there are things that the software community can learn (and has learned) from lean manufacturing, but in many respects software development is much more of a joint creative act. As Dave says, Peter Naur as long ago as 1985 equated programming with collaborative theory-building - in other words, it has much in common with research at the forefront of physics or mathematics, where results are difficult to predict and effort is almost impossible to forecast.