Practical GUI testing using SikuliX

Rejoice with me - after much banging of head against various walls, I have finally succeeded in building a GUI test out of Java, jUnit, Gradle and SikuliX 1.1.0. It starts a new game in the Mac OS Chess application, enters the opening move using drag/drop, and quits the application (answering correctly in the "save" dialog).

The secret was to give up on the idea of using the Python test script fragments created in the SikuliX IDE directly. Rather, I now translate these scripts line by line into Java; e.g. from
wait("xyz.png")
to
screen.wait("xyz.png");

Each short Python script is converted to a simple method on an object representing the application or individual screen under test. These methods can then be invoked as fixtures from any test framework, e.g. jUnit or Fit. The resulting test suite is built with Gradle and can be run as a standalone application to test the target application end-to-end.

One of the tricky bits was to set the image search path correctly so that the images embedded in the jar file can be located by the SikuliX API, whether you're running in the IDE or standalone. By storing a dummy SikuliX script within the src/main/resources folder (e.g. images.sikuli) you can use the SikuliX IDE directly to capture and fine-tune images for use in your tests.

Depending on the CI environment of the project, another tricky part may be to provide both the application under test and the test suite with a graphical pseudo display on which to run, but there is quite a lot of advice in the SikuliX documentation as well as on Stack Overflow about that.

Next steps:

• Test this approach under Windows and Linux too
• Enable the use of FIT in place of jUnit (more appropriate for whole-system tests)
• Make use of the optional Tesseract library to read back values from the screen
• Build and run test suite in various CI environments

BDD for JSP in Eclipse with Groovy Webtest

Pak Wing and I needed to put together a rapid prototype of a document generator that delivers web pages using custom templates together with various back-end sources of information. The first thing we did, of course, was to sketch out a test plan to identify the capabilities of our system. A simple first test was to see that the correct page title would be generated in response to a GET with varying parameters in the query string, including an invalid case.

The next question was how to execute the tests. My previous experience with Canoo Webtest had been favourable, so I was keen on this approach - but not on the traditional XML notation that Webtest uses. Since it has become feasible to use Groovy as the test scripting language, we decided to use this as it can support the behaviour-driven approach (where test cases are expressed in the "given, when, then" form), is a more compact notation than XML and easier for our stakeholders to read.

However, Canoo's manuals are a little light on detail about how to get all this to work in an Eclipse environment, so I'm recording my findings here as I go along.

Software to Install
Packages:

• Java SE JDK
• Eclipse
• Groovy
• Canoo Webtest
• An application server - in our case, we wanted to use JSP, so Tomcat version 7 was the obvious choice. As we're developing under Windows 7, we chose the Windows Service installer version (but note that the service should not be started - Eclipse will run the server when you're ready to deploy the application)

Eclipse plug-ins:

• Eclipse Web Developer Tools
• Eclipse XML Editors and Tools
• Groovy-Eclipse Feature (optionally plus sources)
• Web Page Editor (optional)
• Don't forget integration with your favourite Software Configuration Management system

I hope that was everything - please drop me a comment if you find you're missing some vital component that I've forgotten to mention.

Starting your Project
After a number of experiments, we found that the most straightforward approach was to create a new Dynamic Web Project and to convert it to a Groovy project afterwards, using the Groovy entry in the context menu. You can also convert to a Maven project subsequently, if that is your preferred build system and you have the M2E (Eclipse Maven Project) plugin installed.

I like Maven's organisation of source folders into src/main and src/test. Put your Java files (e.g. servlets) in src/main/java and test scripts in src/test/groovy. HTML and JSP files go into src/main/resources. Set up the Java Build Path appropriately.

It is standard practice to name the package for a JUnit test script identical to the package of the class under test. We followed this practice also for Groovy test scripts. A Groovy Webtest script is in fact a thinly disguised Ant build script, but you run it as a JUnit test (while avoiding all the tedium of writing JUnit test cases in Java). The Ant libraries are supplied with Webtest.

Running Your First Test
Here's my version of the simple test script shown on the Canoo home page. Note the addition of firewall configuration:



If you paste this into your own Eclipse project, you will immediately be informed that the import "com.canoo.webtest.WebtestCase" cannot be resolved. To fix this, open the Java Build Path configuration, click on the Libraries tab and add a new User Library. I called it WEBTEST_LIBS. To define this User Library, click "Add JARs", navigate to the lib folder under your Webtest installation directory, highlight all jar files you find there and click OK.

Obviously you should amend the firewall configuration in the above constructor function TestGroovyWebTest() as appropriate.

To run the test script, simply right-click the Groovy file, select "Run as..." and click "JUnit Test". You should get both the standard JUnit output of a green bar (hooray!) and a Webtest monitor window that shows the test in progress, followed by a result screen in your web browser.

Create and Deploy a Dummy Application
For the next stage, we wanted the Groovy test script to actually interact with the application we were developing.

First configure your server. In Eclipse, go to the View menu, select "Show View", expand the "Server" section and select "Servers". If this is not available, you probably have not installed the Eclipse Web Developer Tools feature correctly.

Right-click in the Servers panel and select "New". Add the server (you should find it in a pull-down list). Configure the correct run-time environment for the application server you have installed, and name it (probably "localhost"). Click "Finish" (don't configure any apps at this stage).

Create a very simple index.jsp page under src/main/resources:



Now you can right-click the JSP file, select "Run as..." and click "Run on server". The output ought to appear in a new edit panel.

You can now add a test case to invoke your new app and check that it has produced the right page:



Gotchas
There are a few problems to look out for:

• If you collaborate with someone else on your project, and their Eclipse and other configurations do not match yours precisely, you may find that some of the build path configuration has to be changed after they import your project. To minimise this, make sure that for example you configure "workspace default" as the JRE for the project.
• In one case, we found that Webtests could not be run more than once. The reason was that each run produces a report under the same folder into which the Java and Groovy compilers place class files. For some reason, with certain installations of the Tomcat server, the synchronisation mechanism meant that the test reports were being deployed along with the application, which meant that Webtest could not delete them before running the next test. In these cases, we found that the only remedy was to restart Eclipse and then start Tomcat again - too tedious in the long run. It might be less hassle just to run the Groovy test script outside Eclipse, but we have not tried this.

Generate test data for Java development easily

A colleague has just pointed out the release of Andy Gibson's DataFactory library as a Maven artifact. It generates arbitrarily long lists of random names and addresses, dates, date ranges (with arbitrary constraints), random selections from lists, numbers, strings, Internet addresses etc. etc.

Test-Driven Design is not testing

I've recently worked with a team doing its first agile project (though one or two team members had been involved in an agile project before). The most difficult concept to get across was TDD - test driven design. I found that people really didn't grok the idea until I pair-programmed with them for a couple of hours. I wonder why that might be.

Dan North has suggested one possibility. He observed that newcomers to TDD often don't get the really big payback because they continue to think that TDD is mainly about testing - even if they will admit that writing the tests before the code leads to better quality code. They never transition to treating TDD as a design process, letting them discover the API to a component they're writing, nor to the realisation that TDD is about defining the behaviour of their component and its interactions with other components of the system.

Keith Braithwaite has put forward another consideration. In physical engineering disciplines, practitioners speed up their work process by using gauges. There are many kinds, from the simple spark plug gap gauge, which is simply a sliver of metal to slide between the electrodes, to electronic vernier calliper gauges that can be pre-set to a precise dimension with tolerances above and below. Each workpiece is tested at each stage of the process by checking its dimensions with the appropriate gauge(s). Workpieces that are out of tolerance are sent back for rework or scrapped. Our unit tests are a bit like that - they provide a safeguard that the software component we're working on still meets all its requirements following any engineering we've done.

It occurred to me today that unit and acceptance tests, particularly if automated, perform another valuable function in the context of an agile (especially a lean) development process. Whereas the waterfall processes familiar to most developers are characterised by "quality gates" at key stages, every single artifact in an agile development has its own little quality gate, manifested in the appropriate tests. This theoretically frees the development process from the usual bottlenecks that the quality gates tend to become.

I say "theoretically", because in many instances agile development projects have to take place within a quality system that doesn't take advantage of incremental delivery. Instead, continued approval and in many cases funding for the project tends to be contingent on passing the traditional quality gates following requirements analysis, functional specification, high-level design, low-level design, coding, integration, system test and acceptance. Project managers are therefore forced to conjure up some kind of spurious linkage between the milestones laid down in the rigid quality system and some arbitrary points along their product release plan. This can hamper their freedom to adjust the release plan in response to changing circumstances and emerging technical insights.

This could be avoided if the quality system could recognise that properly written tests represent every work product of a software development project apart from the code itself. It should therefore simply insist on a verification at each iteration (or at each release, perhaps) that the tests comprehensively and comprehensibly represent the requirements of the business on the system under development and that the required set of tests pass repeatably. I say "the required set" because there's always the possibility that some tests will intentionally fail - e.g. where they have been written to test features that are not yet in the current release.

In other words, TDD can be used to eliminate the quality-gate bottlenecks of quality systems that assume waterfall development processes.

SPA2009 - first impressions

Maybe I'm biased, of course, but for me the SPA2009 conference felt even better than last year's. I think it had the right mix of technical and non-technical sessions, mostly of very high quality, and a few really interesting BOF (birds of a feather) sessions - which for once, didn't feel as if they were just squeezed in at the last minute.

Functional programming (in particular, Haskell) was a major theme this year, as was testing. My eyes were opened to the possibility of doing test-driven development (TDD) in Haskell - in fact, functional languages are better at this than imperative (stateful) languages. It was also good to meet up with a lot of familiar faces at the joint XtC meeting on the Tuesday night.

My session notes are on my other laptop. I will try to post them on the SPA SG site over the Easter weekend, other duties permitting.

Automating testing of web sites with multiple browsers

After several days of trying, I've just abandoned an attempt to automate web site tests with multiple browsers (Firefox, Internet Explorer, Safari and Opera under various operating systems). My findings are written up here. Briefly, I think that Selenium RC has the right architecture, but currently the linkage between the RC server and the browser only works 100% for Firefox. This isn't enough.

It seems from the documentation that Watir should support the functionality needed. Can anyone confirm that it's possible to test pages such as this one in at least three different browser types with it?

Update (9 April): I discovered that Selenium RC has a start-up parameter "-proxyInjectionMode", which is equivalent to choosing the "*pi..." launchers where available. Unfortunately this only works with "*firefox", so you might just as well use "*pifirefox". It isn't clear to me why it causes tests to fail when they launch a browser with "*mock".

I was also reminded at the SPA2009 conference that it might be worth trying out Cucumber, rSpec, Concordion or other tools, so long as they will drive more than one browser.

Test-Driven Web Development on Spring and Hibernate

Adam Shimali and I will be piloting our TDD course at SPA2009 on the Sunday afternoon. Three two-hour samples of a full two-day course for free - and you get a 10% discount off the cost of the full course for yourself or a colleague.

The course is very hands-on. If you bring a laptop (Windows or MacOS) we can provide you with the complete Eclipse development environment installation on a CD or memory stick - you'll be good to go in about five minutes. It will help if you've done a bit of Java programming before, but even this is not essential as you'll be paired with someone experienced.

There's still time to book for SPA2009, but don't delay!

Mock external components using AOP

Mark Nadelson has provided a well-written tutorial for a novel (to me, at any rate) use of Aspect-Oriented Programming in Java. Read the "print" version to get the whole article in one lump without the commercials.

When you write jUnit tests, you often encounter a need to create mock objects (e.g. using JMock or EasyMock) that can simulate some external dependency of the software under test. Mark has extended this idea to any external component that has an interface. In order to determine whether the software under test uses a mock component or the real one, he uses dynamic aspect-weaving by means of Spring AOP with AspectJ annotations.

I've got to have a play with this approach to see whether it can be used flexibly enough to support TDD fully. One problem I could foresee is that the pointcut specification is not in the test script but in the mock object (i.e. the class that implements the aspect). So it might be less than straightforward to link the mock object to specific test cases/steps. By contrast, when you use JMock in a jUnit test, you specify both the mock object and its expectations in one place.

Testing web applications? Cucumber is the cool new kid on the block

I've had quite complimentary things to say about Canoo WebTest, but one thing it doesn't provide out of the box is a representation of the tests that your average customer is able to read and understand intuitively - that is to say, a tabular format like Fit or a narrative of the "Given... when... then... " form.

Cucumber to the rescue! I heard several people mention this at the recent XP Day in London. I hope to investigate it in more depth soon.

I would be interested to know whether it can be used to test anything other than Ruby code and whether it can drive a web browser to interact with Rich Internet Applications (RIAs). Stuart Ervine's and Nat Pryce's GWT development demo at XP Day was truly impressive. Nat has combined WebDriver with WindowLicker to create a clean interface between a test script and synchronous or asynchronous RIAs. But telling a jUnit test to move the mouse cursor into a particular widget and click the button was nothing if not verbose!

SOA adoption reported to be accelerating

I was interested to read this interview with Jason English, the VP of Corporate Marketing for iTKO. It shows that Service Oriented Architecture is coming of age when vendors are offering viable testing and virtualisation solutions.

It also previews the virtual conference SOA in Action, coming on November 19th, 2008, for which it looks as if it is free to sign up. "SOA in Action will feature keynotes by Gartner's Yefim Natis and Forrester's Randy Heffner, as well as a panel discussion led by ebizQ's Joe McKendrick and featuring Phil Wainewright of ZDNet and Dr. Chris Harding of The Open Group."