Testing TextTest

Geoff has just put up a couple of new pages on the texttest website, with some coverage statistics for his self tests. He uses coverage.py to produce this report which shows all the python modules in texttest, and marks covered statements in green. I think it's pretty impressive - he's claiming over 98% statement coverage for the over 17 000 lines of python code in texttest.

I had a poke around looking for some numbers to compare this to, and found on this page someone claiming Fitnesse has 94% statement coverage from its unit tests, and the Java Spring framework has 75% coverage. It's probably unwise to compare figures for different programming languages directly, but it gives you an idea.

Geoff also publishes the results of his nightly run of self tests here. It looks a bit complicated, but Geoff explained it to me. :-) He's got nearly 2000 tests testing texttest on unix, and about 900 testing it on windows. As you can see, the tests don't always pass, some are annoying ones that fail sporadically, some are due to actual bugs, which then get fixed. So even though he rarely has a totally green build, the project looks healthy overall, with new tests and fixes being added all the time.

Out of those 3000 odd tests that get run every night, Geoff has a core of about 1000 that he will run before every significant check-in. Since they run in parallel on a grid, they usually take about 2 minutes to execute. (When he has to run them at home in series on our fairly low spec linux laptop they take about half an hour.)

Note that we aren't talking about unit tests here, these are high level acceptance tests, running the whole texttest system. About half of them use PyUseCase to simulate user actions in the texttest GUI, the rest interact with the command line interface. Many of the tests use automatically generated test doubles to simulate interaction with 3rd party systems like version control, grid engines, diff programs etc.

Pretty impressive, don't you think? Well I'm impressed. But then I am married to him so I'm not entirely unbiased :-)

Domain Specific Languages for Selenium tests

I've been doing some work lately creating automated functional test suites using Selenium RC to simulate user interaction with a web GUI. I discovered quickly that the tests you record directly from selenium are rather brittle, and hard to read. In order to make the tests more robust and readable, I have been extracting reusable chunks of script that make sense from the user perspective, into separate methods. For example when testing a page for registering a new provider, you might have a ProviderPage domain class, with method "createNewProvider". This method encapsulates all the selenium calls that interact with the page, and lets your test be written in terms of the domain.

I just saw this article from Patrick Wilson Welsh basically saying the same thing, only his DSL has three layers of indirection instead of just two. As well as encapsulating page operations in a Page class, he encapsulates operations on widgets within a page. I hadn't thought of doing that. It makes the code in the Page class more readable. I might try that, and see if it improves my code.

What other people have written about new dojos

Gathering ideas for my new dojo :-)

Ivan Sanchez wrote about starting a coding dojo, and he rekons a Randori is best with 10 people or less. We will be more than 10 at JDojo@gbg. He suggests a prepared kata in that case. That might be possible. His favourite starting kata is KataMinesweeper.

Danilo Sato wrote about how to find suitable Katas, and suggests several for beginning dojos, including KataMinesweeper.

Gary Pollice wrote an article about what a coding dojo is, which is quite well explained, but doesn't give any specific advice for new dojos.

The guys running the finnish dojo have a similar article about what a coding dojo is, and some rules. They put a maximum of 15 participants on their randori. They also introduce "iterations" of 30 minutes, and spend 5 minutes planning in between.

Lots of ideas to think about, anway.

a new dojo - JDojo@Gbg

I'm planning to start a new dojo this autumn, called JDojo@Gbg. I was inspired by the guys at Responsive in Linköping, who I met at XP2009. They have been running a dojo for some time now, and find it is an excellent way to introduce programmers from their clients to the ideas of Test Driven Development. I think we could do with more test infected programmers about the place in Göteborg, too.

I already run a dojo as part of GothPy, and Got.rb also runs regular Kata/dojo evenings, but because those programming languages are not mainstream, many developers wouldn't consider coming along. That is why the new dojo is explicitly going to use Java, or at least, the JVM platform.

I'm thinking about what Katas we are going to tackle at the new dojo, and last night I had a go at KataFizzBuzz in Java. It is an extremely simple problem to solve, and initially I thought it was too easy to be a Kata actually. Then at agile2008 I was looking around for a Kata that Michael Feathers and I could perform in 4 minutes for the "Programming with the Stars" competition, and it seemed to fit the bill. I was quite pleased we got done in that short amount of time (in python of course :-)

A couple of people have commented that this Kata is actually quite good for teaching TDD, just because it is so simple to solve. People are forced to think about TDD instead of the problem. It can easily be made more interesting by adding new requirements too. So I think I might try it out at JDojo@Gbg.