A little while back I was back in Stockholm facilitating another Code Retreat, (see previous post), this time as part of Global Day of Code Retreat, (GDCR). (Take a look at Corey Haines' site for loads of information about this global event that comprised meetings in over 90 cities worldwide, with over 2000 developers attending.)
I think the coding community could really do with more people who know how to do TDD and think about good design, so I'm generally pretty encouraged by the success of this event. I do feel a bit disappointed about some aspects of the day though, and this post is my attempt to outline what I think could be improved.
I spent most of the Global Day of Code Retreat walking around looking over people's shoulders and giving them feedback on the state of their code and tests. I noticed that very few pairs got anywhere close to solving the problem before they deleted the code at the end of each 45 minute session. Corey says that this is very much by design. You know you don't have time to solve the problem, so you can de-stress: no-one will demand delivery of anything. You can concentrate on just writing good tests and code.
In between coding sessions, I spent quite a lot of effort reminding people about simple design, SOLID principles, and how to do TDD (in terms of states and moves). Unfortunately I found people rarely wrote enough code for any of these ideas to really be applicable, and TDD wan't always helping people.
I saw a lot of solutions that started with the "Cell" class, that had an x and y coordinate, and a boolean to say whether it was alive or not. Then people tended to add a "void tick(int liveNeighbourCount)" method, and start to implement code that would follow the four rules of Conway's Game of Life to work out if the cell should flip the state of its boolean or not. At some point they would create a "World" class that would hold all the cells, and "tick()" them. Or some variant of that. Then, (generally when the 45 minutes were running out), people started trying to find a data structure to hold the cells in, and some way to find out which were neighbours with each other. Not many questioned whether they actually needed a Cell class in the first place.
Everyone at the code retreat deleted their code afterwards, of course, but you can see an example of a variant on this kind of design by Ryan Bigg here (a work in progress, he has a screencast too).
Of course, as facilitator I spent a fair amount of time trying to ask the kind of questions that would push people to reevaluate their approach. I had partial success, I guess. Overall though, I came away feeling a bit disillusioned, and wanting to improve
my facilitation so that people would learn more about using TDD to actually solve problems.
At the final retrospective of the day, everyone seemed to be very positive about the event, and most people said they learnt more about pair programming, TDD, and the language and tools they were working with. We all had fun. (If you read Swedish, Peter Lind wrote up the retrospective in Valtech's blog) This is great, but could we tweak the format to encourage even more learning?
I think to solve Conway's Game of Life adequately, you need to find a good datastructure to represent the cells, and an efficient algorithm to "tick" to the next generation. Just having well named classes and methods, although a good idea, probably won't be enough.
For me, TDD is a good method for designing code when you already mostly know what you're doing. I don't think it's a good method for discovering algorithms. I'm reminded of the debate a few years back when Peter Norvig posted an excellent Sudoku solver (here) and people thought it was much better than Ron Jeffries' TDD solution to the same problem (here). Peter was later interviewed about whether this proved that TDD was not useful. Dr Norvig said he didn't think that it proved much about TDD at all, since
"you can test all you want and if you don’t know how to approach the problem, you’re not going to get a solution"
(from Peter Siebel's book "Coders At Work", an extract of which is available in his blog)
I felt that most of the coders at the code retreat were messing around without actually knowing how to solve the problem. They played with syntax and names and styles of tests without ever writing enough code to tell whether their tests were driving a design that would ultimately solve the problem.
Following the GDCR, I held a coding dojo where I experimented with the format a little. I only had time to get the participants do two 45 minute coding sessions on the Game of Life problem. At the start, as a group we discussed the Game of Life problem, much as Corey recommends for a Code Retreat introduction. However, in addition, I explained my favoured approach to a solution - the data structure and algorithm I like to use. I immediately saw that they started their TDD sessions with tests and classes that could lead somewhere. I feel that if they had continued coding for longer, they should have ended up with a decent solution. Hopefully it would also have been possible to
refactor it from one datastructure to another, and to switch elements of
the algorithm without breaking most of the tests.
I think this is one way to improve the code retreat. Just give people more clues at the outset as to how to tackle the problem. In real life when you sit down to do TDD you'll often already know how to solve similar problems. This would be a way for the facilitator to give everyone a leg-up if they havn't worked on any rule-based games involving a 2D infinite grid before.
Rather than just explaining a good solution up front, we could spend the first session doing a "Spike Solution". This is one of the practices of XP:
"the idea is just to drive through the entire problem in one blow, not to craft the perfect solution first time out."
From "Extreme Programming Installed" by Jeffries, Anderson, Hendrickson, p41
Basically, you hack around without writing any tests or polishing your design, until you understand the problem well enough to know how you plan to solve it. Then you throw your spike code away, and start over with TDD.
Spending the first 45 minute session spiking would enable people to learn more about the problem space in a shorter amount of time than you generally do with TDD. By dropping the requirement to write tests or good names or avoid code smells, you could hopefully hack together more alternatives, and maybe find out for yourself what would make a good data structure for easily enumerating neighbouring cells.
So the next time I run a code retreat, I think I'll start with a "spiking" session and encourage people to just optimize for learning about the problem, not beautiful code. Then after that maybe I'll sketch out my favourite algorithm, in case they didn't come up with anything by themselves that they want to pursue. Then in the second session we could start with TDD in earnest.
I always say that the code you end up with from doing a Kata is not half as interesting as the route you took to get there. To this end I've published a screencast of myself doing the Game of Life Kata in Python. (The code I end up with is also published, here). I'm hoping that people might want to prepare for a Code Retreat by watching it. It could be an approach they try to emulate, improve on or criticise. On the other hand, showing my best solution like this is probably breaking the neutrality of how a code retreat facilitator is supposed to behave. I don't think Corey has ever published a solution to this kata, and there's probably a reason for that.
I have to confess, I already broke all the rules of being a facilitator, when I spent the last session of the Global
Day of Code Retreat actually coding. There was someone there who really
wanted to do Python, and no-one else seemed to want to pair with him. So
I succombed. In 45 minutes we very nearly had a working solution,
mostly because I had hacked around and practiced how to do it several times before. It
felt good. I think more people should experience the satisfaction of actually
completing a useful piece of code using TDD at a Code Retreat.
Tuesday, 20 December 2011
Scandinavian Developer Conference
Scandinavian Developer Conference will be held in Göteborg in April, and last week we launched the detailed programme on the website. I've been involved with the conference ever since the first event in 2009, but this year I've taken on increased responsibilities, acting as Programme Chair.
When P-A Freiholtz, the Managing Director at Apper Systems*, (the company behind the conference), approached me about taking this role, I jumped at the chance. My business is all about professional development for software developers, and a conference in Göteborg is a really good opportunity for a lot of local people to hear about what's going on in the world outside. Many of the developers I meet in my work don't often get away from their desks, spending the majority of their time caught up in the intrigues and deadlines of a single project and company environment. SDC2012 will be a chance for a lot of local people to broaden their horizons without it costing their boss a fortune, or disrupting their usual schedule too much.
I wrote an editorial on the front page of the conference website which explains more about what's on the programme and who should come. Everyone involved in software development, basically :-)
These days, we compete in a global marketplace for software development. I'm hopeful about the future, I'm enthusiastic about Swedish working culture, and I think Scandinavian Developer Conference is helping Göteborg grow as a center of competence. Please do join us at the conference.
* formerly named Iptor - now under new ownership.
When P-A Freiholtz, the Managing Director at Apper Systems*, (the company behind the conference), approached me about taking this role, I jumped at the chance. My business is all about professional development for software developers, and a conference in Göteborg is a really good opportunity for a lot of local people to hear about what's going on in the world outside. Many of the developers I meet in my work don't often get away from their desks, spending the majority of their time caught up in the intrigues and deadlines of a single project and company environment. SDC2012 will be a chance for a lot of local people to broaden their horizons without it costing their boss a fortune, or disrupting their usual schedule too much.
I wrote an editorial on the front page of the conference website which explains more about what's on the programme and who should come. Everyone involved in software development, basically :-)
These days, we compete in a global marketplace for software development. I'm hopeful about the future, I'm enthusiastic about Swedish working culture, and I think Scandinavian Developer Conference is helping Göteborg grow as a center of competence. Please do join us at the conference.
* formerly named Iptor - now under new ownership.
Sunday, 27 November 2011
XP2012 Team Challenge
At a conference about software development, wouldn’t it be useful to have some people there actually develop software? We want to invite you to bring your whole team, and show off how agile you are. Set up your team information radiators, pair programming stations and servers, and spend part of the conference actually developing your product! With any luck, some of the excellent agilists at the conference will come and pair with you.
The only rules are that you must be agile, and you must deploy into production during the conference. If you deploy several times a day and the product is generally available for conference delegates to use and try the new features, all the better. :-)
See this as a challenge for your team. If you’re going to be able to show your process off to a whole conference of agilists, wouldn’t you like to have the best possible process? Wouldn’t you like all your team members to work effectively with the latest tools and techniques? Wouldn’t the challenge of having to be ready to show off what you can do at XP2012 be a great way to motivate your team to improve over the next 6 months?
This is the challenge that the team behind http://blocket.se (one of Sweden’s biggest ecommerce sites), and a team from SAAB Gripen, (software for jet fighters), have already accepted. Will you join them, and take up the XP2012 team challenge?
The only rules are that you must be agile, and you must deploy into production during the conference. If you deploy several times a day and the product is generally available for conference delegates to use and try the new features, all the better. :-)
See this as a challenge for your team. If you’re going to be able to show your process off to a whole conference of agilists, wouldn’t you like to have the best possible process? Wouldn’t you like all your team members to work effectively with the latest tools and techniques? Wouldn’t the challenge of having to be ready to show off what you can do at XP2012 be a great way to motivate your team to improve over the next 6 months?
This is the challenge that the team behind http://blocket.se (one of Sweden’s biggest ecommerce sites), and a team from SAAB Gripen, (software for jet fighters), have already accepted. Will you join them, and take up the XP2012 team challenge?
Tech Demos at XP2012
At XP2011 we introduced a new kind of presentation - the tech demo. The idea was to give people 30 minutes to demonstrate a new tool or technique. For example, some people performed code katas in diverse languages, and others showed various productivity-boosting frameworks.
For XP2012 we want to continue with these kinds of demos, but with an additional rule. You can’t touch the keyboard yourself when you present. We want you to co-present with someone else, drawn from the audience, who will do the typing and demonstrate the tool or technique. Your job is to coach them into doing the demo you've planned, and to explain to everyone what’s going on.
Your co-presenter could be someone drawn from the audience who you’ve never met before. You’ll have to expertly coach them into demonstrating what you aim to show to the rest of the audience. If you choose this route, it will certainly be a big test of your skills of coaching and pedagogy.
Alternatively you can come to the conference and find a volunteer in advance of the presentation. You could make time for a practice run or two before the presentation. Again, you’ll be showing not only your tool or techique, but also how easy it is for a good programmer to learn.
If you’re thinking this sounds like an awfly scary way to do a demo, then you may be right. We think it’s also a very good way to have demos that engage the audience and really show off what you’re capable of.
If you know someone else who’ll be at the conference you could of course prepare the demo with them well in advance. You’d be able to work out exactly what pitfalls they will fall into, and have a slick commentary ready. Just having one person typing and the other talking is a great advantage in a demo, and this would probably still be good to watch. It might not be quite as challenging though :-)
Will you accept the XP2012 Tech Demo challenge?
For XP2012 we want to continue with these kinds of demos, but with an additional rule. You can’t touch the keyboard yourself when you present. We want you to co-present with someone else, drawn from the audience, who will do the typing and demonstrate the tool or technique. Your job is to coach them into doing the demo you've planned, and to explain to everyone what’s going on.
Your co-presenter could be someone drawn from the audience who you’ve never met before. You’ll have to expertly coach them into demonstrating what you aim to show to the rest of the audience. If you choose this route, it will certainly be a big test of your skills of coaching and pedagogy.
Alternatively you can come to the conference and find a volunteer in advance of the presentation. You could make time for a practice run or two before the presentation. Again, you’ll be showing not only your tool or techique, but also how easy it is for a good programmer to learn.
If you’re thinking this sounds like an awfly scary way to do a demo, then you may be right. We think it’s also a very good way to have demos that engage the audience and really show off what you’re capable of.
If you know someone else who’ll be at the conference you could of course prepare the demo with them well in advance. You’d be able to work out exactly what pitfalls they will fall into, and have a slick commentary ready. Just having one person typing and the other talking is a great advantage in a demo, and this would probably still be good to watch. It might not be quite as challenging though :-)
Will you accept the XP2012 Tech Demo challenge?
Presenting a session at XP2012
The Call for Papers for XP2012 is currently open, and this year we’re doing things a little differently*. I'm one of the co-chairs responsible for programme design, and also involved in reviewing session proposals. (You might be interested in my post about XP2011, which I was also involved in organizing)
Most proposals are going through what will hopefully be a more transparent, agile and effective review process than we’ve had in the past. The idea is that everyone sends in a first draft proposal, and then receives feedback from reviewers who want to help them to improve and refine their ideas. When everyone has had a chance to act on this feedback, the review committee will select the proposals that will be put on the programme for the conference.
More information, dates and benefits of speaking are listed on the conference website. My next two posts contain more information about a couple of the session types we're looking for: Tech Demos
and the Team Challenge.
*Academic papers have a separate review track, and proceed much as they have done previously. The demands of academic rigour and peer review mean we won’t change a formula that clearly works for this kind of submission. See the call for research papers.
Most proposals are going through what will hopefully be a more transparent, agile and effective review process than we’ve had in the past. The idea is that everyone sends in a first draft proposal, and then receives feedback from reviewers who want to help them to improve and refine their ideas. When everyone has had a chance to act on this feedback, the review committee will select the proposals that will be put on the programme for the conference.
More information, dates and benefits of speaking are listed on the conference website. My next two posts contain more information about a couple of the session types we're looking for: Tech Demos
and the Team Challenge.
*Academic papers have a separate review track, and proceed much as they have done previously. The demands of academic rigour and peer review mean we won’t change a formula that clearly works for this kind of submission. See the call for research papers.
Thursday, 10 November 2011
Öredev experiences on my first day here
I've never been to Öredev before, and it really is a very impressive conference. Gathered in Malmö right now, is over a thousand developers, with a collection of speakers representing the elite of the global software development industry. The sessions are overflowing with a plethora of great advice, news and inspiration. That's primarily why I'm here, although I could also praise the excellent conference organization, food, live music, sponsor giveaways and other entertainments.
I'd like to talk about my first impressions of the conference. As with other software conferences I've been too, women are rather unusual here, both among delegates and speakers. I've blogged about this before, and it's not untypical. The sad fact is that the proportion of women is low, and, unlike in other industries, actually falling in software development.
I'd just like to relate two experiences I had yesterday.
Women speakers
I'm here as an ordinary delegate at this conference, but at most conferences I go to, I'm a speaker. I was really happy to be greeted yesterday by Dan North, Gojko Adzic, Pat Kua, Corey Haines and others, since they're people who I really respect. I've mostly got to know them when we've spoken at the same conferences in the past. Of course they all asked me which day I was speaking on at Öredev. Well, the easy answer is that I was not asked to speak here. I looked on the Öredev conference website and there was no call for submissions, unlike other conferences I have spoken at like XP, Agile Testing Days, ScanDev, Agile, JFokus etc. I assumed that it was invitation only, and the track chairs would mail me if they were interested in having me speak. Now I realize that I should have mailed them to point out I wanted to speak here.
I think I just fell into this trap that women apparently often fall into, described in this article I read this week: "Four Ways Women Stunt Their Careers Unintentionally". Apparently we tend to be "overly modest" and "women fail to get promoted because they fail to step up and apply". So I didn't apply, and I didn't get a job to speak at Öredev. #kickingmyself
Neal Ford's keynote
I listened to a keynote by Neal Ford yesterday. I've heard him speak before, and he is always very entertaining and yet makes some interesting points about software. It's just a small thing he said that really bothered me. A keynote speech is supposed to set the tone for a whole conference - you have the everybody gathered, and you're supposed to say something inspirational and thought provoking.
One of Neal's jokes was about some obscure Star Trek reference that I didn't get (although from the audience reaction I'm guessing most others did), that he followed up with a slide showing the top Google image results when you searched for this thing. He made some comment about google knowing that when you search for this, you're really asking for porn. He had helpfully airbrushed out the image results so you could only vaguely see the outline of naked women.
Neal, you really didn't need to do that. Your talk had enough fun stuff in it without alienating me with science fiction references and disguised porn.
Looking forward
There are two more days of the main conference, and plenty more good stuff on the programme. I've spotted a session called "geek feminism" in the "extra" track. I've never thought of myself as a geek feminist, but maybe I am. Having had the experiences described above, I think I'll go along and find out.
I'd like to talk about my first impressions of the conference. As with other software conferences I've been too, women are rather unusual here, both among delegates and speakers. I've blogged about this before, and it's not untypical. The sad fact is that the proportion of women is low, and, unlike in other industries, actually falling in software development.
I'd just like to relate two experiences I had yesterday.
Women speakers
I'm here as an ordinary delegate at this conference, but at most conferences I go to, I'm a speaker. I was really happy to be greeted yesterday by Dan North, Gojko Adzic, Pat Kua, Corey Haines and others, since they're people who I really respect. I've mostly got to know them when we've spoken at the same conferences in the past. Of course they all asked me which day I was speaking on at Öredev. Well, the easy answer is that I was not asked to speak here. I looked on the Öredev conference website and there was no call for submissions, unlike other conferences I have spoken at like XP, Agile Testing Days, ScanDev, Agile, JFokus etc. I assumed that it was invitation only, and the track chairs would mail me if they were interested in having me speak. Now I realize that I should have mailed them to point out I wanted to speak here.
I think I just fell into this trap that women apparently often fall into, described in this article I read this week: "Four Ways Women Stunt Their Careers Unintentionally". Apparently we tend to be "overly modest" and "women fail to get promoted because they fail to step up and apply". So I didn't apply, and I didn't get a job to speak at Öredev. #kickingmyself
Neal Ford's keynote
I listened to a keynote by Neal Ford yesterday. I've heard him speak before, and he is always very entertaining and yet makes some interesting points about software. It's just a small thing he said that really bothered me. A keynote speech is supposed to set the tone for a whole conference - you have the everybody gathered, and you're supposed to say something inspirational and thought provoking.
One of Neal's jokes was about some obscure Star Trek reference that I didn't get (although from the audience reaction I'm guessing most others did), that he followed up with a slide showing the top Google image results when you searched for this thing. He made some comment about google knowing that when you search for this, you're really asking for porn. He had helpfully airbrushed out the image results so you could only vaguely see the outline of naked women.
Neal, you really didn't need to do that. Your talk had enough fun stuff in it without alienating me with science fiction references and disguised porn.
Looking forward
There are two more days of the main conference, and plenty more good stuff on the programme. I've spotted a session called "geek feminism" in the "extra" track. I've never thought of myself as a geek feminist, but maybe I am. Having had the experiences described above, I think I'll go along and find out.
Wednesday, 9 November 2011
What is Text-Based Testing?
Programmers have a vested interest in making sure the software they
create does what they think it does. When I'm coding I prefer to work in
the context of feedback from automated tests, that help me to keep
track of what works and how far I've got. I've written before about Test Driven Development, (TDD). In this article I'd like to explain some of
the main features of Text-Based Testing. It's a variant on TDD, perhaps
more suited to the functional level than unit tests, and which I've
found powerful and productive to use.
The basic idea
You get your program to produce a plain text file that documents all the important things that it does. A log, if you will. You run the program and store this text as a "golden copy" of the output. You create from this a Text-Based Test with a descriptive name, any inputs you gave to the program, and the golden copy of the textual output.
You make some changes to your program, and you run it again, gathering the new text produced. You compare the text with the golden copy, and if they are identical, the test passes. If the there is a difference, the test fails. If you look at the diff, and you like the new text better than the old text, you update your golden copy, and the test is passing once again.
Tool Support
Text-Based Testing is a simple idea, and in fact many people do it already in their unit tests. AssertEquals(String expected, String actual) is actually a form of it. You often create the "expected" string based on the actual output of the program, (although purists will write the whole assert before they execute the test).
Most unit test tools these days give you a nice diff even on multi-line strings. For example:
Which is a failing text-based test using JUnit.
Once your strings get very long, to the scale of whole log files, even multi-line diffs aren't really enough. You get datestamps, process ids and other stuff that changes every run, hashmaps with indeterminate order, etc. It gets tedious to deal with all this on a test-by-test basis.
My husband, Geoff Bache, has created a tool called "TextTest" to support Text-Based testing. Amongst other things, it helps you organize and run your text-based tests, and filter the text before you compare it. It's free, open source, and of course used to test itself. (Eats own dog food!) TextTest is used extensively within Jeppesen Systems, (Geoff works for them, and they support development), and I've used it too on various projects in other organizations.
In the rest of this article I'll look at some of the main implications of using a Text-Based Testing approach, and some of my experiences.
Little code per test
The biggest advantage of the approach, is that you tend to write very little unique code for each test. You generally access the application through a public interface as a user would, often a command line interface or (web)service call. You then create many tests by for example varying the command line options or request contents. This reduces test maintenance work, since you have less test code to worry about, and the public API of your program should change relatively infrequently.
Legacy code
Text-Based Testing is obviously a regression testing technique. You're checking the code still does what it did before, by checking the log is the same. So these tests are perfect for refactoring. As you move around the code, the log statements move too, and your tests stay green, (so long as you don't make any mistakes!) In most systems, it's cheap and risk-free to add log statements, no matter how horribly gnarly the design is. So text-based testing is an easy way to get some initial tests in place to lean on while refactoring. I've used it this way fairly successfully to get legacy code under control, particularly if the code already produces a meaningful log or textual output.
No help with your design
I just told you how good Text-Based Testing is with Legacy code. But actually these tests give you very little help with the internal design of your program. With normal TDD, the activity of creating unit tests at least forces you to decompose your design into units, and if you do it well, you'll find these tests giving you all sorts of feedback about your design. Text-Based tests don't. Log statements don't care if they're in the middle of a long horrible method or if they're spread around several smaller ones. So you have to get feedback on your design some other way.
I usually work with TDD at the unit level in combination with Text-Based tests at the functional level. I think it gives me the best of both worlds.
Log statements and readability
Some people complain that log statements reduce the readability of their code and don't like to add any at all. They seem to be out of fashion, just like comments. The idea is that all the important ideas should be expressed in the class and method names, and logs and comments just clutter up the important stuff. I agree to an extent, you can definitely over-use logs and comments. I think a few well placed ones can make all the difference though. For Text-Based Testing purposes, you don't want a log that is megabytes and megabytes of junk, listing every time you enter and leave every method, and the values of every variable. That's going to seriously hinder your refactoring, apart from being a nightmare to store and update.
What we're talking about here is targeted log statements at the points when something important happens, that we want to make sure should continue happening. You can think about it like the asserts you make in unit tests. You don't assert everything, just what's important. In my experience less than two percent of the lines of code end up being log statements, and if anything, they increase readability.
Text-Based tests are completed after the code
In normal TDD you write the test first, and thereby set up a mini pull system for the functionality you need. It's lean, it forces you to focus on the problem you're trying to solve before you solve it, and starts giving you feedback before you commit to an implementation. With Text-Based Testing, you often find it's too much work the specify the log up front. It's much easier to wait until you've implemented the feature, run the test, and save the log afterwards.
So your tests usually aren't completed until after the code they test, unlike in normal TDD. Having said that, I would argue that you can still do a form of TDD with Text-Based Tests. I'd normally create the half the test before the code. I name the test, and find suitable inputs that should provoke the behaviour I need to implement in the system. The test will fail the first time I run it. In this way I think I get many of the benefits of TDD, but only actually pin down the exact assertion once the functionality is working.
"Expert Reads Output" Antipattern
If you're relying on a diff in the logs to tell you when your program is broken, you had better have good logs! But who decides what to log? Who checks the "golden copy"? Usually it is the person creating the test, who should look through the log and check everything is in order the first time. Of course, after a test is created, every time it fails you have to make a decision whether to update the golden copy of the log. You might make a mistake. There's a well known antipattern called "Expert Reads Output" which basically says that you shouldn't rely on having someone check the results of your tests by eye.
This is actually a problem with any automated testing approach - someone has to make a judgement about what to do when a test fails - whether the test is wrong or there's a bug in the application. With Text-Based Testing you might have a larger quantity of text to read through compared with other approaches, or maybe not. If you have human-readable, concise, targeted log statements and good tools for working with them, it goes a long way. You need a good diff tool, version control, and some way of grouping similar changes. It's also useful to have some sanity checks. For example TextTest can easily search for regular expressions in the log and warn you if you try to save a golden copy containing a stack trace for example.
In my experience, you do need to update the golden copy quite often. I think this is one of the key skills with a Text-Based Testing approach. You have to learn to write good logs, and to be disciplined about either doing refactoring or adding functionality, not both at the same time. If you're refactoring and the logs change, you need to be able to quickly recognize if it's ok, or if you made a mistake. Similarly, if you add new functionality and no logs change, that could be a problem.
Agile Tests Manage Behaviour
When you create a unit test, you end with an Assert statement. This is supposed to be some kind of universal truth that should always be valid, or else there is a big problem. Particularly for functional level tests, it can be hard to find these kinds of invariants. What is correct today might be updated next week when the market moves or the product owner changes their mind. With Text-Based Testing you have an opportunity to quickly and easily update the golden copy every time the test "fails". This makes your tests much more about keeping control of what your app does over time, and less about rewriting assert statements.
Text-Based Testing grew up in the domain of optimizing logistics planning. In this domain there is no "correct" answer you can predict in advance and assert. Planning problems that are interesting to solve are far too complex for a complete mathematical analysis, and the code relies on heuristics and fancy algorithms to come up with better and better solutions. So Text-Based Testing makes it easy to spot when the test produces a different plan from before, and use it as the new baseline if it's an improvement.
I think generally it leads to more "agile" tests. They can easily respond to changes in the business requirements.
Conclusions
There is undoubtedly a lot more to be said about Text-Based Testing. I havn't mentioned text-based mocking, data-driven vs workflow testing, or how to handle databases and GUIs - all relevant topics. I hope this article has given you a flavour of how it's different from ordinary TDD, though. I've found that good tool support is pretty essential to making Text-Based Testing work well, and that it's a particularly good technique for handling legacy code, although not exclusively. I like the approach because it minimizes the amount of code per test, and makes it easy to keep the tests in sync with the current behaviour of the system.
The basic idea
You get your program to produce a plain text file that documents all the important things that it does. A log, if you will. You run the program and store this text as a "golden copy" of the output. You create from this a Text-Based Test with a descriptive name, any inputs you gave to the program, and the golden copy of the textual output.
You make some changes to your program, and you run it again, gathering the new text produced. You compare the text with the golden copy, and if they are identical, the test passes. If the there is a difference, the test fails. If you look at the diff, and you like the new text better than the old text, you update your golden copy, and the test is passing once again.
Tool Support
Text-Based Testing is a simple idea, and in fact many people do it already in their unit tests. AssertEquals(String expected, String actual) is actually a form of it. You often create the "expected" string based on the actual output of the program, (although purists will write the whole assert before they execute the test).
Most unit test tools these days give you a nice diff even on multi-line strings. For example:
Which is a failing text-based test using JUnit.
Once your strings get very long, to the scale of whole log files, even multi-line diffs aren't really enough. You get datestamps, process ids and other stuff that changes every run, hashmaps with indeterminate order, etc. It gets tedious to deal with all this on a test-by-test basis.
My husband, Geoff Bache, has created a tool called "TextTest" to support Text-Based testing. Amongst other things, it helps you organize and run your text-based tests, and filter the text before you compare it. It's free, open source, and of course used to test itself. (Eats own dog food!) TextTest is used extensively within Jeppesen Systems, (Geoff works for them, and they support development), and I've used it too on various projects in other organizations.
In the rest of this article I'll look at some of the main implications of using a Text-Based Testing approach, and some of my experiences.
Little code per test
The biggest advantage of the approach, is that you tend to write very little unique code for each test. You generally access the application through a public interface as a user would, often a command line interface or (web)service call. You then create many tests by for example varying the command line options or request contents. This reduces test maintenance work, since you have less test code to worry about, and the public API of your program should change relatively infrequently.
Legacy code
Text-Based Testing is obviously a regression testing technique. You're checking the code still does what it did before, by checking the log is the same. So these tests are perfect for refactoring. As you move around the code, the log statements move too, and your tests stay green, (so long as you don't make any mistakes!) In most systems, it's cheap and risk-free to add log statements, no matter how horribly gnarly the design is. So text-based testing is an easy way to get some initial tests in place to lean on while refactoring. I've used it this way fairly successfully to get legacy code under control, particularly if the code already produces a meaningful log or textual output.
No help with your design
I just told you how good Text-Based Testing is with Legacy code. But actually these tests give you very little help with the internal design of your program. With normal TDD, the activity of creating unit tests at least forces you to decompose your design into units, and if you do it well, you'll find these tests giving you all sorts of feedback about your design. Text-Based tests don't. Log statements don't care if they're in the middle of a long horrible method or if they're spread around several smaller ones. So you have to get feedback on your design some other way.
I usually work with TDD at the unit level in combination with Text-Based tests at the functional level. I think it gives me the best of both worlds.
Log statements and readability
Some people complain that log statements reduce the readability of their code and don't like to add any at all. They seem to be out of fashion, just like comments. The idea is that all the important ideas should be expressed in the class and method names, and logs and comments just clutter up the important stuff. I agree to an extent, you can definitely over-use logs and comments. I think a few well placed ones can make all the difference though. For Text-Based Testing purposes, you don't want a log that is megabytes and megabytes of junk, listing every time you enter and leave every method, and the values of every variable. That's going to seriously hinder your refactoring, apart from being a nightmare to store and update.
What we're talking about here is targeted log statements at the points when something important happens, that we want to make sure should continue happening. You can think about it like the asserts you make in unit tests. You don't assert everything, just what's important. In my experience less than two percent of the lines of code end up being log statements, and if anything, they increase readability.
Text-Based tests are completed after the code
In normal TDD you write the test first, and thereby set up a mini pull system for the functionality you need. It's lean, it forces you to focus on the problem you're trying to solve before you solve it, and starts giving you feedback before you commit to an implementation. With Text-Based Testing, you often find it's too much work the specify the log up front. It's much easier to wait until you've implemented the feature, run the test, and save the log afterwards.
So your tests usually aren't completed until after the code they test, unlike in normal TDD. Having said that, I would argue that you can still do a form of TDD with Text-Based Tests. I'd normally create the half the test before the code. I name the test, and find suitable inputs that should provoke the behaviour I need to implement in the system. The test will fail the first time I run it. In this way I think I get many of the benefits of TDD, but only actually pin down the exact assertion once the functionality is working.
"Expert Reads Output" Antipattern
If you're relying on a diff in the logs to tell you when your program is broken, you had better have good logs! But who decides what to log? Who checks the "golden copy"? Usually it is the person creating the test, who should look through the log and check everything is in order the first time. Of course, after a test is created, every time it fails you have to make a decision whether to update the golden copy of the log. You might make a mistake. There's a well known antipattern called "Expert Reads Output" which basically says that you shouldn't rely on having someone check the results of your tests by eye.
This is actually a problem with any automated testing approach - someone has to make a judgement about what to do when a test fails - whether the test is wrong or there's a bug in the application. With Text-Based Testing you might have a larger quantity of text to read through compared with other approaches, or maybe not. If you have human-readable, concise, targeted log statements and good tools for working with them, it goes a long way. You need a good diff tool, version control, and some way of grouping similar changes. It's also useful to have some sanity checks. For example TextTest can easily search for regular expressions in the log and warn you if you try to save a golden copy containing a stack trace for example.
In my experience, you do need to update the golden copy quite often. I think this is one of the key skills with a Text-Based Testing approach. You have to learn to write good logs, and to be disciplined about either doing refactoring or adding functionality, not both at the same time. If you're refactoring and the logs change, you need to be able to quickly recognize if it's ok, or if you made a mistake. Similarly, if you add new functionality and no logs change, that could be a problem.
Agile Tests Manage Behaviour
When you create a unit test, you end with an Assert statement. This is supposed to be some kind of universal truth that should always be valid, or else there is a big problem. Particularly for functional level tests, it can be hard to find these kinds of invariants. What is correct today might be updated next week when the market moves or the product owner changes their mind. With Text-Based Testing you have an opportunity to quickly and easily update the golden copy every time the test "fails". This makes your tests much more about keeping control of what your app does over time, and less about rewriting assert statements.
Text-Based Testing grew up in the domain of optimizing logistics planning. In this domain there is no "correct" answer you can predict in advance and assert. Planning problems that are interesting to solve are far too complex for a complete mathematical analysis, and the code relies on heuristics and fancy algorithms to come up with better and better solutions. So Text-Based Testing makes it easy to spot when the test produces a different plan from before, and use it as the new baseline if it's an improvement.
I think generally it leads to more "agile" tests. They can easily respond to changes in the business requirements.
Conclusions
There is undoubtedly a lot more to be said about Text-Based Testing. I havn't mentioned text-based mocking, data-driven vs workflow testing, or how to handle databases and GUIs - all relevant topics. I hope this article has given you a flavour of how it's different from ordinary TDD, though. I've found that good tool support is pretty essential to making Text-Based Testing work well, and that it's a particularly good technique for handling legacy code, although not exclusively. I like the approach because it minimizes the amount of code per test, and makes it easy to keep the tests in sync with the current behaviour of the system.
Subscribe to:
Posts (Atom)