Showing posts with label BDD. Show all posts
Showing posts with label BDD. Show all posts
Friday, 15 March 2013
Principles for Agile Test Automation (2nd Edition)
This post has moved to http://coding-is-like-cooking.info/2013/03/principles-for-agile-test-automation-2nd-edition/
Tuesday, 22 January 2013
Monday, 6 August 2012
Principles of Agile Test Automation
Please note - As of March 2013, I have rewritten this post in the light of further experience and discussions. The updated post is available here.
I feel like I've spent most of my career learning how to write good automated tests in an agile environment. When I downloaded JUnit in the year 2000 it didn't take long before I was hooked - unit tests for everything in sight. That gratifying green bar is near-instant feedback that everthing is as expected, my code does what I intended, and I can continue developing from a firm foundation.
Later, starting in about 2002, I began writing larger granularity tests, for whole subsystems; functional tests if you like. The feedback that my code does what I intended, and that it has working functionality has given me confidence time and again to release updated versions to end-users.
Often, I've written functional tests as regression tests, after the functionality is supposed to work. In other situations, I've been able to write these kinds of tests in advance, as part of an ATDD, or BDD process. In either case, I've found the regression tests you end up with need to have certain properties if they're going to be useful in an agile environment moving forward. I think the same properties are needed for good agile functional tests as for good unit tests, but it's much harder. Your mistakes are amplified as the scope of the test increases.
I'd like to outline four principles of agile test automation that I've derived from my experience.
If your tests have poor coverage, they will continue to pass even when your system is broken and functionality unusable. This can happen if you have missed out needed test cases, or when your test cases don't check properly what the system actually did. The consequences of poor coverage is that you can't refactor with confidence, and need to do additional (manual) testing before release.
The aim for automated regression tests is good Coverage: If you break something important and no tests fail, your test coverage is not good enough. All the other principles are in tension with this one - improving Coverage will often impair the others.
If your test case is not readable, it will not be useful. When it fails you will have to dig though other sources outside of the test case to find out what is wrong. Quite likely you will not understand what is wrong and you will rewrite the test to check for something else, or simply delete it.
As you improve Coverage, you will likely add more and more test cases. Each one may be fairly readable on its own, but taken all together it can become hard to navigate and get an overview.
Aspects of Robustness you often run into are tests that are not isolated from one another, duplication between test cases, and flickering tests. If you run a test by itself and it passes, but fails in a suite together with other tests, then you have an isolation problem. If you have one broken feature and it causes a large number of test failures, you have duplication between test cases. If you have a test that fails in one test run, then passes in the next when nothing changed, you have a flickering test.
If your tests often fail for no good reason, you will start to ignore them. Quite likely there will be real failures hiding amongst all the false ones, and the danger is you will not see them.
As you improve Coverage you'll want to add more checks for details of your system. This will give your tests more and more reasons to fail.
If your test suite is slow, it will not be used. When you're feeling stressed, you'll skip running them, and problem code will enter the system. In the worst case the test suite will never become green. You'll fix the one or two problems in a given run and kick off a new test run, but in the meantime someone else has checked in other changes, and the new run is not green either. You're developing all the while the tests are running, and they never quite catch up. This can become pretty demoralizing.
As you improve Coverage, you add more test cases, and this will naturally increase the execution time for the whole test suite.
I also find these principles useful when I'm trying to diagnose why a test suite is not being useful to a development team, especially if things have got so bad they have stopped maintaining it. I can often identify which principle(s) the team has missed, and advise how to refactor the test suite to compensate.
For example, if the problem is lack of Speed you have some options and tradeoffs to make:
I hope you will find these principles will help you to reason about the automated tests in your suite.
Images Attribution: DaPino Webdesign, Lebreton, Asher Abbasi, Woothemes, Iconshock, Andy Gongea, from www.iconspedia.com
I feel like I've spent most of my career learning how to write good automated tests in an agile environment. When I downloaded JUnit in the year 2000 it didn't take long before I was hooked - unit tests for everything in sight. That gratifying green bar is near-instant feedback that everthing is as expected, my code does what I intended, and I can continue developing from a firm foundation.
Later, starting in about 2002, I began writing larger granularity tests, for whole subsystems; functional tests if you like. The feedback that my code does what I intended, and that it has working functionality has given me confidence time and again to release updated versions to end-users.
Often, I've written functional tests as regression tests, after the functionality is supposed to work. In other situations, I've been able to write these kinds of tests in advance, as part of an ATDD, or BDD process. In either case, I've found the regression tests you end up with need to have certain properties if they're going to be useful in an agile environment moving forward. I think the same properties are needed for good agile functional tests as for good unit tests, but it's much harder. Your mistakes are amplified as the scope of the test increases.
I'd like to outline four principles of agile test automation that I've derived from my experience.
Coverage
If you have a test for a feature, and there is a bug in that feature, the test should fail. Note I'm talking about coverage of functionality, not code coverage, although these concepts are related. If your code coverage is poor, your functionality coverage is likely also to be poor.If your tests have poor coverage, they will continue to pass even when your system is broken and functionality unusable. This can happen if you have missed out needed test cases, or when your test cases don't check properly what the system actually did. The consequences of poor coverage is that you can't refactor with confidence, and need to do additional (manual) testing before release.
The aim for automated regression tests is good Coverage: If you break something important and no tests fail, your test coverage is not good enough. All the other principles are in tension with this one - improving Coverage will often impair the others.
Readability
When you look at the test case, you can read it through and understand what the test is for. You can see what the expected behaviour is, and what aspects of it are covered by the test. When the test fails, you can quickly see what is broken.If your test case is not readable, it will not be useful. When it fails you will have to dig though other sources outside of the test case to find out what is wrong. Quite likely you will not understand what is wrong and you will rewrite the test to check for something else, or simply delete it.
As you improve Coverage, you will likely add more and more test cases. Each one may be fairly readable on its own, but taken all together it can become hard to navigate and get an overview.
Robustness
When a test fails, it means the functionality it tests is broken, or at least is behaving significantly differently from before. You need to take action to correct the system or update the test to account for the new behaviour. Fragile tests are the opposite of Robust: they fail often for no good reason.Aspects of Robustness you often run into are tests that are not isolated from one another, duplication between test cases, and flickering tests. If you run a test by itself and it passes, but fails in a suite together with other tests, then you have an isolation problem. If you have one broken feature and it causes a large number of test failures, you have duplication between test cases. If you have a test that fails in one test run, then passes in the next when nothing changed, you have a flickering test.
If your tests often fail for no good reason, you will start to ignore them. Quite likely there will be real failures hiding amongst all the false ones, and the danger is you will not see them.
As you improve Coverage you'll want to add more checks for details of your system. This will give your tests more and more reasons to fail.
Speed
As an agile developer you run the tests frequently. Both (a) every time you build the system, and (b) before you check in changes. I recommend time limits of 2 minutes for (a) and 10 minutes for (b). This fast feedback gives you the best chance of actually being willing to run the tests, and to find defects when they're cheapest to fix.If your test suite is slow, it will not be used. When you're feeling stressed, you'll skip running them, and problem code will enter the system. In the worst case the test suite will never become green. You'll fix the one or two problems in a given run and kick off a new test run, but in the meantime someone else has checked in other changes, and the new run is not green either. You're developing all the while the tests are running, and they never quite catch up. This can become pretty demoralizing.
As you improve Coverage, you add more test cases, and this will naturally increase the execution time for the whole test suite.
How are these principles useful?
I find it useful to remember these principles when designing test cases. I may need to make tradeoffs between them, and it helps just to step back and assess how I'm doing on each principle from time to time as I develop.I also find these principles useful when I'm trying to diagnose why a test suite is not being useful to a development team, especially if things have got so bad they have stopped maintaining it. I can often identify which principle(s) the team has missed, and advise how to refactor the test suite to compensate.
For example, if the problem is lack of Speed you have some options and tradeoffs to make:
- Invest in hardware and run tests in parallel (costs $)
- Use a profiler to optimize the tests for speed the same as you would production code (may affect Readability)
- push down tests to a lower level of granularity where they can execute faster. (may reduce Coverage and/or increase Readability)
- Identify key test cases for essential functionality and remove the other test cases. (sacrifice Coverage to get Speed)
I hope you will find these principles will help you to reason about the automated tests in your suite.
Images Attribution: DaPino Webdesign, Lebreton, Asher Abbasi, Woothemes, Iconshock, Andy Gongea, from www.iconspedia.com
Wednesday, 14 January 2009
Behaviour Driven Development at agile2008
At agile2008 I attended a session with Dan North about Behaviour Driven Development. Someone on the agile sweden mailing list was asking about it, so I decided to write up my notes here.
Most cellphone and computer software is delivered late and over budget. The biggest contributing factor to cost bloat is building the wrong thing. So what software and business people need is "a shared understanding of what done looks like".
Test Driven Development is about design, conversations, and writing examples for a system that doesn't yet exist. It's not really about testing. However, once the system exists, your examples turn into tests, as a rather useful side effect.
A User Story is a promise of a conversation, and it is in that conversation that things go wrong. The customer and developer rarely agree what "enough" and "done" look like, which leads to over- or under- engineering.
Dan suggests a format for User Story cards which aims to prevent this communication gap.
On the front of the User Story index card, you have the title and narrative. The narrative consists of a sentence in this format:
As astakeholder
I want feature
so that benefit
where benefit is something of value to stakeholder.
On the back of the card, you have a table with three columns
Given this context | When I do this | then this happens
Then you have 4 or 5 rows in the table, each detailing a scenario. (If you need more than that then the story is too big and should be split)
Dan finds that in his work, this leads to conversations about User Stories where "done" and "enough" are discussed, and defined.
User Stories should be about activities, not features. In order to check that your User Story is an activity, you should be able to do a thought experiment where you implement the story as a task to be performed by people on rollerblades with paper. You must think about it as a business process, not a piece of software.
When creating the story cards, the whole team should be involved, but it is primarily the business/end user stakeholders and business analysts who write the title and narrative on the cards. They then take in a tester to help them to write the scenarios.
Are people familiar with the V model of software testing? When this was conceived, they thought that the whole process would take 2 years, and span the whole project. Dan ususally does it in 2 days. Many times for each project.
Then Dan offered to show us how to do BDD using plain JUnit. He requested a pair from the audience, so I volunteered. At this point my notes dry up, and I am working from memory, but I think the general idea is like this.
You talk about "behaviour specs" not tests. The words you use influence the way you think, and "behaviour specification" gives much better associations than "tests".
Each behaviour specification should be named to indicate the behavour it is specifying. Not "testCustomerAccountEmpty" rather "customerAccountShouldBeEmpty".
In the body of the spec, you can start out by typing in the prose of one of the scenarios you have on the user story, as a comment.
//given we have a flimble containing a schmooz
// when we request the next available frooble
// then we are given a half baked frooble and the schmooz.
Then you can fill in code after the "given" comment. When you have code that does what the comment says, delete the comment. Repeat with the "when" and "then" comments.
In this way, you build up a behaviour specification that drives your development of the system. A few minutes later (hopefully) you have a system which implements the specification, and at that point your spec helpfully turns magically into a regression test which you can run. At that point you can start calling it a test if you like. But actually it is more helpful to your brain to continue to think of it as a behaviour specification. It leads to much more constructive conversations about the system.
Most cellphone and computer software is delivered late and over budget. The biggest contributing factor to cost bloat is building the wrong thing. So what software and business people need is "a shared understanding of what done looks like".
Test Driven Development is about design, conversations, and writing examples for a system that doesn't yet exist. It's not really about testing. However, once the system exists, your examples turn into tests, as a rather useful side effect.
A User Story is a promise of a conversation, and it is in that conversation that things go wrong. The customer and developer rarely agree what "enough" and "done" look like, which leads to over- or under- engineering.
Dan suggests a format for User Story cards which aims to prevent this communication gap.
On the front of the User Story index card, you have the title and narrative. The narrative consists of a sentence in this format:
As a
I want feature
so that benefit
where benefit
On the back of the card, you have a table with three columns
Given this context | When I do this | then this happens
Then you have 4 or 5 rows in the table, each detailing a scenario. (If you need more than that then the story is too big and should be split)
Dan finds that in his work, this leads to conversations about User Stories where "done" and "enough" are discussed, and defined.
User Stories should be about activities, not features. In order to check that your User Story is an activity, you should be able to do a thought experiment where you implement the story as a task to be performed by people on rollerblades with paper. You must think about it as a business process, not a piece of software.
When creating the story cards, the whole team should be involved, but it is primarily the business/end user stakeholders and business analysts who write the title and narrative on the cards. They then take in a tester to help them to write the scenarios.
Are people familiar with the V model of software testing? When this was conceived, they thought that the whole process would take 2 years, and span the whole project. Dan ususally does it in 2 days. Many times for each project.
Then Dan offered to show us how to do BDD using plain JUnit. He requested a pair from the audience, so I volunteered. At this point my notes dry up, and I am working from memory, but I think the general idea is like this.
You talk about "behaviour specs" not tests. The words you use influence the way you think, and "behaviour specification" gives much better associations than "tests".
Each behaviour specification should be named to indicate the behavour it is specifying. Not "testCustomerAccountEmpty" rather "customerAccountShouldBeEmpty".
In the body of the spec, you can start out by typing in the prose of one of the scenarios you have on the user story, as a comment.
//given we have a flimble containing a schmooz
// when we request the next available frooble
// then we are given a half baked frooble and the schmooz.
Then you can fill in code after the "given" comment. When you have code that does what the comment says, delete the comment. Repeat with the "when" and "then" comments.
In this way, you build up a behaviour specification that drives your development of the system. A few minutes later (hopefully) you have a system which implements the specification, and at that point your spec helpfully turns magically into a regression test which you can run. At that point you can start calling it a test if you like. But actually it is more helpful to your brain to continue to think of it as a behaviour specification. It leads to much more constructive conversations about the system.
Subscribe to:
Posts (Atom)