Failing with Acceptance Testing
I’ve had this one sat in my FireFox tabs for months and only just got around to reading it. Gojko Adzic lists the outcome of an openspace session at CITCON Eurpose: http://gojko.net/2009/09/24/top-10-reasons-why-teams-fail-with-acceptance-testing/
Most of the reasons rung true with me too, especially at the start. It’s been two years now since we adopted FIT as our acceptance test tool. We’ve improved greatly since then, but we still have challenges.
- No Collaboration
Initially we had developers writing tests and proving the usefulness of the approach. Unfortunately, the same developers were then tasks with writing a battery of tests against the existing buggy code. It was a between-project slack period and there wasn’t anything better for them to do. Unfortunately #2 and #10 came in to play with junior and intermediate devs mirroring the existing system in tests, rather than describing the business requirements. - Focusing on how, not on what
We definitely had a lot of this going on. We have far more procedural ‘this then this then this’ tests that have a huge amount of setup actions than I like to admit. We have definitely turned this around somewhat more recently with more abstract ‘business requirement’ tests rather than ‘technical requirements’: the tests don’t change just because the implementation does. - Tests unusable as live documentation
This is my biggest chagrin with the tests, lack of prose around the FIT test tables explaining WHY. When there are comments they describe what we’re doing with the ‘this then this then this’ style. Again, we have turned the corner recently but it’s a bugger to retro-fit that kind of knowledge into the tests. - Expecting acceptance tests to be a full regression suite
I think I can own the blame on this one. I pushed for this to be the case. In my defense I did push for quick, localized acceptance tests that could run quickly and give the required feedback. As you may guess, that’s not what we ended up with. Rather, we got a whole slew of System tests. They are still very useful tests, they just take longer to run that I want. In a way it’s a reflection of the system we’re testing: very coupled system begets very coupled tests (maybe). - Focusing on tools
Not really something we struggled with too much. If anything it was the opposite – few tools in our price range. We settled on FIT and went forward. We also use the FitPro Eclipse plugin though it’s limited Mac and Linux capabilities are still a bit of a challenge. We did look at FitNesse for a while and knowing what I know now I would probably have chosen that as it is more actively developed. Our biggest perceived challenge was one of integrating with Subversion, in the end it was a non-issue. - Not considering acceptance testing as value added activity
We are doing so much better than before with QA/BA developing the first example FIT tests but we’ve still got a ways to go. - “Test code” not maintained with love
Another sore point for me. Developers treat our test code as an afterthought and seem to think that good design principles and refactoring don’t apply to test code. I couldn’t disagree strongly enough. I am always looking for ways of making my tests simple, robust and expressive. I often spend a lot longer on my (unit) tests than my production code. I have hear developers whinge and complain about the test code and how “it’s hard to do such and such”. I swear that in the time they’ve spent complaining they could have improved the test code ten-fold – but they don’t treat it the same as production code! - Objectives of team members not aligned
We’re actually doing pretty well at this one, we’re a million miles from where we once were. Our QA guys now sit with the developers and BA’s and there is a true collaborative spirit at times. I was utterly stunned (shocked, awed, caught-out) recently when a developer said to me “I’ll just commit it and QA can find the bugs”. I hadn’t realized that that attitude still lived on within our teams. I guess old habits die hard! - No management buy-in
This one isn’t a problem – I’ve seen too many bugs and mis-understood/implemented features and also seen the benefits of automated acceptance tests to ever go back (so’s my boss). - Underestimating the skill required to do this well
As mentioned above, we definitely struggled with this one initially. Things are improving as we get better and better with this. I think the biggest drag was the large amount of tests and fixtures that were written without too much thought that were then copy/pasted or at least aped.
Despite our challenges I think one reason we didn’t fail was that there were several people who held the opinion that ‘failure wasn’t an option’. We needed to a rigor to our development process: Automated Acceptance Driven Development was a large part of the added rigor. Yes it takes longer to develop initially, but it pays for itself over and over once you get over the hump.
Related posts: