Ivan’s agile thinkings

Last weekend I wasn’t feeling so good due to a bit of a cold.  As I manage more than I do these days I don’t often get time to code.  Interestingly I’ve noticed a bit of a trend that I code more when I’m feeling ill.  I don’t really understand it, but as long as my brain is working and my body doesn’t feel like doing much I suppose it’s nothing to worry about.

On to the main point of this post.  My company is currently using Subversion as its source control mechanism. I’m pretty sure I want to change this to something better.  I have the influence and power to be able to enact this change, potentially overnight if I REALLY wanted, but that would be as successful as flying to the moon in a sherbet-powered watermelon.

For a start, I’m not 100% sure of what ‘better’ would be.  We’ve had a small grass-roots movement trying different types of DCVS for a while.  Basically this meant Mercurial and Git.  Git seems to be winning out at the moment, and its got to be said that the git-svn seems very solid.

I’ve been tracking svn using git-svn for a while – I might not ‘do’ so much any more, but I know the answers to lots of things and need the code to refer to.  But last weekend was the first time I’ve used Git in anger.  So, I set about tidying and refactoring a set of unit tests that are horribly coupled to a large chunk of the system – and are slow because of it.

What a difference Git made to my work-flow!  Edit, compile, re-run tests, COMMIT, edit, compile, re-run tests, COMMIT.  I was committing many, many times more than I ever would have with Subversion.  Even if I had my own Subversion branch to isolate my changes, the time to commit over the network would have stopped me doing it so often.  Git was a breath of fresh air.  Knowing I could just do my work and commit as part of my workflow was very liberating, confidence giving and really quite enjoyable.  At the end of the session, I simply ran ‘git svn rebase’ re-ran my tests and ‘dcommit’ed back to Subversion.*

I have a longer term plan to get a GitHub style development system internally at work and this experience has encouraged me tremendously.  So much so that this long-weekend I installed gitorious and gerrit on a VirtualBox VM to give them a test drive.

Anyone else changed from Subversion to Git?  Got any pointers or war stories?

* Actually I got a little help from a colleague to move my changes onto a git branch so then I could merge/squash them before committing. Thanks for the help and learning Mehran.

A lot of my reading recently seems very coincidentally related to the concept of complex systems.  What I’m reading proposes that linear (cause and affect) systems are the special case, the vast majority of systems in the universe are complex, emergent systems.

Since getting my Kindle about a year ago I’m reading (and buying – go frictionless commerce!) more books than ever.   And because the Kindle is so convenient and I can read the same books on my Kindle, Mac(s) and IPhone it means that I generally read e-books rather than paper books.

However, its seems that I have defeated myself a little and should have read two of the paper books that have been sat on my shelves for over a year (since before buying my Kindle) and perhaps been that much ahead.

Read the rest of this entry »

The majority of the individual pieces are done.  There are a few ‘technical’ pieces left that can wait a while: the main focus now is sticking everything together and decorating it.  We’ve been making some good progress on this, once we decided how we were actually going to decorate everything.

So here’s the new ‘construction’ burndown.  It shows those components (made of several pieces) that are complete.

As with most (all?) projects, we learn something new along the way.  This happened this morning with the Halloween Costumes.  There’s one component that needed a bit of a redesign and that means new pieces.  So, I’ve frozen the burndown image in the previous post and will include a new updating one here.  This new image reflects the need for a previously unexpected 8 reenforcement pieces.  I had a stellar preparation day yesterday and even did some construction where it made sense (composing components from the pieces).  My mind is starting to think about how we are actually going to do the decoration that will bring the costumes to life: decorate before or after construction?

Interesting to note that the way my spreadsheet was set up, when I upped the total number of pieces it gave me a negative velocity.  That obviously can’t be the case.  The velocity shouldn’t change downward because of the new work, I’m am still completing work at a similar rate to previous: there’s just more work to do now.  I fixed up my spreadsheet so that velocity is appropriate (the number of pieces ‘done’) and the remaining work has gone up too.  The day is still young(ish) and if I choose I could make a concerted effort to bring the project back onto the previous schedule, or I can have a chat with the PO about pushing the schedule or reducing scope.  As it is I’ve started with plenty of time and effort cost is only my time (and I’m enjoying the work).

The more I do, the more pumped I get that this is going to be a seriously awesome costume!

My Halloween costume this year is going to be epic.  We gave everyone a chance last year, but the return of something on the level of paper mache Pinky and The Brain must be seen this time around.

The (secret) costumes I’ve selected (plus other friends) requires a whole bunch of ‘pieces’.  We’re in the prep phase so we’ve got piles of inventory building up.  This isn’t very lean, but I thought at least we try a little agile.  Note that the total number of pieces may go up at any point as we refine the design.

So, here’s the burn-down for the preparation phase – still got a ways to go!

Piece Burndown Snapshot

Just a quick post to let everyone see the latest incarnation of the big, visible build monitors that I have been building.  The current version connects directly to the network and scrapes Hudson’s json api for a specific view of the builds we have.

Here’s a list of the component parts:

• an Arduino ATMEGA328
• an Internet Shield
• code to parse the json api output of Hudson and drive the LEDs based on the status (colour) of the builds
• a few high-power LEDs to glow as brightly a possible (might whack some more in there one day to really make it glow)
• a dome (with previous lamp removed) from IKEA

This version is much prettier than the tupperware version I produced previously.  However, I did have to up the LED brightness so they can actually show through the pearlised lamp dome.

The next version will trigger a sound when the build status changes so the team knows that something’s gone wrong sooner (and that normality has been resumed when the build is fixed).

NOTE: The ‘bling bling piggy’ is always nearby to help lighten the pockets of those who break the build.

I had the good fortune to attend the latest VanDev meetup on EAI tools versus custom build solutions.  David Dossot gave a high-level overview of Enterprise Application Integration tools and how they compare to the ad-hoc chunks of scripting and home-grown glue that are used to integrate systems.  David covered several aspects that might affect whether you pick on or the other.  His presentation is online at Prezi (which I think is a very fancy presentation system).

Update:  David actually blogged on this too: http://blogs.mulesoft.org/presentation-eai-when-tools-can-help/

The big thing for me about this presentation was that it finally tripped a switch in my brain: Enterprise Application Integration patterns don’t have to be about asynchronous queues and such, it’s about all the different ways systems integrate.  That includes simply copying files around, for example.  I honestly wish I’d realised this a few months ago, it would have given me the insight to stop a strange project using rails and leverage an existing EAI engine/framework.  Despite the learning curve we would now have a less custom, and more tested, system.  Better late than never?

I’m also currently on a mild clojure kick.  Having done the Bowling Game Kata in Groovy and then followed it up with a clojure version (thanks to this link for some hints) I thought I’d try to get a simple Apache Camel example up and going in clojure.  The first hit on Google proved very useful for guidance.

Simply trying to get that (and the bowling) example up and running made me have to dig into so many different things:

• leiningen - a clojure wrapper for maven
• Polyglot Maven - a wrapper for maven in several different languages so XML can be shunned
• test-is – a simple unit testing library in clojure-contrib
• AOT – ahead-of-time class creation, amongst other things this allows clojure to run in a binary form (no need for source in production).
• Counterclockwise - an Eclipse plugin that adds support for clojure editing and running (with very pretty, and effective, rainbow bracketing)
• Clojars.org - a maven repo for clojure jar files
• Cloduino can be used to allow clojure to interact with Arduino boards
• Getting started with clojure - this link has a bunch of detail on getting started with clojure.
• labrepl - a learning environment for clojure
• I fired up Idea for the first time in ages and install La Clojure just to try it out.

Obviously, I have only scratched the surface of these things, but at least I know about them now and have a better overall handle on clojure and EAI.

I really do love learning new things and the VanDev meetup triggered more of this than normal.  So I encourage you all to come to VanDev or find a similar meetup near you which covers things that you really enjoy (I’ve got my eye on the local Erlang group next).  

InfoQ is one of my favourite techy sites on the interweb.  It’s full of really great presentations, interviews and articles about many different aspects of software development.Here’s an example where Marc de Graauw writes about not needing reliable messaging.  If the business operation needs to be once-and-once-only or the operations need to be in-order then the mechanics should be pushed down into the business message/processing.  Don’t leave this to the transport layer, create appropriate idempotent operations and most of the problems go away.If you don’t already have InfoQ on your feed list then I can’t recommend adding it in enough

I’ve had this one sat in my FireFox tabs for months and only just got around to reading it.  Gojko Adzic lists the outcome of an openspace session at CITCON Eurpose:  http://gojko.net/2009/09/24/top-10-reasons-why-teams-fail-with-acceptance-testing/

Most of the reasons rung true with me too, especially at the start.  It’s been two years now since we adopted FIT as our acceptance test tool.  We’ve improved greatly since then, but we still have challenges.

1. No Collaboration
   Initially we had developers writing tests and proving the usefulness of the approach.  Unfortunately, the same developers were then tasks with writing a battery of tests against the existing buggy code.  It was a between-project slack period and there wasn’t anything better for them to do.  Unfortunately #2 and #10 came in to play with junior and intermediate devs mirroring the existing system in tests, rather than describing the business requirements.
2. Focusing on how, not on what
   We definitely had a lot of this going on.  We have far more procedural ‘this then this then this’ tests that have a huge amount of setup actions than I like to admit.  We have definitely turned this around somewhat more recently with more abstract ‘business requirement’ tests rather than ‘technical requirements’: the tests don’t change just because the implementation does.
3. Tests unusable as live documentation
   This is my biggest chagrin with the tests, lack of prose around the FIT test tables explaining WHY.  When there are comments they describe what we’re doing with the ‘this then this then this’ style.  Again, we have turned the corner recently but it’s a bugger to retro-fit that kind of knowledge into the tests.
4. Expecting acceptance tests to be a full regression suite
   I think I can own the blame on this one.  I pushed for this to be the case.  In my defense I did push for quick, localized acceptance tests that could run quickly and give the required feedback.  As you may guess, that’s not what we ended up with.  Rather, we got a whole slew of System tests.  They are still very useful tests, they just take longer to run that I want.  In a way it’s a reflection of the system we’re testing: very coupled system begets very coupled tests (maybe).
5. Focusing on tools
   Not really something we struggled with too much.  If anything it was the opposite – few tools in our price range.  We settled on FIT and went forward.  We also use the FitPro Eclipse plugin though it’s limited Mac and Linux capabilities are still a bit of a challenge.  We did look at FitNesse for a while and knowing what I know now I would probably have chosen that as it is more actively developed.  Our biggest perceived challenge was one of integrating with Subversion, in the end it was a non-issue.
6. Not considering acceptance testing as value added activity
   We are doing so much better than before with QA/BA developing the first example FIT tests but we’ve still got a ways to go.
7. “Test code” not maintained with love
   Another sore point for me.  Developers treat our test code as an afterthought and seem to think that good design principles and refactoring don’t apply to test code.  I couldn’t disagree strongly enough.  I am always looking for ways of making my tests simple, robust and expressive.  I often spend a lot longer on my (unit) tests than my production code.  I have hear developers whinge and complain about the test code and how “it’s hard to do such and such”.  I swear that in the time they’ve spent complaining they could have improved the test code ten-fold – but they don’t treat it the same as production code!
8. Objectives of team members not aligned
   We’re actually doing pretty well at this one, we’re a million miles from where we once were.  Our QA guys now sit with the developers and BA’s and there is a true collaborative spirit at times.  I was utterly stunned (shocked, awed, caught-out) recently when a developer said to me “I’ll just commit it and QA can find the bugs”.  I hadn’t realized that that attitude still lived on within our teams.  I guess old habits die hard!
9. No management buy-in
   This one isn’t a problem – I’ve seen too many bugs and mis-understood/implemented features and also seen the benefits of automated acceptance tests to ever go back (so’s my boss).
10. Underestimating the skill required to do this well
    As mentioned above, we definitely struggled with this one initially.  Things are improving as we get better and better with this.  I think the biggest drag was the large amount of tests and fixtures that were written without too much thought that were then copy/pasted or at least aped.

Despite our challenges I think one reason we didn’t fail was that there were several people who held the opinion that ‘failure wasn’t an option’.  We needed to a rigor to our development process: Automated Acceptance Driven Development was a large part of the added rigor.  Yes it takes longer to develop initially, but it pays for itself over and over once you get over the hump.

I picked up my long-unread copy of Emergent Design by Scott L Brain.  I’m not too far in yet, but something he wrote about software patterns struck me in a way that hadn’t before.  He mentions that other professions all have their own patterns. Doctors, Lawyers and Carpenters, for example.  That made me think of the joints used by carpenters, like the dovetail example on the left[1].

Disclaimer: I’m no carpenter

Just like software patterns, carpentry joints aren’t a drop in solution.  Joints are a well understood carpentry pattern but they still must be used in context.  The type of wood being joined constrains the joint you can use, the desired appearance constrains which joints can be used, the use of the jointed pieces further constrains the choice of joint.  Further, the thickness of the wood will determine the size of the dovetails for example.  The constraints go on an on.

The difference between carpentry and software development is that the understanding of which joint to choose has been developed and refined by master craftsmen for thousands of years!  You can now buy laminate flooring that just clicks together – but it took the legacy of many, many craftsman hours to make that a possibility.

The take away for me is the re-enforcement of the truth: we’re still the software pioneers.  We’re still lashing bits of code together with rope, we can’t quite get a catalog of time-tested, tried and true joints.  We know enough to know there must be better ways of doing things, we just don’t have a clue of what they are yet.

Software development might just be about pioneer spirit, and anyone who tells you they’ve got it sorted is probably trying to sell snake oil to the prospectors.

[1] Image used under the terms of the Creative Commons Attribution 2.5 Licence – Author: Dumitru Rotari