Time for Agile to GROW

Fri, May 20, 2016

Many years ago, when I worked in industry as a driver developer, I used what was essentially the Waterfall method of software development. I had no choice as that’s what the company used. The team wrote and maintained detailed requirements documents. Lots of bullet pointed paragraphs, sub-paragraphs, indices, glossaries, what’s the point of these?

Then, not so many years ago I moved to academia, developing software for things no-one seemed to think would work. The landscape was different. There were lots of dugouts, each with a megaphone hailing the message “we’ve looked into this and it can’t be done”. The noise was deafening and the stench of stalemate was everywhere. From creating accounts via LDAP to creating users from a Blackboard Building Block and populating Groupwise postboxes, the message from the dugouts was the same. “Can’t be done”.

Usually it was a simple case of reading the documentation (LDAP), reading the Javadocs (Blackboard) or using dumpbin on the Groupwise mail client (COM) to find out that everything was indeed possible, as it usually is in the world of software development.

And so began my long love affair with systems integration. I’ve integrated just about everything that can be integrated (and a few that apparently couldn’t). I’ve built reliable messaging systems to facilitate the integration and feedback loops have sprung up to inform and improve business processes and the dugouts have largely vanished. These days it’s a case of simply asking “what API does it have?” and the rest (excuse the pun!) falls neatly into place.

REST, SOAP, COM, DCOM, CORBA, RPC, XML-RPC, MOM. I’ve seen and used the lot and it’s been great fun introducing disparate systems to each other, designing middleware to hook up their respective APIs, usually via messaging of some sort. But in the beginning, I had to get there somehow, without using my existing design skills, which were largely Waterfall based.

It was obvious from the start that a detailed functional specification wasn’t going to work, mainly due to comments such as “If you expect me to read that, you’ve another think coming!” so I had to come up with a different way of working. A way that could deal with last minute changes of product, let alone change of requirements. The rug was royalled pulled once when someone decided they didn’t want the current forum product but instead demanded we use a different one, a few days before release. But that was fine. No really, not a problem. The design could cope as those people were part of it.

To be more precise, the attitudes of people were part of the design. By decoupling systems via an extra layer of middleware, rather than plumbing them directly, I could fairly easily switch one out and trundle in the latest faddish behemoth, as long as it had an API. I did have to bypass some products and interact with their databases occasionally but that method of integration was very bad, very brittle and broke on every release of the product but at least I could cater for them.

Other instances of change at the last minute could be an email from Consultant B telling me the recommended method of integrating with their product was no longer supported and Consultant A should never have recommended it. Such are consultants. They rarely consult.

Some products were a delight to work with. Some were so badly designed they were destined for the bin. Over time I developed x-ray vision so to speak. Give me a system, its API and I could spot all the integration points, their potential problems and creative solutions to get round them as well as opportunities to enhance other systems no-one had considered. All while juggling several languages. For integration ranges across many domains. Java, C#, C++, Ruby, Go. Someone once told me “integration is hard” but it’s not really. The golden rule is “Know Thy System” and have a healthy collection of languages in your cupboard from which to start the journey.

Then one day I found myself using Jira. Reading about User Stories, Sprints, Iterations. The more I read, the more I realised the methods I’d “just been using” had a name. I’d been doing “Agile”. Asking people what they wanted, when they wanted it. Offering suggestions to guide them to a better product or integration, quick iterations. I’d work on the current version while they decided what they wanted in the next. Bartering, negotiating, releasing, cheering!

I ran my own Subversion server, Jira server, Jenkins server. I discovered VirtaulBox and Puppet. I read some more and found I was doing DevOps. I like DevOps. I time I moved to my own Git server and I like that a lot.

It was an Agile world. Things were working. The megaphones were in the museum and reliable messaging was taking care of underlying oddities in networks, allowing business critical integration to function quite happily. I began developing full stack applications. From backend REST APIs to front end user interfaces, to iOS apps that used the REST APIs. It was just another form of integration. By the time I’d evolved to this level of software designer, the landscape had changed to Flexbox, Bootstrap, AngularJS etc. I managed to avoid the more cumbersome Javascript frameworks but delighted in Bootstrap’s then Flexbox’s CSS design kits. User Stories, Iterations, Sprints. They were all still there. Happy Days LOL! as a friend is want to say.

But not that long ago things began to change. Someone asked me if I could change how Jira transitioned between User Story states for their project. It took some effort to work it out. I was creating different notifcation/security permissions stuff for different project owners, largely because they couldn’t be bothered learning the Agile tool. It was too complicated for them but they still said they “were agile”. They worked in iterations with stories but someone else had to build their infrastucture to allow them to do that. They didn’t want to know. But they “did agile” you know. “Yes I’m agile but I don’t check anything in until it works” is another one. So the Continuous Integration server never gets to prove it works outwith their computer. Not that they know what that is but because they have stories, they “do agile”. Something stank. Then I found the reason.

Agile is dead.

The current state of Agile reminds me of the market segmentation of the outdoors. I’ve been mountaineering for around 35 years and back when I started, if I wanted to cycle to a mountain to climb, I’d bung a load of gear in the panniers and cycle there. If I added a detour on the way back I’d be “cycle touring”. These days it’s called “bikepacking” and has a million different flavours, all supporting small dugouts of people chucking social mud at each other over their choice of gear. Walking is now segmented into “fast packing”, “raft packing” “section hiking”, “thru-hiking” (sic), each defined by the gear used to pursue a particular strain of “walking”. The gear has come to define the experience.

So it is with agile. The things we always did have been assigned names. SCRUM (the Friday morning meeting), Sprint (what we’re working on today), XP (writing code not knowing whether it’ll be used), TDD (well the documentation has to go somewhere). We’ve been sold tools to support those things we’ve always done that now have names. The tools have come to define the experience and the tools are becoming increasingly non-interoperable and time consuming to use.

Jira is no longer Jira. It’s something else called Jira but with a rather interesting upgrade path. I can no longer keep our Stash server up to date as Stash no longer exists. It’s now BitBucket Server. That would probably explain why Jira no longer talks to Stash to show commits in User Stories.

I once showed an ERASMUS student how to “do agile”. We sat down with the product ownwer, a large sheet of paper and a stack of stickies. A few hours later we had three columns of stickies on the paper and the student had lots to do, developing the stories and shuffling the stickies around on the paper and the product owner was ecstatic.

These days, I’m facing upgrade hell in order to keep our “agile” infrastructure up to date.

The tool vendors are now telling me how to “do agile”. In their way. Not mine. Theirs. To quote Dave Thomas, I think the tool vendors are effectively

creating a trade union for people who breathe [1]

then changing how the people breathe.

Agile is market segmented. Time to go. But to where?

Back to basics land is where. I like the sound of MVP (Minimum Viable Product) and CD (Continuous Delivery)[2]. They’re what I’ve been doing for years, just not called those things. Agree the initial spec, get it out the door, let them play with it, get their feedback, changes, enhacements. Develop, test, release. Get more feedback. Sorted. Just as the tool vendors squeeze that bit harder, become a bar of Agile SOAP (ouch another integration pun!) and fly out of their hands.

Unless you’re developing systems that preserve life or send people to Mars, as a developer, you can probably get by with a few whiteboards, large sheets of paper and plenty of stickies. As Dave Thomas says:

anyone who comes up with something bigger or more complex is just trying to sell you something [1]

I’m sure “Agile” will live on, largely as a reporting mechanism between developers and those further up the chain. For managers have reporting requirements. They need ways to work out how “Agile” their team is. They work all that out from the tools. They could let the developers develop “with agility” and let them develop as crafters of quality software. When developers reach that level of skill, all a manager need do is ask how long it will take and trust the answer. Rather than treating developers as “business units” who’s output should be fed into a large data mincing machine, in order to produce burndown charts to refine future estimates, going forward.

But we as developers (and occasional managers like me) would like some sort of guidance. A conductor who’s baton can inform our timing, to allow us to beat out a rhythm that’s suited to our current personal stage in life.

For not all developers are at the same stage, obviously, doh! And this is where I’m getting rather excited, for someone has decided to integrate (you see? I just love integration!) the Dreyfus model of skill acquisition with software development.

The Dreyfus model is just about the only academic theory I believe. I believe it wholeheartedly for the simple reason I see it in action. Every. Single. Day.

I recognise the traits of each level in different developers. In my own pursuits. In everything I do and I can thoroughly recommend Andy Hunt’s wonderful book Pragmatic Thinking and Learning as a very readable introduction to this theory.

The integration I’m talking about is called “Growing Real-World Oriented Working Systems” (GROWS). GROWS recognises the different stages of a developer’s professional life and that’s why I’m so excited about it. It has the potential to work for everyone I work with.

Anecdotal evidence suggests Dreyfus is important in software development, as rather than embrace a full (and risky) agile process

most simply prefer to follow a narrow set of rules for a set of practices because it’s safer [3]

and

it’s necessary to have simple rules for beginners to follow while they come up to speed with the development method [3]

Adoption of Agile methods is

motivated by the belief that agile practices can improve cross-functional collaboration, software quality, and customer satisfaction, while shortening time to market [3]

To me that sounds like market-speak. The segmenters are at work here and the above could only ever be true if everyone involved in a company’s agile processes is at a high Dreyfus skill level. It’s certainly not true for teams with a mix of Dreyfus levels.

Statements such as

Kanban board where they have per person swim lanes … separating people and their effort completely from each other [3]

bring back the sound of those dugout megaphones and make me shudder.

Andy Hunt, in explaining GROWS states

The model will provide support for beginners while accommodating individuals with more advanced skills [3]

which I can only applaud as those are the people who make up real life. Not segmented lane swimmers, either spluttering on the point of going under in a sea of unfamiliar technology or ultra frustrated Dreyfus Level Fivers trying to cut themselves free of the level One lifejackets.

Matthew Kern offers what I think might be an explanation of the term “agile is dead”.

Many big things keep moving after death, like giant zombie worms [4]

Agile may be on the way out for real world developers but the tool vendors will probably eke out a fading revenue stream from the doubters. As to who are the ‘giant zombie worms’, those left in or those still feeding them, I shall leave that as an exercise in deduction for the reader.

So I’m very much looking forward to taking the first steps into a world where developers have a real voice and a framework in which they can grow and flourish. Developers will be happier. Customers will be happier. Managers will be ecstatic and software quality will become a given. Will it work? I hope so.

Happy Days LOL indeed!

1 Agile is Dead (Long Live Agility)
2 Agile is Dead, Long Live Continuous Delivery
3 Is the Agile Manifesto dead?
4 Agile is Dead