What is the purpose of Agile?

It’s been a while since I posted. This is a reflection not of busyness or malaise, but that I’ve reached a new phase in my Agile journey. Things that felt certain are once again in flux as I question many of my previous assumptions about Agile.

This has led me to one fundamental question:

“What is/was the purpose of Agile?”

In many ways Agile was created as a shared space in which we could explore better ways of delivering software. It defined the boundaries of the space so that constructive dialogue could emerge. However, it would appear that those boundaries have not necessarily framed the right questions.

It has helped us define new methods and techniques for many issues in software development, such as a lack of feedback and learning. However, what it didn’t capture correctly is a focus on outcome and therefore became bogged down in methodology.

The Agile space was and is a fertile breeding ground for methods and tools, but it does not, by its very nature, help us devise ways of improving outcomes or delivery of value. We only need look at the plethora of methodologies and the ecosystems of agile tool vendors to see clear evidence that Agile, although successful in spreading and amplifying many positive behaviours, has also amplified behaviours which actively damage the Agile movement.

Therefore it is clear that it is time to re-frame Agile. To be clear about its purpose as a defined space for cultivating improvements in software development and delivery – for those improvements to be about the outcomes of a piece of software or the value that it delivers. In fact one of the first questions is the definition of value itself. Value to us as developers, value to society, value to business, value to customers?

Outcome based Agile is not new. In fact Tom Gilb wrote a paper 2010 speaking about Value-Driven development. Not only  that, Tom Glib has probably been arguing for this very thing since the 1960’s.

As part of this re-framing, we need to be clear that the framing is for the creation of a shared communication space and language. This space would be for collaboration, not competition, and not for commercial gain. For example, materials and books should be released for the benefit of the community and not for the benefit of a small cabal of vendors. In this way we ensure that the very essence of what it is we are trying to achieve is in the DNA of the movement. That the value is not in the system conditions but the outcomes that the system enables (in short the creation of a system is not the outcome but the value it delivers is).

Through questioning of Agile and by examining my thoughts and beliefs I’ve come to discover more and have taken another step forward on my Agile journey. This is a valuable technique and I suggest you consider taking some time to question the purpose of Agile/Lean/Kanban/Scrum now and in the future. It is by questioning our beliefs and ideas we come to learn so much more about them.

Would you consider exploring these questions further in focussed discussions?

References and further reading:

https://flowchainsensei.wordpress.com/2010/08/15/tom-gilb-laments-the-10-wasted-years-of-agile/

AgileByExample 2013: Tom Gilb – Agility is the Tool, not the Purpose

12 TOUGH QUESTIONS May06.pdf – Tom Gilb & Kai Gilb

Evolving organisational structures

There’s a debate going on at the moment about Zappos and their imposition/adoption of a new management structure based on Holocracy.

Basically Zappos want people to self manage and to speed up the process they’ve mandated it, offering severance to anyone who doesn’t want to work in this way. Broadly there seems to be a couple of responses to this:

  1. It’s a good thing because self organisation is a good idea
  2. It’s a bad thing because self organisation is a bad idea

It’s the classic media/internet response. Binary by nature (as if the very essence, 1s and 0s, of the technology underpinning the internet somehow makes our thinking more digital in nature… Anyway I digress). However, the issue here is not what one company decided to do and thinks is right for themselves, but the fact that one company’s structure or way of working is held up as a good or bad thing.

Organisations evolve. They are complex, the larger they are the more complex they become. The conditions in which they operate are unique to them and a product of the unique individuals that inhabit them.

Therefore a structure cannot be imposed or decided upon. However it can be evolved. It is not the management structures or way of working that needs to be codified and repeated but the evolutionary journey and thinking that codifies improvements into the organisation.

The response to the Zappos story also reveals another interesting facet to how we think about evolving organisations: that evolution is a constant conveyor belt of improvements. That somehow a new form of organisation is better than a previous form just because it was created after the first form. Evolution is the adaptation of an organism to the conditions it finds its self in at the time – not some constant refinement of the inferior to the superior.

By seducing ourselves that we are somehow doing things better than we were in the past, we fall into the trap of looking at the previous incarnation of organisations as somehow inferior and therefore miss the context in which they operated at the time.

Organisations need to stop looking at each other for inspiration or trying to improve on the past. We need to codify what is valuable in our current situation and reduce that which is no longer useful. This is how evolution works. It does not work by imposing a new supposedly superior form of structure borrowed without context from others.

Projects are actively harming your products

There are many things that can cause a software project to fail. I suspect we’ve all experienced a failed project at some time or another. The users didn’t like it, it was over budget, it was the wrong thing, it was late, it was buggy, it got cancelled – the list is often endless. But there is one common factor, the project itself. By project I mean a temporary team formed to work together to complete a batch of work.

What’s so wrong with projects?

Projects by their very nature are the antithesis of software development. Software is an evolving and flexible medium. It doesn’t have a fixed end state. There are no pieces of software (or websites for that matter) I use today that didn’t exist in another form in the past. The software and the sites evolved and changed based on new technologies, shifts in the market and customer feedback. This is the very definition of software development. However, when we come to structuring the work of software development, we usually use a project model. A project infers a start and an end. A point that when we reach it, means the project is complete – we can pack up our tools and walk away. When we undertake a software project we draw an artificial line in the sand (usually based on some inaccurate estimates) and pretend that the project will end.

This artificial end point leads to thinking in terms of big releases. The reason we tend to think in terms of big bang releases with projects is that when the project ends we assume that we will no longer be able to deliver any more value. We therefore need to ensure we have as much jammed into the project as possible, because when the project finishes we won’t have a chance to add anything that’s missing. Couple that with the knowledge that our team is likely to be disbanded at the end of the project and we have a situation where we have to ensure as much as possible is delivered for fear of missing an opportunity, forcing us to into a big bang release just because we used the project model.

The waste of project teams

When we incept a project we’ll build a new team from the remnants of the last project team, or we might co-opt some departments together to form an ad-hoc team. This new team will dutifully plough into the new project, slowly learning how to become more effective.

This is a known phenomena described in Tuckman’s stages of group development:  Forming – Storming – Norming – Performing

Working in projects is counter to this dynamic and means that just as we get to the performing stage we take a pickaxe to the team and break it apart, often loosing much of the social capital and working practices that have been built up.

Build capabilities not projects

So what’s the answer? If projects are actually damaging to how we build software what can we do instead?

I’d suggest that we begin to think in terms of building capability to deliver ongoing software improvements – rather than building pre-defined projects. What I mean by this is evolving an environment where we think in terms of continuous delivery of value rather than working on large chunks of value in a project model.

A project is a great big chunk of value, where the value is not realised until the end. This is very risky as any delays will delay the whole set of value. If we break down the project into a continuous stream of smaller chunks of value we spread that risk out. We also allow for course corrections in case we discover something new or the market changes underneath us.

To do this we’ll need to look at how we form and keep teams together. We’ll also need to look at how the work gets to the team and how this work is broken down so that it delivers value in smaller chunks.

In addition we’ll also need to look at how we release software. As without the ability to release small chunks of software we’re doomed to always work in projects, because how we deliver the value dictates the model for creating the value in the first place.

Less projects less estimation

Another benefit to moving away from projects is an intrinsic reduction in the amount of estimation we do. In fact you might even get to the point where you can eliminate estimation altogether. Let’s imagine a big project to rebuild a website, re-write all the copy and migrate it to a new CMS platform. There’s a ton of risk here. The normal business strategy is to estimate and plan in order to work out what might happen so we can understand and mitigate the risk.

However, we can never know exactly what is going to happen and we’re estimating at the point of least information. Which means the moment we start executing the project we’ll find that our estimates are wrong and we’ll have wasted the time we spent estimating. Also, by the time we realise our estimates are wrong they’ll have become commitments. At this point the whole management and reporting model for the project becomes a fig leaf covering the truth of the situation. Not only that we’ve probably got a newly formed team (usually with external experts) that’s learning how to work with one another – so the chances of the estimates ever being right was likely negligible.

Instead imagine the same scenario where we have a stable team, a process optimised for the continuous flow of value and a mindset whereby we look to break the project down into manageable chunks of value.

Now instead of estimating we can decide on the smallest chunk that will deliver value to start with. That’s our risk exposure – the cost of the team for that short period. Then we can see how much is delivered and use real data to do future forecasts on when the rest of the value will be delivered based on that data. Or we can have a phased release of funds for the work. Deliver the first chunk of value, then based on the value delivered get funding for the next chunk. In the phased funding model we don’t even need to estimate, we just get on with the work at hand. I’ve talked about this kind of approach before in my go/no go blog post.

Time to move beyond projects

This has been a fairly quick overview of why I think projects are a bad idea and why I think we should be looking for solutions outside of the traditional project model. This non-project future is not dependant on any particular methodology and you can start thinking about how this might work for you even if you’re not doing Agile. This is just a call to start the debate within your organisation. Are projects the best way we can organise the work? Are projects the best way for us to deliver value to our customers? It may be in some instances they are but I ask you to consider the alternatives and to run some experiments.

Comments welcome as always.

Inspiration, references and further reading:

A recommended Agile reading list

The following books are the ones I have found most influential and useful in my Agile journey. I recommend them as essential reading for anyone working in or with Agile teams.

These books will help both managers and teams shift to a more Agile mindset. They also help frame why focusing on things like “Flow” over “Resource efficiency” will get you better results than just following practices blindly. Lots of them have practical advice for how to implement Agile or improving Agile practices and give you some of the theory behind the practice.

My Agile library

What books have you found useful/influential? Let me know in the comments!

P.S. Here’s the list of the books in the pic (with Amazon links, should you want to buy them):

  1. This is Lean: Resolving the Efficiency Paradox
  2. Implementing Beyond Budgeting: Unlocking the Performance Potential
  3. Kanban
  4. User Story Mapping: Discover the Whole Story, Build the Right Product
  5. Kanban from the Inside
  6. Scrum and XP from the Trenches (Enterprise Software Development)
  7. Succeeding with Agile: Software Development Using Scrum (Addison-Wesley Signature)
  8. Lean from the Trenches: Managing Large-Scale Projects with Kanban
  9. Agile Retrospectives: Making Good Teams Great (Pragmatic Programmers)

Are your clients eroding your culture?

I’ve been reading quite a bit about encouraging or growing certain types of culture in organisations. Cultures which promote continuous improvement or fostering greater empowerment with the organisation. However it seems that one vital facet of the culture of your organisation is always omitted. That of your clients (I’m speaking from an agency perspective here, although this may also be applicable in other contexts).

Much of what is written about changing working cultures is about the internal world of the organisation, but this is only one facet of what makes an organisation. An organisation is made up of many complex interactions and at these points of interaction is where culture appears. For example: how we behave to one another and the shorthands we use. These unthinking actions are the cultural life of the organisation.

Our clients are another cultural interaction point. The way they interact with us can have amplifying or cooling effects on the cultures we are trying to grow. We should pay attention to how a client behaviour may permeate down into our organisation as this behaviour may erode the work we are doing to embed certain values. For instance trust and transparency. Perhaps the client embraces our new way of thinking and reinforces our work culture. Or perhaps more likely they are mystified as we begin to talk in a language alien to them and reject our new behaviours.

So when we look to our organisation in terms of a culture change, or new values we want to adopt, perhaps we should spend some time thinking about how this will effect our clients. Work out strategies for bringing them on the journey with us, or even no longer working with clients that hold us back from our wider goals.

Our organisational growth (wether we like it or not) is subject to factors both within and outside of our control. We should therefore attempt to ensure that the direction of growth is towards the light and is not stunted by unwelcome or unperceived factors.

Are you in denial about value?

I’d like to talk to you about something, lean closer dear reader as I’m going to have to whisper this. The intricate ways you are accounting for value on your Agile projects is probably wrong.

Maybe you don’t believe me, maybe you’ve always had doubts, but a lot of the time I see teams who can confidently proclaim they are delivering value in their iterations. They can show me burndown charts, burnup charts, velocity and a whole raft of metrics. But the real question is: who did they deliver the value to?

This is in fact the key question when considering value. If you are delivering user stories just to say you’ve delivered them and to notch off another mark on the burn down chart, you are very much in denial. There is no value here because more often than not you’re not actually delivering to anyone (your Product Owner doesn’t count, nor does your client if you’re doing contract development).

Value is only realised when the code or solution is in production. That means it’s thrumming away on a live server somewhere, not sitting in your source code repository waiting for the “big” release at the end. Now it should be argued that you still don’t know that it has value. As more often than not there are no metrics gathered showing actual customers using and finding value in your new feature. However for the purposes of this and as a first step to thinking about value in a more realistic way we’ll imagine an ideal world where any feature developed has some value to your customers (no matter how rare that really is).

If value is not in delivering user stories or tasks, then where is it? Well, as you’ll probably already be thinking after reading the above, the value is in the release. Everything else is just work to get you to the release.

That prototype you built, it has no value, it’s just a step to release. Something to steer you towards a more valuable release. That feature you built that’s waiting in source control for deployment. Again it has no value.

This normally means that your Scrum board, task board or Kanban board is a lie. Take a look at it now. Does it represent the steps to realise value? Does it only have cards on it that are representation of units of value (things that can actually be released when they get to the end of the board)? There’s a good chance that they are not. They’re probably just tasks waiting to get batched into a release. Your board is probably just a fancy to do list. Useful but not really representative of the actual process. Also not terribly useful in tracking if value is being delivered in a timely and consistent fashion.

I’d suggest that you try and visualise your actual unit of value on a board. So in most cases this will be a project or a release. It might not feel as nice as having a lovely busy board but at least it will be truthful about the situation. A situation where huge chunks of work are being shoved through the system and blocking up the whole process (all whilst your previous board was showing a lovely stream of falsehoods). This will also go some way to address the feeling that you are working incrementally when in reality you have a micro-optimised development process that is thrashing through large projects. Finally puncturing the illusion that you are agile/incremental/iterative when really you are just blind to the whole value stream.

Don’t worry you can keep your old board, but now it will be a representation of reality – in that it’s just a way to help you organise the work within the large batch of value that you’re trying to deliver.

Once you can actually see the problem then you can start to address it. Slim down your projects, do smaller releases. Work towards single piece flow. But that process can’t start whilst you’re still in denial about the value that you are delivering.

Photo Credit:  ShnapThat!  – https://www.flickr.com/photos/shnapthat/6464681311/

Agile is not a methodology, it’s an Albatross…

One of the hardest things to understand when starting out on the Agile journey, is that although most often sold as a methodology, this is looking at it at the wrong way. Scrum is a methodology, as is Kanban or DSDM or XP. But none of these are Agile itself.

The problem is that often the first experience of anything termed Agile are the methods and tools of a particular methodology. It’s how I was first introduced to it. The methods and tools however are of no benefit in and of themselves. Yes, they can help uncover issues in your current process, Yes they may have some use in helping frame agile in your mind. But this is really of limited value, as a new set of tools is not the answer to a dysfunctional organisation or industry. Frankly that’s fiddling whilst Rome burns. A methodology is not meaningful unless accompanied by a change in mindset, without that you’re just going through the motions.

The problem with Agile today is the word Agile has baggage. So much so that it is pretty much meaningless. If I asked a room full of people, who thought they knew what Agile was, what the term Agile meant, a large portion would describe a methodology (which it isn’t) and the others would probably all give me conflicting answers.

If Agile is not a methodology and it’s not really got a definition – what use is it as term?

The language and the terms we use, help define and bind us as a group. Shared terms, shared understanding, shared beliefs.

So the real meaning of Agile (if we can reclaim the term from the methodologists) is that it is the expression of a collective mindset. A shared set of values if you will. Are those values the ones defined in the Agile manifesto? I don’t think so. I think the mindset and values are evolving and we’re yet to see anything close to a definitive set of values. That’s why we need a collective term to represent and be the vessel for the inchoate and evolving values.

The language we use and the definitions that they collectively hold are important. Shared language is what defines the groups that we belong to. That the word Agile is now so laden with baggage that it has lost its meaning is perhaps a symptom of a deeper underlying issue. An expression of a community that no longer has a shared vision or language.

A community that is becoming so myopic and fixated on the methodology and tools that it is losing its collective vision. In its place a morass of commercial vendors competing to reduce Agile to a packaged and marketable commodity.

Like the term Free software morphing to open source isn’t it time to define a new shared language and reclaim the essence of Agile without the methodology?

Managing your input queue will seriously increase the chances of agile success

The smoking ruins of failed agile seem to be wherever I look these days, with people bemoaning how it doesn’t work and has if anything made things worse. There can be a multitude of reasons for agile to fail (frankly it’s not just agile that fails but the hype sometimes does agile a disservice), however a common reason is the lack of proper prioritisation of work, particularly when teams service more than one product or project.

Prioritising
Prioritising

At this point some people will be smugly looking at their neat prioritised backlogs and feeling distinctly superior. However a neat backlog alone is not the solution to the problem. For instance is your backlog iterative? Do you have stories that build on earlier stories? Have you just got a list of things to do where the business value is delivered after a whole batch of user stories are delivered? Is your backlog prioritised infrequently or only once? And finally, is your backlog a mess of chaotic user stories in some random order that you dump on the team from time to time in order for them to try to pick their way through. (Isn’t that what self organising means?)

If you recognise any of the questions above it’s normally because the queue of work to be done is badly managed. In Scrum that might be that the product backlog is too granular and disorganised. In Kanban it might be that managers are just slapping stories into the input queue with little regard for what needs to be done next – then re-ordering minute by minute for no clear reason (Isn’t that why you ditched scrum and did Kanban instead?)

By helping to prioritise the backlog or the queue of work, instead of a great morass of work being thrown at the team for them to work out the priorities, the team is empowered to pull the next item in the list without having to pick through a messy backlog.

This is often where managers can begin to resist agile practice, normally with excuses about not being able to prioritise because everything needs to be done. Really what they are saying is “I don’t want to be responsible for prioritising”. In that way they push the decision-making to the team without the required empowerment and therefore the blame of things go wrong. Or worse they try to take further control away from the team and tell them which user stories must be done in minute detail, accounting for every minute of a developer’s time.

However there is a way to wrest back control. Have a firm queue limit (in scrum your sprint backlog, in Kanban a WIP limit on the input queue) with a defined policy for how work can get into the backlog and be removed from the backlog. The manager or more commonly the managers then must talk to each other to ensure that the highest value work is being done in relation to other projects and other parts of the business. Also it may encourage more communication within the team during prioritisation. Scrum prescribes this sort of behaviour, Kanban allows this behaviour to emerge.

This simple area is often where agile begins to fall apart as it is commonly grown from within a development team. The team is then often unable to do anything about the input queues of their own work. This usually means agile adoption ends in work being done to the “whoever shouts loudest” model of management. This bakes in existing bad behaviour, or worse allows it to emerge!

The key is to ensure the person(s) who own the queue take responsibility for it. In Scrum that’s the  team, including the product owner (most scrum implementations I’ve seen seem to neglect this vital role). In Kanban that’s limiting WIP with clear policies for how the queue is stocked and how work is pulled from the queue.

The reason this is so vital to agile success is that it acknowledges that agile practice does not exist in a development vacuum. It is part of the wider aims of the organisation and it requires us to look at the entire value stream and manage the flow of value/work in relation to the overall organisational objectives and priorities.

Photo Credit: https://flic.kr/p/57vCMw

Trying to make sense of Rightshifting

I’ve been learning about Rightshifting of late and am finding the mental models used difficult to imagine. This led me to think about how I could redefine them so they made more sense to me.

However first up let me briefly surmise what Rightshifting is.

If you haven’t got time to look at these links now, skip forward, but they do a much better job than I will of explaining!

Rightshifting is a word coined by Bob Marshall, here’s his post explaining it. http://flowchainsensei.wordpress.com/rightshifting/.

In addition there is also the Marshall model, here is the explanation: http://flowchainsensei.wordpress.com/rightshifting/the-marshall-model/

Basically Rightshifting is based on the premise that an effective knowledge work organisation is a function of the mindeset of the organisation.

There are 4 basic mindsets in terms of organisational effectiveness:

  1. Adhoc
  2. Mechanistic/Analytic (these terms seem to be used interchangeably in the Rightshifting community)
  3. Synergistic
  4. Chaordic (if you’re interested in where the word chaordic came from, see this book: http://www.amazon.co.uk/One-From-Many-Chaordic-Organization/dp/1576753328/ref=sr_1_1?ie=UTF8&qid=1411052155&sr=8-1&keywords=one+from+many )

It is then suggested that the closer to Chaordic you get the more effective the organisation is. However most organisations are stuck at the analytic stage. Hence the need to right shift organisations.

Here’s Bob Marshal’s diagram of the Marshall model, taken from the above blog post:

Marshall model
Marshall model

Note the red transition points, denoting how hard it is to break through into a new organisational mindset.

What I really struggled (am struggling!) to understand was what the different organisational mindsets really meant.

So  I wrote down some words that might be used to describe the different stages:

  • Adhoc: Cowboy, messy, chaos, creation
  • Mechanistic: Process, tools, rules, methods, control
  • Synergistic: Learning, flow, empowered, value
  • Chaordic: Opportunistic, come together, mushroom, sharing

As you can see by the words, the adhoc organisation sounds like an early start up. The mechanistic sounds like most businesses! Synergistic sounds like an Agile or Lean organisation and Chaordic sounds more like an open source community.

I then thought further and came up with these definitions, to help me map it to my understanding:

  • Adhoc: Finding value
  • Mechanistic: Building value
  • Synergistic: Flowing value
  • Chaordic: Enabling value

Take a look at Rightshifting, if you haven’t already, and let me know what you think about my definitions of the various organisational mindsets and if you think they are a useful  or helpful way of describing them.

More Rightshifting posts:

Go/No Go? Simple question powerful results?

I was thinking about the process of building a solution and it occurred to me that in many ways, each stage of developing a product or project is a simple Go/No Go question.

It doesn’t matter how complex or simple it is to reach the Go/No Go question, the fact still remains that all we want to know is if we should stop or continue. Let’s look at an example.

Imagine I have an idea for a fantastic new website that tells you how many weeks it’s been since your last run and prods you to go running. (I didn’t say it was a good example!)

At first it’s just an idea, but already I’m at my first Go/No Go gate. Is the idea worth spending some time on? At this stage I might just mull it over for a bit. But in the end I’ll either decide it’s worth looking into further or it’s a rubbish idea.

Assuming I said it was a Go. The next phase might be to do a quick hack to see if it’s interesting or so I can share it with some people. Once I’ve done this I’m again faced with the same question. Go/No Go.

Now we have an idea that’s made it though at least two gates perhaps I’d like to think about how it might become a proper product. So my next move might be to do a Lean Canvas, where I outline the business model. Again the purpose of this is to get to a Go/No Go gate.

Eventually I’ll start doing development on the product. I might even identify a MVP (minimum viable product), but again I just want to get it up online so I can get to another Go/No Go gate.

Finally, making one last leap of assumption, my product is successful and I want to build some new features. Again for each feature we can use the Go/No Go cycle.

This can go on indefinitely but the point is by using on the Go/No Go question I’m focusing on what I need to know to make that decision. Also, as it’s a choice that means I could throw away what I have done so far, it makes sense to work in smaller chunks. So there is less waste if I do hit a No Go.

What happens if we do hit a No Go? Well if it’s a product/project in development then it means just that. We’ve hit a point where there is no value in continuing. Do we want to launch something that has no or negative value? If the product/project launched and is delivering value then the No Go is only for the feature(s) about to be added. Again do we want to launch features without value?

What might happen if we can’t make such a binary decision? I’d suggest in that instance we need more data. Perhaps conduct an experiment or experiments to test the hypothesis. Whatever happens, in the end we’ll get to a Go/No Go choice.

The key to the Go/No Go approach is to ensure we eliminate the waste in getting to the Go/No Go gate. Take only the steps that allow us to make that decision, whether that be some thinking time, a napkin sketch or an iteration of development. Also that if we deliver value as early as possible in the process we minimise the risk of major waste.