Most organisations have gone some way now towards implementing agile development techniques. It’s likely as well that they will be using lean in other areas of innovation. The idea that innovation can and should be done as cheaply as possible has taken hold in business as we all seek to innovate more and faster for the sake of survival (and prosperity of course!). Along with it, though, should come the realisation that a lot of what passes for innovation is wasted effort. We are applying methods to a process without asking if the process itself is really necessary or efficient.
Part of the challenge is that traditionally IT and the business have been at a distance (the IT business gap). We are in silos and have silos within silos. Once you accept that, you can see that many of the processes in firms are actually there to support siloed working. They are not there to support efficiency, velocity or innovation. Agile processes are designed OUT as part of the basic management philosophy.
In the modern IT shop, however, we are constantly seeking ways to do new things – innovation, by definition. We are now continuous innovators. That is the nature of much work in engineering environments and is part of the engineering DNA.
By and large, however, the rest of the organisation might not see IT that way. Companies have taken to setting up innovation labs and accelerators, effectively creating innovation silos! They do this unaware that their IT department may have undergone profound process change over the past five years and has moved to a new innovation model where they are innovating every day.
I think that means two things:
- IT’s transformation can be a learning model for other sectors of the company. That’s what happens when people read books like Lean Innovation, which in effect is an account of IT method.
- However, IT itself still has some important lessons to learn. And one of them concerns “wasted work.” It behoves us to step from behind the new IT methods (devops, microservices etc) and ask questions about what we are innovating and the process and learning models that guide it.
Many people now want to do a similar kind of innovation. They express it in different ways – like how do I act more like a startup? But that question is really a question about software. Software powers most startup ideas.
To do lean software development is good but what if the work we do in a very lean way is actually a waste of time? Who is making the judgement about work value? How, indeed, can we make that judgement?
The concept of flow is the answer. It’s not a simple answer so I will first recap a little on where why we have reached a limit with agile/scrum and what we need to change and then go on to describe flow.
The rise and (some time soon) fall of agile/scrum
Agile has been a good companion to the software world. And has spread its tentacles out to movements like lean innovation. In essence agile has three major goals:
It is a method for taking software development closer to customers and working iteratively with them to capture their needs and then building in continuous customer feedback loops in order to pivot quickly (not fail fast which is a misnomer in my opinion).
It has allowed software teams to bring different elements of the old waterfall process together, so now we are less likely to produce huge requirements documents that we then hand over to a coding team, who then had it on to a QA team and finally an operations team; agile led to devops – the combination of development and operations and that’s a much slicker way to build software, especially when coupled with the automation of testing and the immediate deployment to “live”.
And finally it has gone hand in hand with the idea of visible work – scrum/kanban boards that get what is essentially brain work (planning, coding, integrating) out into the open where it can be seen and critiqued.
The problems with it are two fold:
- Agile does nothing to help assess what work should be done. As work lines up in front of the development team there is no real evaluation of its final benefit which means there is always the risk of work that has low value.
- It isn’t really a method or framework for designing a work process. Software still needs a grand plan if the CIO/CTO is to allocate resources efficiently. In the absence of a new framework two things can happen. You get the very valuable trend towards microservices – break everything up into smaller packages and focus on stitching them together; or you get waterfall by default – very large projects that just happen to get done in two week sprints. The upside, microservices, calls on a software architect to be very astute in planning which services are going to be relevant so you still come back to that problem – what is the framework? And to the first problem – where is the value being created?
The idea of flow
In my last article I placed a lot of emphasis on visualisation. To be able to see work and have many eyes critique it is an extremely important check on CIO planning skills but it can also be directed at the problem I mentioned above. Are we just lining up work for the sake of it?
The Pareto 80/20 rule applies to most areas of innovation. As we proliferate ideas for new functions and new services, it is highly likely that the most value (say 80%) is going to come from about 20% of what is proposed (though we are often tasked with implementing 100% of those ideas).
Because organisations want to be more innovative and they want IT to support that, actually to drive it now in many cases, getting this balance right is extremely important. Resources are always limited and the development process can create a lag in the valuable functions that make a difference to the business. That lag can simply be a resource allocation issue: too many ideas, too many requests, plenty of developer time being used to add very little value.
There is then the planning issue – now we are not planning waterfall projects and not wasting resources on handovers, we still need to plan how to use resources in the best possible way.
The best way is to bring together interdisciplinary holistic teams that can take a really critical look at features from the outset – but bearing in mind there is no “outset” like there used to be. There is now a continuous list of things the business wants.
In my area of responsibility I have had to argue for those teams to come together. Therefore I have also had to negotiate with the managers of team members to have them in my “flow” team if you like. It is hard to create those teams because corporate politics are what they are – managers have a strong identification with the size of their departments and authority is often sucked up to the highest levels and walls are actively encouraged.
The DevOps model of holistic teams are often talked about but are rarely created within other areas of the enterprise. But in fact these teams are the essence of a startup and a model which we seek to emulate in order to get stuff done quicker.
The way round it for me was to focus on “one” holistic persistent team which contains all the skills in order to prevent any handoffs or unnecessary wait times. The team included Business folks (Product Owners, Business Analysts etc) and IT people (Developers, Testers, Ops and Security etc).
A very high value is placed on people spending time together as part of their overall development. With senior leaders and experts guiding the team but more importantly removing blockers and helping to improve the flow process, not the project.
This is an important point because when one improves the process or the flow, every subsequent project benefits from that improvement. It is actually very powerful and simplifies the holy grail of continual improvement.
In the team the work focuses on three things:
- A realistic valuation of new features and their benefit to customers so that we can prioritise (value based decision making)
- A more conscious approach to the process of coding to identify inefficiencies in how we work together and to automate as many manual processes as possible – even the creation of a change request!
- The merging of as many tasks and responsibilities as possible so that the work gets done without handovers – team members with fungible skills confront cultural barriers and question the old fashioned siloed operational models by helping to design them differently – yes, each team is now a mini startup
All that in a very visual environment where everybody’s eyes can focus on the detail.
As these teams come together, we find that we can make changes in work processes, for greater efficiency; and we can be surer we are doing the right work; plus by focusing on the flow, we are inadvertently helping every future piece of work.
Flow as a learning model
Flow principles and holistic teams cannot exist without a strong learning model at the heart of it. When team members become proficient, they need to be transplanted into a another team so that the new team can benefit from embedded mentors & experts. This method of working enriches the knowledge of each employee about the overall business and allows for infinite scale. Teams of 7 or 8 people, mini startups, all in the flow but scaled to Enterprise levels.
Automation plays a vital role in the world of Flow and as we move towards continuous delivery we also need continuous planning, continuous architecture development and continuous engineering – infrastructure that builds itself and scales as your business takes off. Nothing can stand still.
That is all too much for one brain. Nobody can plan it all out and nobody should. In effect what you want to have happen is that the team itself through its interactions, through a more conscious approach to responsibilities and the mission of work redesign, constantly modifies the flow of work as it goes through.
In fact what you get is a true learning organisation. Your flow teams are not students. They are researchers, teachers, critics and learners, studying the issue of what adds most value; and simultaneously designing the best ways to get things done.
The mission to redesign the process coupled to the requests we get to produce new features actually creates the startup atmosphere that many big companies are looking for. The challenge for large companies is they want to see all that in a document or business case whereas in reality what you want to do is point at the Kanban walls and say – that’s my plan and these people are helping me improve it by building the flow of value that our customers want.