The Intuition Gap
If there’s one nearly-universal challenge that all engineering leaders face, it’s that there is a constant pressure to “move faster.”
This is an all-too-familiar story. A project was started with a “hand wavy” estimate of when it will be done that is no longer achievable and important deadlines are going to be missed. The engineering team professes its subscription to an agile philosophy with features delivered “when they’re ready,” not by a certain date. A seemingly straightforward request is scoped at weeks of work by the engineering team. A complex project is needed “ASAP” because it “should be really simple, right?”
Even the most cursory research on this topic gives a wealth of advice, information, and approaches. There are entire books written about improving transparency and observability of our projects. We stress the importance of accountability when things appear to be “off track” from our initial estimates and place a high importance on the accuracy of our delivery target dates.
At the heart of it though, this isn’t about moving faster. Everyone has, in their heads, an intuition about how complex a task is, and therefore how long it should take to complete.
Have you ever noticed that anybody driving slower than you is an idiot, and anyone going faster than you is a maniac?
- George Carlin
We are all the protagonists in our own story, and velocity is relative to that perspective. From the developers to their manager to the executive stakeholders to the cross-functional partners: everyone has their own biases and experiences that influence how fast they feel the team should be able to execute. And, left unchecked, this disconnect will lead to an incredible amount of friction and discontent within the organization.
Closing the Gap
There are three main ways to close this gap between intuition (what does my gut say about how long this project will take?) and reality (how long is it actually going to take?)
Capabilities & Efficiency
An important aspect of improving the software development process is to improve the software development experience. Does the team have the right tools for the job? Start by evaluating the state of the team and their capabilities in three main categories:
Technology: Are there areas of the code that are error-prone, problematic, or not well-understood? Are features difficult to test and validate? Is the team prevented from using new, modern tooling in their development process?
Processes: Are there specific handoffs that are inefficient? Is communication not happening as effectively as it should? Is the team regularly pulled away to work on something unplanned?
People: Are there certain skills on the team that are missing or underdeveloped? Are there performance management issues that have not been addressed? Are there a limited number of “subject matter experts” who need to be frequently consulted?
Identifying these opportunities for improvement is a hands-on exercise that takes time and persistence. Fortunately, the existing team meetings can be useful for gathering these signals and starting to notice patterns. Standups and Retrospectives, especially, are great opportunities to identify small, recurring snags that the team is facing. Some of the techniques in Agile as a Communication Framework can help, since daily commitments to each other can serve as early indicators of tasks that are taking longer than originally expected. Taking note of these day after day — and week after week — can yield inspiration for areas the team can start to remedy.
Transparency
A powerful technique to close the intuition gap is to increase transparency. Knowing that everyone involved or invested in the project (at any proximity) has their default and unexamined intuition of how long it will take, it is important to continuously communicate to build (and rebuild) alignment. Depending on the existing state of the relationships, establishing this open dialog will either be constructive or delicate. This is part collaboration, part education, and part negotiation — and it takes a skilled and experienced communicator to help drive this.
There are countless frameworks and well-defined processes to bring structure and data to these conversations. Ceremonies like standups, demos, scrum-of-scrums, or even a weekly email can bring everyone together. Artifacts like burn-down charts, status reports, and launch plans can be conversation starters.
James Stanier’s post on The Engineering Manager is a fantastic read that offers additional strategies for bringing people into the conversation:
Broadcast. You could publish a regular newsletter, written for a non-technical audience (e.g. the commercial mailing list), describing the progress in your team, division or department. It sounds simple, but in my experience, people really engage with these communications and enjoy reading them. Especially so if they are light-hearted and entertaining.
Invite outsiders in. Always give other departments the chance to engage. Invite them to be stakeholders in your projects and demos. They will then better understand the pace of progress and the difficulty of technical challenges.
Be social. Hold regular catch-ups with those in your network to discuss everything that you’ve been up to. Ask their advice candidly as a peer and friend.
Offer Q&A. Ask to drop in on existing non-Engineering meetings where required as a guest. You could even set up your own regular session with the rest of the business.
Sequencing
Unforeseen situations arise from every project, and these discoveries and detours are key contributors to the intuition gap. Some teams will pad for “not knowing what we don’t know yet,” but this is, at best, a blunt instrument and makes it difficult to be as transparent as possible. Another approach, well-defined in Shape Up, is to treat work like a hill and address the uncertainty and risk as early as possible.
When planning a project, the team should do an audit of the project for the biggest unknowns, riskiest integrations, dependencies, unclear requirements, and siloed knowledge. They can then create a series of prerequisite tasks to help mitigate risk and bring clarity to the work that needs to be done. Going hand-in-hand with Capabilities & Efficiency, this is also the time to identify parts of the technology and process that could introduce friction and risk. This exercise also helps teams with Transparency, as these conversations can invite a constructive collaboration with all interested parties across the organization.
By front-loading the risk and uncertainty, the team has also accelerated their opportunities to pivot and reassess the scope and estimates of the work.
Underlying the pressure and scrutiny that all engineering leaders face to “move faster” are a set of assumptions held by others. These assumptions are instinctive and based on an intuition that is deeply personal to every individual. This gap between intuition and reality can breed distrust and frustration within the organization. As leaders, it is our responsibility to ensure that our teams have the right tools for the job, that we are communicating effectively, and that we are structuring our projects in a way that gives us the best opportunities for success.