Using Flow to Tame the Testing Bottleneck

Manage The Feature Factory Before You Shift To a Product Lab

It's 2025. It has become fashionable to dismiss Feature Factories and replace them with Product-forward operating models. However, it is challenging to become product-oriented, align around outcomes, and iterate based on evidence if you don't even have a well-managed feature factory with a good flow from idea to validation. This article is based on a paper I wrote in 2011, which remains relevant in many organizations that are still struggling to address their testing bottlenecks. Even those who have tamed that specific bottleneck can still benefit from applying the thinking and principles to what is currently limiting their ability to innovate and deliver great products that delight their customers.

Blast From The Past - Let's Go Back To 2011...

A growing number of organizations are seeking to become more agile these days. Several field case studies demonstrate that when an enterprise-level organization undertakes this endeavor, it typically experiences a significant stabilization or hardening phase, accounting for approximately 30% of the effort and schedule for the entire release. This is a result of the high transaction costs associated with running testing cycles, aiming to locally optimize the testing efficiency by reducing the number of cycles and pursuing scale. The lean/kanban practitioner will be concerned by this point and will not be surprised to learn that a significant amount of waste, ineffective handoffs, and rework usually ensue. The result is typically low predictability regarding whether this stabilization phase will end on time and whether the end result will meet release-level quality standards. This low predictability is the culmination of the big-batch thinking inherent in traditional software development approaches. It also drives a destructive cycle of increasingly longer stabilization phases, as we realize that we cannot deliver quality on time. We set a policy of a more extended stabilization feature freeze. We have less time to develop, so we run an even faster and more reckless development cycle.

Testing is becoming increasingly expensive, so we make it more efficient by running fewer cycles. Integration is considered costly and problematic, so we have a policy of integrating parts only after they are complete and of high quality. However, we overlook the fact that the real quality challenges of the day are integration aspects, which should be addressed as early as possible.

Organizations Are Trying To Tame The Testing Bottleneck

When an organization starts its transition to agile, several scenarios can emerge. The ideal alternative is for agile teams to take on all stabilization work as part of their definition of done, delivering working, tested software that is release-ready within each increment, iteration, or sprint. Furthermore, accompanied by massive test automation, this minimizes the effort required for regression and stabilization cycles, and the stabilization phase can be expected to shrink dramatically.

Agile in Complex Environments

Another scenario happens in more complex environments. When transitioning to full-scale agile teams, the regression effort and the scope of testing coverage required remain incredibly high, making it an impossible endeavor for the agile team to solve the problem as easily as before. First, it is not always feasible to complete all the required testing work on a story within a sprint. Second, even if it is possible, ensuring no regression impact due to those stories is a transaction cost that is either not feasible or not economically reasonable. Third, sometimes the work required to take a story to release-level quality is again more than a sprint can contain. Aspects such as Enterprise Readiness, Performance, and Security, when applied to an extensive legacy system, require specialized approaches, specific skills, dedicated labs, and considerable time. Typically, the agile team runs at a certain pace, delivering a certain level of working and tested software. Then, there is a process outside the team of taking that software and making it release-ready. The focus of this work is to assist organizations in managing this complex process by applying lean and agile principles of product development flow, thereby reducing the overall cost of quality and improving overall predictability.

A more evolutionary approach to agility

Another scenario involves organizations that aim to enhance their development processes through a more evolutionary approach (Anderson, 2010). In this context, the concept of an agile team doesn’t even necessarily exist, so the level of “doneness” of work is even lower at the outset.

How can we reduce/tame escalating release costs?

Lastly, we see several organizations that are looking specifically for a solution to this problem. They are not looking to become more agile per se, but they are focused on reducing the cost of releasing and improving their due date performance. Yes, they recognize that it will enable shorter releases and enhanced maneuverability, but this is a secondary benefit for them.

How can we better manage testing/stabilization in a complex context?

This paper presents a recipe that can be applied to these scenarios to reduce the cost of stabilization while improving its predictability by incorporating the principles of agility and flow, without assuming full team-level agility and engineering practices. The application of this recipe can lead to significant improvements in your testing processes, offering a more efficient and predictable approach to managing complex scenarios.

Leading Indicators For Improved Testing Flow

How will we know we have a good solution for our problem? We will have to improve in two key aspects. The first is to achieve a major reduction in the length of the stabilization/hardening phase. The second is to dramatically improve the predictability of the release due date while still meeting the release criteria. Identifying and implementing a good solution is crucial for enhancing the efficiency and predictability of your testing processes.

What causes the problem?

Let us begin our journey towards a solution by understanding the root causes of the problem. Figure 1 shows a current reality tree (Dettmer 1998), a visual tool used in the Theory of Constraints to help us understand the cause-and-effect relationships. This tool will guide us in identifying the key factors contributing to the inefficiencies in our testing processes.

Why is the stabilization cost so high? Primarily because we receive late feedback on our decisions, and we know that late quality is often more expensive than timely quality. Why do we get late feedback? Because we prefer to work in big batches due to the high costs of every stabilization cycle, and since it is the efficient and proper way to do things. We cannot achieve good coverage and certainty about the quality of the release without running a large batch once everything is ready. Additionally, features may not be ready for feedback until late in the cycle. Why? Sometimes it is because they are prominent features. Another reason is that we are not focusing on just a few features; we are working on many features in parallel (sometimes one feature per engineer), which means they will require a significant amount of calendar time due to excessive multitasking at the organizational level. Note that even if each engineer or team is single-tasking, if we are multiplexing stories or work from various features, then at the organizational or release level, we are still multitasking. (Why, by the way, do we typically see many features in progress? We believe it is more efficient, minimizes coordination overhead, gives more ownership to the Feature Owner, and many engineers prefer to run solo.)

All this means quality is “added” after the fact. As documented in Capers Jones' Cost of Quality (1980), the later we find and deal with defects, the more expensive it becomes. Late quality is costly.

With this dynamic in place, we have two options: protect the quality and compromise on due date performance, or protect the due date and compromise on quality, whether explicit or implicit. The team is driven hard, way beyond sustainable pace, and they find ways to "make ends meet." Bugs are often ignored, and quick and dirty local fixes are applied. This increases the unpredictability of how this saga will end and when.

Working towards a solution

So, what do we do about the problem? One option we previously discussed is to adopt an agile approach from start to finish. We will not outline the details of this solution here to maintain brevity. Note, though, that even in some agile team scenarios, leftovers of untested unfinished work will accumulate after each sprint. Beyond high stabilization costs, sometimes the Dev-Testing gap grows to such an extent that there is no option but to throw away features that were already developed, once it is understood that there is no way to test them in the release. When this happens, the morale of the team sinks. We usually see it happening around the time the developers are asked to help the testers deal with the remaining features that are not finished.

In addition, there is typically a drive towards developing functionality as fast as possible and leaving defects for later. This policy makes sense when a feature freeze phase is mandated. The main thing the developers can do during this period is fix defects. If you fix defects earlier, you deliver less functionality until the feature freeze. If you CAN defer defect fixing to after the feature freeze cutoff date, this is what you will do. This problem exists even in some agile teams.

Done is Done

The first step towards a solution is to consider a feature as "Done" only if it is completely finished and can pass its release-grade criteria. This means it has passed all of its test coverage, and all the defects that need to be fixed and verified to pass release grade criteria have been dealt with. (For project management purposes, we will track which features are done and which are still in progress.) The vital point to note is that even if stories or aspects of this feature have been completed in an agile team, the feature itself will be considered just a work in progress.

Limiting Features in Progress

Next, limit the number of features that can be in progress. Allow the starting of a new feature only if a previous one is really finished and release-ready, and the number of open defects is below a certain low threshold. This limit synchronizes the project. However, it doesn’t require the extreme synchronization that happens inside an agile team.

Dealing with the testing bottleneck at the tactical level

Developers will be asked to optimize features reaching Done rather than features reaching Coded. This might mean investing more in quality at the coder level (automated unit tests come to mind), reducing the number of defects delivered with the code by doing code reviews or pair programming, etc. Readers familiar with the Theory of Constraints will recognize this as exploiting the testing constraint – exploit being one of the 5 5 focusing steps–[The goal Cox, Jeff; Goldratt, Eliyahu M. [1986]

Avoiding the dangers of bottlenecks

In many cases, even after increasing the quality of coded features, a testing gap will still be apparent. Are we really asking developers to slow down and avoid coding new features? This might lead to lost capacity in the release and to lower capabilities over time if the developers “forget how to run fast” (Anderson 2004 ). We want to avoid this. Instead, let’s identify work types that don’t need to go through the bottleneck or that require less capacity from it. Choose work that is not testing-heavy, or items that can be tested without the involvement of the testing bottleneck. This introduces an internal operations factor into the priority filter, but clearly, this optimizes for the overall business value outcome.

Investing to elevate bottlenecks

To alleviate a testing bottleneck, create a backlog of engineering investments that will enhance testing capacity, typically by reducing the amount of work required per feature. A prime example is test automation, which developers can conveniently develop. This backlog solves the slowdown problem. It is also a more sensitive solution than “Have slack in the system” to the slack created by sprint commitment and Limited WIP. Involve your teams in building the Engineering Investment backlogs to increase their commitment to this approach. Experience also shows that a higher commitment exists when working to elevate rather than locally addressing a problem.

The Test Automation Pyramid helps elevate bottlenecks.

In the context of testing, leverage the recommendation of the “Test Automation Pyramid” (Crispin and Gregory, 2009) to achieve a win-win: Reduce the cost of test automation by doing more of it in layers that are cheaper to maintain, like unit testing and service/api-level testing rather than at the UI leve, and benefit from the fact that these layers are closer to the comfort zone of the developers. Extend the “Pyramid” to encompass all testing aspects required to make a feature release-ready. Considering this work as part of the flow and bottleneck optimization picture is key to achieving lower stabilization costs and improved release flows. Consider this a form of “Continuous Stabilization” or “Continuous Hardening”.

Safely transitioning to a shorter stabilization period.

Finally, choose when to go into stabilization (and stop developing new features) based on the metrics presented earlier. If the number of open defects is low and there is a limited amount of feature work in progress, it may be safe to proceed with new features. If defects are mounting and numerous features are in progress, the risk is high, and it is best to aim for stabilization as quickly as possible, or at least to halt the line, reduce the work in progress, and then reconsider.

Conclusion

Are stabilization/hardening costs and a testing bottleneck still an issue in your organization despite "doing agile"? It might be an indication that you're dealing with "Agile Theater". In this article, I outline an evolutionary-focused improvement by examining the entire flow of value and aiming for global optimization.

I've used this approach as the first step towards establishing a well-managed feature factory, or as a fix to Agile Theater.

By enabling fast feedback and smoother, more sustainable flow, we are building the foundation for a product operating model.

REFERENCES

Anderson, David (2010). Kanban, Blue Hole Press, Sequim, Washington

Dettmer, H. W. (1998). Breaking the constraints to world-class performance. ASQ Quality Press, Milwaukee, WI, pp 69-102.

Jones, Capers (3rd edition 2008) Applied Software Measurement, McGraw-Hill, New York, NY, USA

Goldratt, Eliyahu M. & Coxx, Jeff(1986). The goal: a process of ongoing improvement, North River Press, Great Barrington, MA

Anderson, David (2004). Four Roles of Agile Management, Cutter IT Journal.

Crispin Lisa & Gregory Janet (2009). Agile Testing: A Practical Guide for Testers and Agile Teams, Addison-Wesley Professional, Reading, MA

Reinertsen, Don (2009). The Principles of Product Development Flow: Second Generation Lean Product Development, Celeritas Publishing, Redondo Beach, CA