It’s impossible to develop software without a bug or failure of some type — and every check to find and detect a bug costs something. Welcome to one of the greatest quandaries in development: how much should you spend on quality, knowing perfection isn’t possible, near-perfection is more expensive than it’s worth, and poor-quality is also tremendously expensive.
There’s a line where the value of detection and prevention is not high enough to justify its cost: this threshold, also called the “quality coverage” of a project, should be a clear line that’s agreed upon by all stakeholders at the onset of a project. But how?
Thankfully, there’s a metric for that. One way to frame the relationship between cost and quality is to use the aptly named Cost of Quality metric, or COQ. Discussing the Cost of Quality for a project can help determine when that threshold is crossed, so the entire team has the clarity on the quality coverage, or the, “We’ll make sure it’s at least this good” line. Exactly where this line falls depends on the project and the budget. You can imagine, for example, the quality coverage for a five-star hotel room is much different than the quality coverage for a campsite.
Let’s dive in.
What is the Cost of Quality?
The Cost of Quality metric has been around for years. It was first introduced in 1995 by Armaund Feigenbaum, a quality control expert, and had a huge impact on the way we think about and measure quality globally. In its simplest form, the Cost of Quality is how much it costs to ensure a product is of good quality, outside of production costs.
The overall COQ is a combination of the costs of a products poor quality (COPQ) and the costs of its good quality (COGQ).
COQ = COPQ + COGQ
We’ll break down both of those components in greater detail, starting with the cost of poor quality.
The Costs of Poor Quality (COPQ)
On Prime Day 2018, Amazon went down for about an hour. During that time, the cost to the company was nearly $100M in lost sales. At Amazon, they certainly know their Cost of Poor Quality — it was very high in 2018 and it’s probably even higher today. When you know your Cost of Poor Quality, it’s a lot easier to know if and when preventing poor quality (that is, investing in the Cost of Good Quality) is wise.
The Costs of Poor Quality are made up of two types of failure costs — internal and external.
COPQ = INTERNAL FAILURE COSTS + EXTERNAL FAILURE COSTS
- Internal failures are any defects found before a product is released by the internal team and their associated costs. This includes: defect finding, rework, retesting, root-cause analysis, bug fixing, retesting, scenario recreation, and redesign required to remedy defects.
- External failures are any defects found after a product is released or sold by the customer, plus the associated costs. These costs include: the costs to process complaints, customer service, returns, warranty claims, the cost of any loss of competitive advantage, the loss of customer trust or goodwill, damage to the company reputation, lawsuits, and company devaluation — plus the costs to remedy any of these things.
Why it’s hard to calculate the cost of poor quality
External failures are often much more expensive than internal failures because their impacts are broader and harder to quantify. While it’s pretty simple to calculate the actual dollar outlay for an engineer to find and fix a bug, it’s much harder to calculate the dollar value of a customer’s lost trust.
What’s more, only a small portion of the costs of poor-quality software are typically visible at first. There’s a lot more of the iceberg hiding underwater. It’s often pretty easy to spot external failures in the form of customer issues and customer service calls, warranty claims, and service outages. It’s harder to spot some of the less visible costs of poor quality including the work of understanding complex code, maintenance of poor quality software, the stress of crisis mode, staff turnover, and the domino damage to future projects that must be delayed or cancelled.
We’ve found that a metric that’s hard to see at first becomes a lot more visible simply by listing out, and then looking for, those defects. You can ask your team, take a look at your historic numbers including if costs are declining, increasing, or level over the life of a project, and make some estimates. This doesn’t need to be an exact science — is anything exact when you’re in the development process? The point is that you’re not driving your boat blindly into an iceberg.
The Costs of Good Quality
In 2017, Google ran 4.2M tests continuously and 150M tests a day, much of it on proprietary software designed and built by Google, for Google. That’s an impressive amount of investment in Good Quality.
The Cost of Good Quality is the other half of the Cost of Quality. Just like the Cost of Poor Quality, the Cost of Good Quality consists of two elements: the cost to detect defects and the cost to prevent defects.
COGQ = DETECTION COSTS + PREVENTION COSTS
- Detection costs are any costs associated with determining if a product meets quality requirements. This can include any type of appraisal or audit, such as measurements, inspections, evaluations, peer or senior reviews, tests, and the tools and systems needed to run and track those tests. You might also include the costs of defect finding, rework, retesting, root-cause analysis, bug fixing, retesting, scenario recreation, and redesign required to remedy defects.
- Prevention costs are any costs to prevent defects. These can include quality planning, precise functional requirements, project management, feature reviews, product reviews, process reviews, and even such prevention measures as hiring a great team, training that team, and supporting them with strong processes. We also think that management control costs fit into this category, things like reviewing contracts, establishing quality goals, setting gating/release criteria, and updating project plans.
Remember, there’s a project-specific line for how much quality is worth the costs to achieve it — and every stakeholder needs to agree on where that line sits.
Our Take: All stakeholders should align on the quality coverage
COQ metrics can help identify and reduce unnecessary costs. When the team has an understanding of the quality-cost tradeoff, you can put a dollar value to the Cost of Poor Quality and make smart decisions about the spending on the Cost of Good Quality to prevent future defects and reduce the “hidden costs” of poor quality software.
Don’t just want to advocate for avoiding poor quality, and ready to make the case for high-quality software projects? COQ metrics are awesome for earning executive buy-in. Using COQ, you can make a business case for smart investing in quality, especially when you slice the costs in these ways:
- The share of Cost of Quality in software development out of Total Costs
- Percentage of Cost of Poor Quality out of Total Costs
- Percentage of Cost of Quality out of Total Sales and Maintenance
