Does Your Software Project Have Scope Creep? The Answer Is Always No.

In 2020, the article The Impact of Scope Creep on Project Success: An Empirical Investigation presented this chilling finding: “Every year almost 50–60% of software projects end up in partial or total failure…According to the 2018 project success report, 70% of the organizations have faced software failure…[s]uch that they are over-budgeted, late or have fewer function[s] or feature[s] than required. Different studies have provided various factors responsible for project failure, but scope creep is listed as one of the most prevalent. Scope creep is defined as ‘any change in project scope or pressure to deliver more than what was agreed between customers and vendors initially.’”

Numbers like these are disheartening, and they contribute to the prevailing belief that scope creep is a dreaded, negative problem that must be “managed” above all else. But does the scope of a project truly change over time? Or is it that expectations surrounding the project are subject to micro- and macro- evolutions? Is it possible that the true scope of a project is rarely understood at the time that customers and vendors are making their initial agreements? What can we do to better understand the composite elements behind what is commonly known as “scope creep”? Let’s ask a 3000-year-old dead guy.

One of the Greatest Discoveries in Archaeology

In November 1922, Egyptologist Howard Carter uncovered a step leading down into the rock of the famous Valley of Kings. This step grew into a 10-year-long excavation of the tomb of King Tutankhamen, the most intact royal burial from ancient Egypt.

Howard Carter started with a wealthy sponsor, who expected him to find small caches of burial goods in the already heavily-excavated valley. The discovery of a full royal burial dramatically increased the number of stakeholders—the Egyptian and British governments, multiple museums, even The Times newspaper. Over the course of the next decade, volatile politics, change of project ownership, unforeseeable practical challenges, a growing team, and, yes, even user demands all made impacts on the excavation. Sound familiar?

Conflicting Expectations

Software building involves many stakeholders, and they all have needs. Unfortunately, the negative concept of scope creep typically sets the needs of clients and project teams at odds with one another. For example, the client wants the best solution at a reasonable price but may end up asking for more than what was originally agreed upon. The scope is perceived as expanding, or “creeping,” because the team expected or assumed that the client knew exactly what they wanted and would not ask for more.

Similarly, the project team aims to meet realistic goals and stick to the plan, but they might encounter challenges that require extra time and budget. Any overages are seen as scope creep if the client expected that every assumption made at the beginning of the project would prove to be exactly true.

Software contracts are often formed at a stage when there's minimal information available about the project's requirements and delivery timeline. Unless the project has been done before, the contract is essentially built on best guesses. These guesses, albeit well-informed, are still guesses, and they end up forming the basis of the contract.

The Cone of Uncertainty

Steve McConnell’s book Software Project Survival Guide presents how estimations become more accurate as uncertainty decreases over the lifecycle of a project. This is illustrated by The Cone of Uncertainty.

At the beginning of software projects, there's often a lot of uncertainty, and many requirements might not be clear. For example, at the beginning of a project, it may be clear that there needs to be a login page and users need to be able to manage their profiles. However, only later does it become clear that in addition to managing their password and email, the application also needs to support image uploads for profile pictures. Perhaps users need to be able to set up their account and log in with third-party accounts like Microsoft or Google. These details add complexity that may not have been included in the original requirements.

This means that any early estimates for tasks can be quite rough and sometimes very off the mark—as much as plus or minus 40% according to McConnell. As the project progresses, new estimates become more accurate. The true scope of the project becomes increasingly clear, and by the end, your estimates can be exact. Anything prior to that is uncertain.

Our man Howard Carter experienced The Cone of Uncertainty at its widest state from 1917 to 1921. He suspected there was another tomb to be found, but others thought The Valley of Kings was exhausted by previous excavations and grave robbing. Tutankhamen was virtually unknown and not expected to have had a royal burial, and four years of digging had yielded nothing of note. Carter’s sponsor had to be convinced to pay for another season of excavation. In 1922, the cone began to narrow when the tomb was discovered. But when entering the first room (of what would be four), Carter discovered more objects than he or his initial stakeholders had ever expected to find, and that was just the antechamber!

Discovering the Scope

Carter was a dedicated archaeologist, not a smash-and-grab opportunist as so many others were at the time. He was committed to the preservation and restoration of the objects found in the tomb. As his discoveries grew, so did his team. He ended up needing specialists in chemistry and botanical specimens for textile conservation, architects for scale drawings, expert language translators, and a specialized photographer. He needed extra foremen and porters to transport the 5000+ objects out of the tomb, and he needed supplies and man-hours exceeding anyone’s initial expectations. From a project management viewpoint, the scope of this dig was expanding wildly. This wasn’t “scope creep” so much as “scope attack.”

In a modern software context, a project following these lines would be considered a disaster and would no doubt be recorded as one of the many “total failures” by an empirical investigation. But did the project scope actually expand? Or was the true breadth of the project simply not known at the outset?

Evolving business needs, technological changes, and integration requirements are almost never fully understood at the beginning of a software project, but they all have a major impact on the final scope. Regulatory changes, market shifts, advancements in technology, and changes in company strategy are equally unpredictable. And sometimes, you simply don’t know what you don’t know.

One of the biggest unpredicted challenges Carter ran into was King Tut himself. His mummy was encased in three nested coffins. At the time of burial, both the inner coffin and the mummy were coated with ointments. Over time, these ointments solidified into a hard resin, bonding the mummy and its adornments into a single mass that stuck to the base of the innermost coffin. This, in turn, was stuck to the bottom of the middle coffin. Over the centuries, the ointments underwent a chemical change that caused the linen wrappings of the mummy, and even some of the mummy's tissues, to become carbonized. This process made Tutankhamun's remains extremely delicate.

Untying this little knot set Carter and his team back a whole year. Could this rightly be called scope expansion, or scope change? No. King Tut had been a stuck lump for centuries, the team just didn’t know. The only thing expanding or changing was the team’s understanding of scope that had existed the whole time.

This type of unexpected challenge is common in software building. Sometimes there are existing data sets that must be accommodated but were overlooked during the planning phase of the project. Once they are identified and examined, it turns out they are poorly structured and require significant additional programming. These data sets were always part of the scope of the project. The stakeholders just weren’t aware of them at the beginning. Unfortunately, you can bet that more often than not this type of normal project evolution gets negatively labeled as “scope creep.”

How and What vs Why

Scope shouldn’t be just about the work (how) and the end-product (what)—it should be about the desired business outcome (why). The Why should be constantly revisited and used as the true measuring stick against the project scope, rather than a collection of previously determined best guesses. At TSG, we use Lean Startup Methodology to enable us to expose and validate assumptions, empowering our clients to stop treating guesses as guarantees.

For example, a client may decide they want to replace their outdated internal app. Their Why is to increase employee engagement and efficiency. If they are doing it right, user feedback will be an early stage of the project.

As users interact with a developing software product, they may identify additional features or changes that they believe would enhance the product. Additional user-driven features might appear to be a form of “scope creep,” but if you are clearly focused on your Why and not just your How (budget, timeline, initial best guesses, etc.), they may prove to be exactly what you needed.

If you were overly focused on avoiding “scope creep” by disregarding user requests for changes, you could even end up working against your desired business outcome—to engage and equip your employees more fully.

Back in the 1920s, King Tut’s tomb inspired an unprecedented spike in public interest. “Tutmania” swept across the Western world, tourists flocked to the excavation site, and high-profile visitors demanded entrance into the tomb. While this slowed Carter’s How (working undisturbed in the tomb, with items safe from manhandling by non-experts), it ultimately resulted in more funding for the endeavor. More funding meant that the objects in the tomb could undergo extensive conservation and thus be available for generations to come—fulfilling the Why of the whole excavation.

A Positive Take on “Scope Creep”

So how do we shift our understanding of scope so as not to feed into negative concepts that pit clients and teams against each other?

First of all, let’s stop using the words “scope creep.” Scope doesn’t creep; it is simply known or unknown to different degrees during the lifecycle of a project. It’s an ugly label that shouldn’t be applied to clients who, during the business process, learn what they actually need, and it equally shouldn’t be applied to teams discovering how to build something that fits their client’s needs.

Secondly, we need to acknowledge inevitable communication gaps and catch them quickly via short feedback loops. What a client says they need may not be what they actually need, and what a team builds may not be what a client said they needed. By building in small increments and getting client and user feedback early and often, we close those gaps and narrow the cone of uncertainty.

Finally, we need to expect our understanding of the scope to change. Most of our assumptions about the future will be wrong. Most of the client’s assumptions about the future will be wrong. Rather than casting blame or using defeatist terms, we should aim to learn and build together.

Following these principles will create better products and serve desired business outcomes more fully. Discovering that a project’s scope has expanded beyond our original assumptions isn’t necessarily a bad thing; it could be the beginning of discovering something better than anyone initially expected.

Let’s try to view “scope creep” the way Howard Carter did when he first breached the sealed doorway of King Tut’s antechamber: “At first I could see nothing, the hot air escaping from the chamber causing the candle flame to flicker, but presently, as my eyes grew accustomed to the light, details of the room within emerged slowly from the mist, strange animals, statues, and gold—everywhere the glint of gold.”