AG's Report on our Largest Transformation: Delays and Costs in the Spotlight
AG's Report on our Largest Transformation: Delays and Costs in the Spotlight
The Struggle for Timely Public Service Reform
Happy Tuesday, Transformation Friends. Another week, another opportunity to go Beyond the Status Quo.
This week, we’re looking at another of the recent reports by Canada’s Auditor General that's particularly relevant for those passionate about public sector transformation: Report 8—The Benefits Delivery Modernization Programme, which scrutinizes one of the government's most significant initiatives to modernize services.
The audit of the Benefits Delivery Modernization (BDM) programme raises red flags over persistent delays, escalating costs, and possible shifts in government focus that could threaten the programme's transformative goal: to enhance how approximately $150 billion in benefits are delivered to Canadians through the critical social programs Old Age Security (OAS), Canada Pension Plan (CPP), and Employment Insurance (EI).
The AG shines the spotlight on one critical finding:
Delays and cost overruns could jeopardize the modernization programme’s transformation plans to improve benefits delivery
Today, we'll walk through the report's key points and then discuss the challenges of IT cost estimation in large-scale transformations, as well as the crucial role of building momentum—that is, the importance of getting things started.
Again, before we start, a disclaimer: I work within the Office of the Chief Information Officer of Canada at the Treasury Board Secretariat, overseeing some of the most significant modernization and transformation efforts. Today, I’m giving an overview of the report, and at the end, I’ll add my take on a few aspects. These insights are purely my own and don't reflect any official stance of the Government of Canada or the OCIOC on this audit.
Now that we've got the formalities out of the way, it's time to settle in with your morning coffee. Let’s get started.
AG’s Report on the BDM Programme
Let’s start with what the AG's report says.
In this section, I intend to provide a summary based purely on the report's content; I haven’t added any editorial or personal insights here.
The Audit’s Context
The systems responsible for disbursing Old Age Security, Employment Insurance, and the Canada Pension Plan benefits are between 20 and 60 years old. The AG points out that these aging systems are alarmingly close to failure, posing a significant risk to over 10 million Canadians dependent on them for daily necessities.
The modernization programme began its planning phase in 2017, with implementation starting in 2021 and an anticipated completion by 2030. Initially, the project was estimated to cost $1.75 billion. However, despite being more than halfway through its planned timeline, all three benefits still operate on these outdated systems.
The Audit's Focus
The audit centred on the Benefits Delivery Modernization (BDM) programme led by ESDC. This initiative is the federal government's most extensive transformation endeavour.
BDM’s primary objectives are twofold:
Migrate the systems delivering Old Age Security, Employment Insurance, and Canada Pension Plan benefits to a modern cloud-based platform.
Transform the delivery of these benefits and enhance the user experience, catering to various groups like seniors, Indigenous peoples, refugees, and those in remote locations.
The audit examined the current state of these systems, their risk of failure, and the programme's progress and challenges. It also assessed the government's approach to planning and executing such a large-scale IT project and the potential implications for millions of Canadians.
As part of the audit, the AG looked at the key players:
Employment and Social Development Canada (ESDC): The department responsible for delivering OAS, CPP, and EI benefits. It’s also accountable for delivering the BDM programme.
Treasury Board of Canada Secretariat (TBS): Provides strategic advice on the government's IT and cybersecurity. Participates in committees for the BDM programme and provides oversight through the Office of the Chief Information Officer of Canada (OCIOC).
Public Services and Procurement Canada (PSPC): Handles procurement for federal departments. Oversees contracts for the modernization programme related to system integration and technologies.
Shared Services Canada (SSC): Delivers and supports IT networks and digital services for 45 federal government organizations, including ESDC. Provides hardware, software, and data centre services.
The Audit’s Finding
Finding: Delays and cost overruns could jeopardize the modernization programme’s transformation plans to improve benefits delivery
The Benefits Delivery Modernization programme, initiated by ESDC, aims to enhance the delivery of crucial benefits to Canadians. However, the programme has faced significant challenges since its inception. The transformation, a core component of the initiative, seeks to improve the efficiency, service quality, timeliness, and accuracy of benefits delivery. This transformation is vital as it directly impacts diverse and vulnerable groups, including seniors, Indigenous people, refugees, and those in remote locations.
The findings and recommendations from the audit shed light on the current state of the programme and the path forward.
Key Facts highlighted by the report:
Since the programme's inception in 2017, ESDC has faced numerous challenges, leading to difficult decisions about the sequence of critical steps.
The current IT systems, which are decades old, are at imminent risk of failure. Delays in migration increase the risk of outages, potentially disrupting benefit payments.
Cost estimates for the programme have risen by 43% since 2017, with potential further increases predicted.
A significant review in June 2022 by TBS highlighted concerns in areas like timelines, scope, and costs. In particular, it found that schedules were aggressive and overly optimistic.
A case study by PricewaterhouseCoopers in November 2022 projected a potential delay of 4 years or more for the programme. It also indicated that the programme's costs could increase by $1 billion, potentially leading to a total cost of $3.4 billion, a significant rise from the initial estimate.
Several obstacles related to the transformation component could delay its implementation, such as unclear data on unclaimed benefits, undeveloped tools for detecting payment errors, and undefined fraud risk assessments.
Recommendations
In light of these findings, the AG provided two recommendations. It advised ESDC, in collaboration with TBS, to,
refine cost estimation and reporting methods for IT projects to enhance accuracy and reliability, setting a standard for future government initiatives; and,
ensure that any future changes (including cost estimates or timelines) do not result in the transformation component being curtailed or eliminated.
A few takeaways
Let's look at two points that stood out to me. First, let’s start with the AG’s recommendation on costing.
Costing
We’ve discussed some of the challenges with costing IT in this space before.
In particular, when IT costing goes wrong, it really goes wrong. The mean cost overrun for those projects that go wrong is about 4.5x. IT projects are the worst in this regard compared to any other project type, according to the Oxford School of Major Program Management.
Balancing the Inside View and the Outside View
The inside view and the outside view are concepts introduced by psychologist Daniel Kahneman to describe two different ways of forecasting the future.
The 'inside view' focuses on the specifics of a situation, while the 'outside view' compares it to similar scenarios to predict outcomes more accurately. What’s important is that we combine both to avoid overconfidence and align strategies with broader evidence.
However, too often, I hear this idea that everything we do is somehow unique: it’s the first time we’ve done it, or it’s a special set of requirements, or no one else does what we do. Flyvbjerg (2023) said,
Your project is special, but unless you are doing what has literally never been done before—building a time machine, engineering a black hole—it is not unique; it is part of a larger class of projects.
Flyvbjerg encourages us to think of our projects as “one of those” and to do our best to place our work into a reference class of similar projects to learn from them and spot and mitigate risks. Paradoxically, this focus on the outside view leads to a more accurate understanding of our project.
Anchoring and Confidence Intervals
The evolving cost estimates for the BDM programme are telling:
August 2017: $1.75 billion
April 2019: $1.9 billion
October 2021: $2.2 billion
April 2022: $2.5 billion (43% increase from original estimate)
November 2022 (possible increases noted by PwC): $2.7 to $3.4 billion
This progression highlights two key points: First, PwC's latest analysis suggests an overall increase of 54% to 94% from the initial estimate. Second, all estimates, except PwC's, present single figures.
If we assume BDM's overarching goals haven't shifted since its inception, the rising cost estimates suggest initial miscalculations, with each new figure moving closer to the actual cost.
Here's where the concept of anchoring comes into play. Anchoring is a cognitive bias that tells us that the initial estimate sets a benchmark in our minds. Subsequent adjustments, then, are influenced by that first number. Indeed, our expectations of the cost are.
What we see going on here is "anchoring and adjustment:" setting an anchor and adjusting it from there. The problem is that we’ll compare future cost estimates to that initial number, no matter how incorrect that initial number was. Even if the original estimate is significantly off, it still shapes our expectations around future estimates.
This leads to my second point.
The initial cost estimate, labelled a "rough estimate," is presented as a precise figure. Given the programme’s magnitude, this number likely carried a lot of uncertainties. It would have been more practical to communicate a range of potential costs—a confidence interval—rather than a single figure to acknowledge the inherent unpredictability of such large-scale endeavours.
As we progress forward in time, rather than anchoring and adjustment, we can instead refine this range. As the initiative progresses and we gather more information, we can narrow this range, eventually reaching a single point (at closeout).
Momentum (or lack thereof)
Reflecting on the audit's timeline, we see a significant gap: planning kicked off in 2017, but implementation was delayed and only began to turn in 2021. That's four years from thinking to building. Moreover, within the government context, 'starting implementation' doesn't necessarily mean tangible changes are immediate; it simply signals the beginning of efforts to produce the outcomes we're aiming for.
From what I see, the first substantial change, improving OAS, didn't occur until the summer of 2023—six years from the drawing board to the first real-world impact. That's an exceedingly long wait.
If it takes six years to make the first change, maybe that first piece of work is too big?
Momentum is critical in large-scale transformations. We must break away from the trap of endless planning—a pursuit of perfection that's ultimately an illusion—and start enacting real, observable changes.
The AG describes BDM as a programme, suggesting incremental benefits rather than a single bang at the end. The AG’s second recommendation hints at a crucial insight: we shouldn't pin transformation solely on the most complex parts of the work. Transformation is all the pieces, the big ones and the small ones. Starting with the small ones helps us to get started with less uncertainty and deliver value sooner.
Remember one of the key principles of programme management: "Bring Pace and Value." Here, pace is crucial. While planning is valuable, it doesn't inherently produce results. Looking back, I wonder what minor changes could have been initiated and completed alongside the planning for more significant changes. It may be that the transformation component was curtailed, to use the AG’s words, but focusing on the significant technology changes and missing opportunities to build momentum.
Wrap up
ESDC’s BDM programme's journey underscores some of the major challenges faced by public sector transformations: managing outdated systems at risk of failure, escalating costs, and delayed timelines. It also highlights the need for accurate forecasting, agile implementation, and safeguarding transformative visions amidst various changes and shifts.
As we digest the lessons from the BDM programme, let's consider these questions:
How might we better balance the aspirational goals of IT projects with the practical realities of their execution to avoid underestimation and over-optimism?
How might we embed agility into public sector processes to ensure that momentum from planning phases leads to timely and tangible outcomes?
How might we embed the original goals in all aspects of planning and execution and protect them against changes in management, budget, the journey and its outputs?
Until next time, stay curious and I’ll see you Beyond the Status Quo.
References
Flyvbjerg, B., & Gardner, D. (2023). How Big Things Get Done: The Surprising Factors that Determine the Fate of Every Project, from home Renovations to Space Exploration and Everything in Between. Signal.