Book Review: The Phoenix Project
DevOps used to be a scary word. Largely because I didn’t quite know what it meant - I always thought of DevOps as strictly Ops, or system and network administration - and it seemed opaque and difficult. The Phoenix Project has cleared any preconceived notions about DevOps and its effect on an organization much like the main character Bill (and his trusty mentor Erik) did with his employer Parts Unlimited.
The book is a fictional story about a sinking company in disarray, with the Board of Directions looking to split the company up in an attempt to salvage shareholder value. They are losing marketshare to competitors, financial analysts have continually lowered the price target, and outsiders are calling for fresh leadership. The company has placed a large bet on an internal project codenamed The Phoenix Problem but it is three years behind and millions of dollars over-budget. The story’s hero Bill, Director of Midrange IT Operations, is thrust into a position he doesn’t really want as his boss and his boss’s boss (VP of IT Operations and CIO) are let go. Bill is interim VP of IT Operations and with the help of an eccentric prospective board member begins to whip the IT department and company into shape.
Murphy resides in Parts Unlimited and everything that can go wrong does go wrong. Bill begins to follow Erik to the manufacturing plants onsite at Parts Unlimited, where it is revealed that technology organizations can learn a lot from manufacturing operations. It becomes clear that DevOps has many parallels with manufacturing, such as:
- The biggest drain on output is the amount of work in progress (WIP). Extra WIP causes excess inventory and build-up of releases, typically resulting in bugs that cannot be discovered and patched quickly. It also contributes to multitasking and priority conflicts. More WIP decreases the actual amount of output.
- There are a very small number of resources that dictate the output of the entire system. This can be men (or women), machines or materials. This is the constraint, or the bottleneck. The constraint must always be managed to work at full capacity. Any improvement to a system that does not improve the output of the constraint is an illusion, as the constraint will still dictate the pace of output.
- Work must be standardized. Work must be repeatable and must not require any specialized knowledge that has not been disseminated elsewhere. This is closely related to the bus factor and is also related to point 2, as any non-standard piece of work becomes a constraint.
- To identify constraints, each task must have a bill of materials, much like a manufacturing order. This will give all the prerequisites needed for this piece of work. Work can be separated by those that require main constraints and other pieces of work that can flow unimpeded. It allows work to be organized in the most efficient manner.
The underlying theme of the book is that DevOps has such a profound impact on how a business can run and respond to constantly changing environments! The executives and non-technology stakeholders at Parts Unlimited are not aware that many of their headaches and stresses are contributed to by poor IT practices, and that their success is aligned with IT’s success. Bill realizes that a culture change is needed, and works with the managers of Sales, Product and Finance to ensure that they know how improvement in IT means improvement in their metrics and output, and that IT is a critical part to ensuring integrity of data and quick product iteration time. Bill makes this clear in when thinking about his discussions with Dick, the CFO:
No wonder Dick just has a vague sense that IT is screwing up - it’s a dull, throbbing ache that he can’t localize. Our next step is obvious: We must make those pains very specific and visible to convince Dick that IT is capable of not just screwing up less often but helping all of the business win,
IT was previously a drain of company progress and a place that was often citied as the reason for failure. This prevented upper-management from wanting to invest more resources into improving the IT operations, as they misunderstood IT as a necessary dysfunction. In pinning Sales / Financial / Product KPIs to actionable items in ITs world, Bill was able to illustrate to non-technology groups that their performance was inherently tied to ITs performance and that their support was needed to improve the visibility, budget capabilities and influence of company direction to ensure IT and the business could ultimately win. IT must find where it is underscoped in business goals (where the impact of IT was not obvious to a groups success) and make that clear - it stops IT from being a phantom menace that obstructs success to a critical part of the business machinery. The company culture must change to include IT as part of the value chain for the organization.
I sped through this book, enjoying the story format (I have not read many business / technology books with this format) and realizing how important DevOps and ultimately technology operation is to the success of a company. It turns out that the responsibilities covered by DevOps has always been a part of my workflow as a developer with small, startup companies. When I started my professional career, many of the concerns stated in the book had mostly been unknown to my generation of technology workers. Cloud computing was the default - who wanted to run a server themselves? - and we rarely had to think about infrastructure choices thanks to the elasticity of our infrastructure providers. Deploying code multiple times a day? Of course! As the single developer or on a team of six software engineers, we rarely had concerns about multiple deploys a day. The process was code, review and ship! Thankfully many of the ideals presented in The Phoenix Project have been ingrained in my developer mindset, and are usually things that I need to worry about as a developer. That means parity between test, staging and product environments; including infrastructure and hosting choices in application designs; ability to continuously deploy; being a steward over your code until it’s released into production.
However, I could still see some of the issues discussed in The Phoenix Project within my own organization. A few really stood out to me: 1) the constraint of a single individual who was the only one who knew parts of the system and 2) the poaching of developer time from other parts of the organization. These are situations that lead to many of the disasters that occur in the story, and were completely avoidable. It’s easy to fall into these trends, but they must be fought against to ensure that the technology group can work as efficient as possible for the benefit of the entire company.
The theme of the book (made clear by the subtitle) is that DevOps is embedded into the fabric of every company whether that’s clear or not. Most companies divide the technology organization into its own group, much like other parts of the business - however, it is the soul of an organization as it the thing that all other groups depend on to complete their work and move at their desired pace. The technology group cannot be quarantined to its own section of the company - it is a part of every decision, every process, and every success and failure a company has. The sooner a company embraces this fact and begins to not see IT as a necessary cost center but as the keystone to an organization, the sooner than company can begin to win as its IT department works in lockstep with other parts of the organization.
P.S.:
This is an entirely different post and something that I’d like to dive deeper into, but most teams I’ve worked on use a variation of kanban rather than hard-nosed Agile, with sprint accountability . Most startups can’t run this way as things are constantly changing, priorities shifting ever so slightly, unplanned work occurring, and BD / sales having large impacts on a quick turnaround time. This points to a trend that Agile and the DevOps ways have not been properly enforced in these teams, but certain segments have seeped into teams with the label of Agile. It feels a bit like a carbon-copied Agile flow, and I’m excited to dive deeper to gain a better understanding of what a pure Agile flow should look like.
