TL;DR

If you've spent any amount of time in this field you will have noticed a "lost-and-found" cycle of various truisms in software engineering. Through the years the discourse on agile patterns and practices has been both the beneficiary and victim of a recurrent novelty effect. It is continually paraded as the agent of change, when in truth it is really the result of how underlying changes have shaped its form. It's a rite of passage that everyone in tech goes through a "discovery catharsis" around agile at some point (hopefully) early in their career - and their opinions on lean software are often shaped by that initial encounter with that deeper understanding of how the world of team-built software works. But the risk - and what I've seen - is that with each iteration of software development populism another lean software development principle is lost to history, or worse - is actively discarded due to it's misalignment with the commercial interests of the process-du-jour.

This is allowed to happen through failure to recognize that agile itself is not an agent of its own design, but rather the outcome of deeper principals, laws that reshape it as the world of technology continues to remake itself. It's useful to take a moment to pause to review where agile came from and how it gained such a hold in the zeitgeist, and from that perspective determine which parts to reform, disband or retrench as current demands dictate.

Artificial urgency and superficial calendar management can be used to obfuscate that subject matter experts have lost control of a project. Just like a bad zombie movie, flashy action scenes can't hide the fact that they've lost the plot.

Before we can look forward, we must look back through the winding technology landscape and assess how Conway's and Moore's Laws wield fundamental power in the formation of software working groups regardless of the labels it currently enjoys.

How, Where and When Did Agile Come From?

The title of this post - by design - is a mixed metaphor that typifies what has happened to "agile". The term jumping the shark originated in pop culture and here it certainly fits the bill. It's the dual concern of trying to remain current and relevant while also attempting to reinforce its aphorisms - the tension between those two competing interests can fragment any long-lived concept. More to the point "agile" as a success story was more an accident of emergent history than a result of design intent. And if you play the "20 years later" video embedded below, Kent Beck himself will tell you.

While the agile manifesto might have sprung from a small conclave of senior software engineers, its outsized growth in popularity was a reaction to the environment in which it was conceived - Six Sigma primary among them. As the article on Six Sigma outlines - its popularity was piggy-backing on the wave of Welch-ism and its reflexive devotion to efficiency at the expense of innovation. eXtreme Programming was a dual citizen - both an outgrowth of and reaction to other practices at the time, and with its inception it showed some fairly specific ideas about what could and more importantly could not work in the process of adhering to its patterns.

And before either of those frameworks there was Kanban. Like Six Sigma, kanban grew out of manufacturing practice (some attributions go as far back as 1940's post-war Toyota). There were some presumptions on "virtuous alignment" of all parties involved - a projection that everyone in the working group had to be equally adept at both the delivery and the process in order for it to be truly effective. And like SCRUM, it more-or-less blindly accepted unspoken common goals as the driver for short-term delivery, expecting long-arc concerns to "work themselves out" in due course. Since the advent of those other agile-ish labels kanban has morphed into more of a chameleon, and much like other popular frameworks has become a Rorschach test for whichever role is running the working group.

How, Where and When Did Agile Go Sideways?

As with any open-ended question, the answers can vary widely depending on whom you ask. And their answer will tell you more about which parts of agile history they've lived and the surrounding environment that shaped their opinion. As with SAFe, the "filling the shape of its vessel" effect tends be billed as a virtuous result of adopting that process, but it also has some inherent short-comings when the underlying problem space in the software is really a symptom of a larger organizational issue. While this can happen when any agile label is adopted by a team, it's almost always a facet of SAFe project charters because SAFe pretends that it can perform systemic course corrections with process wrapped in ceremony. The inherent lie in "digital transformation" is that technology changes can carry the full burden of organizational change. This flies in the face of Conway's Law, which asserts that historical systems are ostensibly a reflection of the company's long-standing strengths and weaknesses.

Unfortunately, instead of dealing directly with spanning issues across working groups, these disposable tech monikers provide a form of cover for managers to whistle past or severely discount technical debts - a perverse incentive for both the buy-side and sell-side of that exchange. The "ask me no questions and I'll tell you no lies" social contract in these types of projects often doom them from the start. There's a famous survey that showed SCRUM Masters indicating that nearly half of their agile transformation projects fail. I'm not sure I'd adopt process or product that only had a coin-flip's chance of not working, regardless of whether it was the actual root cause of that failure. And more to the point, the central conceit is that they're attempting to solve for the wrong problem. And any proposed solution that ignores the potentially negative consequences of Conway's Law is a sure sign that its motivations are not founded in a company's core interest.

A Label Doesn't Automatically Inform the Process

Just because you refer to "agile" doesn't make it so. As I've mentioned before, much of my early inculcation with lean software engineering was informed by my time at a medical software device startup where XP was the "lighthouse" that guided how we worked. It was a common admonishment from my boss, Bob Cymbalski that you may choose to not carry on all practices, but you couldn't actively work against any one of them without consequences. And to be sure, while it was often discussed as an embrace of the burdens it presented, in the context of a medical software device those practices were also viewed as a shield. Whether a contributor came from engineering, compliance or the medical team it was understood that each person had to be equally invested in curating the process as much as the product.

Fancier names as a faster path to your wallet

In today's commercial process management landscape there's a kaleidoscope of labels and buzzwords designed to bedazzle the non-initiate. And while working groups will often brandish the newly minted (and often expensive) label of their process like a magic talisman, the actual practices can be performative or outright ignored. And just like "agile" itself this isn't a recent phenomenon. SCRUM has long been excused as a "lighter" version of XP, where the team had to guard for any shell games that may allow members to selectively discount or ignore whatever was deemed inconvenient. That's part of what's been lost over the years with "agile" as a term - the constant erosion. Simply glomming onto the phrase doesn't imbue software with superior properties. And far too often the misuse of agile has opened a side door for non-adept participants to unduly influence the outcome - to the detriment of the end product and its users. Without a healthy skeptical review of how processes are used the cadence and workload can be manipulated to keep very busy contributors from noticing that the wrong motivations are running the show. Artificial urgency and superficial calendar management can be used to obfuscate that subject matter experts have lost control of their project. Just like a bad zombie movie, flashy action scenes can't hide the fact that they've lost the plot.

OK - so how do we fix it?

The problem is us. The key, as I've been alluding to throughout this post, is that this is a human problem, not a technology problem. In many cases it's a situation where everyone is acting in good faith, but mis-appropriating which portion of work should receive priority. This is because day-over-day and release-over-release process is a sub-domain of broader application lifecycle management. One symptom of that can be found here...

While I don't disagree with the sentiment, it betrays the larger issue that the app in question has moved beyond the "move fast and break things" addiction cycle. This is where "agile" again might return to manufacturing for guidance - where a product goes through distinct phases - from design to prototyping and on to various levels of production. But due to agile's mixed history playing against six sigma that notion is often discarded, whether out of naivete, willful ignorance or outright pique. In the above example there's an implication that the product was beyond the prototyping phase and on into delivery and maintenance, and in software shops this is often cast as a shift in emphasis from "customer acquisition" to "customer retention". It's how the "guiding hands" in the process respond to this reality that reveals the level of capability maturity within the group.

Any software methodology worth its salt recognizes and actively balances the natural tensions between creativity and control.

If the company isn't mature enough to move beyond a reflexive rapid prototyping mindset then the process guiding the software will never mature. This is a symptom present in software shops of all sizes - where managers are so caught up on the feature merry-go-round (with promotions and advancement tied to that behavior) that the working pattern for the app cannot move forward in a way that supports the application's purpose. The formal terms for dealing with "paper cuts" - refactoring and re-engineering, two hallmarks of mature systems - never materialize. In that light it's almost inevitable that the customer's needs outgrow the working group's ability to fulfill its mission. Again - the foundering of a product in continuous roll-out of half-baked features is a symptom of a broader organizational problem.

Get Your Story Straight, Agile

It's an Achilles Heel that the agile label can so easily conform to the shape of whatever problem it's poured into. From my first encounter with it in the early aughts I found the "choose your own adventure" aspect to be off-putting. Even today - there are entire areas of engineering concern that focus solely on one XP tenet and no one bats an eye. And for those that do bat an eye, a quick acronym switch-eroo is all that it takes to get people on board. Mind you, I highly value Test Driven Development. I just happen to think it's one aspect of a larger solution space - and in fact this video says as much, while also describing how renaming "TDD" to "BDD" was the way that one notable adherent managed to get engineers to a clear-eyed view of the practice.

I don't mean to cast this talk in a negative light - it just happened to have the perfect anecdote to illustrate my point. In fact, I think the entire video is a worthy watch if only because Ian Cooper does such a great job of presenting the topic from an honest perspective. He's as forthright about the failures he encountered along the way as he was clear in showing the successful exhibits in the presentation. It's a hallmark of agile to be honest, sometimes brutally so, and embracing the course-corrections is valuable in creating better systems. He gets it - and showed how he eventually grokked what TDD is meant to be as a practice - a proxy for user interaction to validate the outward-facing elements of the solution. In truth - what he illustrates by accident of history underscored what I was taught early on - that you could choose to not explicitly embrace XP practices but to contravene them was trouble. While the focus here is on proper TDD, he unwittingly shows that other XP practices emerge as an accidentally virtuous consequence of honoring TDDs most earnest goals.

Essence, Accidents & Beyond

While at TherapyEdge there was regular mention of "the bookshelf" - a list of recommended readings for Bob Cymbalski's managers. One entry included The Mythical Man Month, which had a Fred Brooks essay that was the 1995 follow-on to the No Silver Bullet - Essence and Accident in Software Engineering, originally written in the late 1980s. The group spent more time talking about those two essays than any text on the shelf. And I think they still help to inform many of the concerns around software engineering today. A new favorite analysis of Brooks' comments is Mark Seeman's 'Yes silver bullet' which echoes some of my own quibbles with those works, and echoes many of my own concerns around the topic. While I understand the Aristotelian tool of bifurcating into "substance or essence" versus "accidents", in the realm of software design the two natures are not equal in size or impact. That's more obvious than ever today but was equally evident in my first manager role twenty years ago.

My biggest issue with "No Silver Bullet" at the time was the cursory, nearly dismissive mention of "management". I mean, I get it - the paper is about software design. But if we're going to talk about influences - the whims of the "product owner" has a much bigger influence on complexity than whether a developer decides to use memoization or tabulation to deal with (O)n concerns. That was just as true in the 90s as it is today. The other major concern I found had almost nothing to do with the article, and everything to do with sometimes willful disregard for what the essays were attempting to convey. Again, like the principals of agile software, adherents would sometimes play fast-and-loose with the "essential" versus "accidental" distinction as though they were in a hierarchy or tiered relationship. And to echo Mark Seeman's article there are so many more accidental concerns that it dwarfs consideration of the essential problems in software design.

As far back as my time at IBM Global Services (Y2K, y'all) I was classifying sub-categories of accidental complexity considerations that included Organizational, Institutional, Legal, Accounting and other complexities - each of which where orders of magnitude larger than "essential" software considerations. In many cases development teams would actively work against bringing those considerations to the table because "they weren't the customer". The excuse was "oh, that's an accidental complexity - we don't have to worry about that." If you hear sentiment like that in a meeting, recognize it is a giant red flag waving over your project.

Don't Let This Be Your Company

And like seeing yet another zombie horror story play itself out, my recent consulting experience has been littered with "digital transformation" and similar software forensics exercises. In some cases the clients were multi-national conglomerates operating under consent decrees from the federal agencies - and the root cause of the issues always stemmed from earlier engineering teams choosing to ignore those accidental complexities. This is part of the potential devil's bargain that can come along with misuse and abuse of refactoring and re-engineering. In many cases long-since-gone "talent" simply chose to ignore the work because it wasn't glamorous or required the team to face the real costs of those early decisions. That's "institutional complexity" in a nutshell, and is a pernicious technical debt.

That was part of my motivation in creating AD4S. All of those higher-order decisions that deal with these subjects have to be addressed and noted in a way that they can be tracked along with software engineering decisions. In a world with GDPR, CCPA, HIPPA, SOC 1/2/3, ISO 127001, FIPS 140-2 and myriad other acronyms and regulations on the horizon these and future concerns must have a home to be proactively faced by the working group. And it must be done regardless of whether they are deemed "essential" or "accidental" by those who only evaluate whether they can see their next job promotion in it. Regardless of how Aristotle would segment the problem space, it's all "the work" and must be addressed.

Here's a final tidbit to get you thinking: If facebook is already paying for marketing to plant their flag in front of this parade, where do you think your company should be? Regardless of whether your shop utters "Internet" in every sentence like facebook, it would be prudent to take note which way the wind is blowing.

This is a really great synopsis of the history of "agile" and its nearly-accidental roots. Kent Beck is not only patient 0 in this zombie apocalypse, he's also funny and delightfully self-effacing. I had no idea that he ever worked at facebook until seeing this video, found while researching this post. So it somewhat serves double-duty, both historical context and an interesting early bookend that pre-dates the facebook commercial above.

Rediscovering Healthy Tensions in Engineering Process

Any software methodology worth its salt recognizes and actively balances the natural tensions between creativity and control. Ensuring that those tensions contribute to the health and wellness of the system is the job of every contributor in the working group. By example, it's not supportable for engineering to continuously deliver fragile code that causes constant production triage issues in operations and support groups. Similarly it's also not supportable for managers to stoke artificial urgency into a project only to ceremonially plant their flag at the front of the parade when the group drags the work across the finish line. Addressing these adverse behaviors and potential negative outcomes must be the result of an active management posture. Simply applying "agile" as a label to the group's performative activities is not going to move the needle in a positive direction. Any time I find that a manager has lost their bearings I return to the "lighthouse" lessons outlined above - how control-obsessed dogma is just as detrimental as the reflex of anything goes code slinging, and sometimes a cliche can help bring the point home.

I will sometimes offer a gardening metaphor - beyond removing weeds and pruning plants to focus on producing fruit - you also have to rotate crops so as to not drain the soil of nutrients for future seasons. Pruning is refactoring. Crop rotation is re-engineering. And it requires more forethought than burying seeds and staking a scarecrow into the ground. To shift gears I sometimes mention working around electric vehicles. Most often I relay how dealing with variances in the range of vehicles could be significantly affected (up to 10%) by air pressure in the tires. And similarly, the difference between the interior climate control and outside temperature could also impact drain on power - and therefore the vehicle's range. The choices you make while operating the system have significant impact on how it performs. You may not want to think about putting air in the tires, but compare that to being stranded by the side of the road with a dead battery.

Choices have consequences, and in complex systems not making a choice can have equally broad implications.

I haven't come up with clever shark quip, but if anyone wants to make a suggestion I'm all ears. No matter how you choose to make yourself the hero of the story, put away the water skis and focus on the work that actually moves people forward with competency, context, honesty and trust. That is the true determinant of success with software in the 21st century.