Summary of Decline & Fall of the American Programmer (1992) by Edward Yourdon

Motivation

American Programmers currently have a lot of anxiety about LLM-related job loss and a recent resurgence of job offshoring. In many ways, this environment is very similar to the early 1990s, when pundits argued that special CASE tools would generate source code from diagrams and the industry was worried about foreign competition from Japan and India.

I believe that studying the past is helpful for thinking about today, so to this end, I recently purchased and read Decline & Fall of the American Programmer (1992) by Edward Yourdon. Yourdon was one of the leading theoreticians around Structure Programming. He grew up in the IBM mainframe data processing universe, and he had a deep interest in software engineering topics, which for him meant things like requirements analysis, software paradigms, and the software development lifecycle.

Yourdon read a software engineering book a week, he ran a newsletter called the American Programmer, and he was very active in online debates. He was known for 1980s "viral content" and "spicy takes."

Core Argument: The Decline of the American Programmer

Published at the start of the 1990s, Edward Yourdon's Decline & Fall of the American Programmer offered a provocative critique of the American software industry. At a time when many considered the U.S. the global leader in software development (evidenced by a market share increase from 70.8% in 1982 to 73.2% in 1987, per the July 1988 Communications of the ACM), Yourdon made the provocative contrarian argument that complacency and stagnation would lead to the downfall of American software leadership during the 1990s.

Yourdon wrote that American programmers would "suffer the same fate as the dodo bird and the dinosaur" due to rising foreign software powers such as Japan and India, who he believed had several key advantages:

Cost

Yourdon highlighted that foreign software engineers, particularly from Japan and India, cost one-fifth to one-sixth as much as American programmers. Advances in telecommunications and widespread English proficiency eliminated traditional barriers of distance and language.

Productivity

Yourdon identified several factors here:

1. American Programmers are lazier than foreign competition. They have a sense of superiority and entitlement and in fact work less than the self-reported 50 hours a week they claim. Software developers in rising powers work substantially harder.

2. American Programmers are obsessed with new technologies, so they have a slight advantage around adopting new productivity-enhancing tools, but this isn't that important because "There is no such thing as a Silver Bullet!"

3. American Culture is extremely individualistic, has a "cowboy" culture around software, and valorizes "loner" hackers. This is a cultural disadvantage for large-scale software projects requiring hundreds or thousands of developers. This also results in a cottage industry mentality where software professionals think of themselves as artists or craftsmen and instinctively dismiss attempts to formalize software processes, software methodologies, etc. In comparison, Japanese software teams are much more collective in mindset and have disciplined, team-oriented practices. They are more open to quality improvement techniques (like those pioneered in Japanese auto manufacturers)

Quality

Americans prioritize speed and features over quality. In contrast, Japanese software teams have 1/10th the number of software defects.

Expected Impact of these Advantages

Because foreign software teams had advantages in all three of these areas, and because many of the key technologies of the 90s such as CASE (Computer Automated Software Engineering), build upon mature process management, Yourdon expected loss of American software leadership during the 1990s, leading to massive unemployment of American programmers, systems analysts, and software engineers. He believed this would spread to other industries given how critical software had become in electronics and consumer devices.

He pointed to the fact that US citizens had fallen below 50% of computer science PhD students at US universities in 1992 and the fact that undergraduate majors in Computer Science had declined during the 80s.

Additionally, some foreign competitors, like India, lacked domestic mainframe software industries and thus were free from maintaining legacy systems. This gave them the chance to "leapfrog" the US.

International Competition

The book identified Japan and India as the primary emerging threats to U.S. software dominance in 1992, with Russia and China likened to "sleeping giants." State-sponsored initiatives, such as Japan's Sigma project and European programs (Esprit, Eureka, Alvey), underscored growing global rivalry and the role of governments in sponsoring growth of software industries to take on the USA. Europe is briefly mentioned here, but Yourdon seems to not consider European software a meaningful threat to US dominance.

Possible US Countermeasures

For the US to be able to defend its software industry, Yourdon expressed that American software organizations would have to become "world-class" by the mid-1990s. In this context, a "world-class" software organization was defined as having a 10x productivity advantage relative to their global competitors.

The remainder of the book details a variety of areas of improvement to help an American software organization become "world-class" during the 1990s to head off foreign competition.

Areas of Improvement

Peopleware

Much of this content is pulled from the book Peopleware. Yourdon saw this as the biggest potential area for improvement. He stated that Peopleware could cause a tenfold productivity improvement by itself. In contrast, all other factors are each only a 30-40% improvement.

Because the author believes in 10x Developers, he argued that companies should focus on recruiting top talent like professional sports teams. Additionally low-performers and malingerers should be fired because some employees are net-negative contributors.

Yourdon also argued that American programmers should massively ramp up education. In an appendix, he listed 80 suggested books all programmers should read. He suggested reading one complete book per week to be able to level up in time to become "world-class" by the mid-1990s.

Fourth-Generation Languages (4GLs)

Yourdon was generally skeptical of programming language improvements. He believed software organizations mostly struggle with defining and analyzing requirements, which can't be helped by programming languages.

Additionally, he expected software engineering would gradually move away from writing source code by hand. Requirements and architecture would be created using CASE tools, and this would generate COBOL or some other language behind the scenes.

CASE (Computer Automated Software Engineering) Tools

Yourdon described CASE as an expansive vision for the future of software engineering. The broadest vision was that software organizations would be able to encode and store the problem domain, schedules, resources, budgets, work charts, strategic plans, requirements, project planning documents, software architecture diagrams, etc. as metadata in a CASE repository. This would include a set of rules (predicates) called a metamodel, which could be evaluated (as in Prolog) to check for correctness. The metadata would eventually be used to generate programming language source code, automating the task of coding.

CASE tools also included so-called "Lower CASE" tools that we now think of as developer tools: interactive debuggers, interactive compiler language servers, IntelliSense, automated refactorings, static analysis, fuzzing, etc.

Yourdon highlighted how challenging and expensive it was to adopt CASE. The per-seat license for CASE tooling could exceed the annual salary for a developer. Additionally, it could several years to learn a specific software methodology and the associated CASE tools, causing the business not to realize an ROI until the 18-month mark.

Given that CASE automated processes, it was unsuitable for immature software organizations. Yourdon discussed the SEI maturity model that delineates five levels of software org maturity (initial, repeatable, defined, managed, and optimizing), and concluded that US software teams were nearly all Level 1, which meant they were not ready for CASE compared to Japanese software teams.

And Other Topics...

The author also had chapters discussing possible improvements around software metrics, software quality assurance, software processes, software methodologies, dealing with brownfield legacy systems, and code reuse. I do not believe these areas merit notes because they are similar to other publications.

Conclusion

While Yourdon's predictions didn't entirely come to pass, there was substantial offshoring from the US to India during the 1990s. Banks, insurance companies, and other Fortune 500 type companies worked with IBM consultants to ship maintenance work for legacy mainframe software to India. However, the microcomputer and dot-com software boom occurred simultaneously in the US (perhaps anchored by US venture capital). In this manner, the US shifted to "higher value work." COBOL maintenance went to India, but new client-server work stayed in the US. While IBM and the Fortune 500 offshored, the Internet companies grew into Big Tech and realized Yourdon's vision for "world-class" organizations. Today Big Tech in the US is obsessed with metrics, continuous improvement, "bar raising," etc. They've also implemented a great many of his suggestions.

Immigration has also played a huge role in enabling the US to retain software leadership by scooping the best-and-brightest of its competitors, particularly China and India. Yourdon's cultural criticism of US software engineers sounds similar to some of Vivek Ramaswamy's cultural criticisms of native-born US workers. This suggests that some of the cultural differences between countries are now fault lines in the US culture war.

India has massively grown around software, but it seems to have struggled to move beyond services and offshoring into building leading products. Japan seemingly never was able to break into software and generally isn't discussed as a threat any longer, but the Chinese "sleeping giant" has awakened and now elicits angst similar to how Yourdon described Japan. Due to CCP policies and the Great Firewall, China does have its own Big Tech, research universities, and leading software products comparable to the US.

Yourdon was largely incorrect about Computer Automated Software Engineering (CASE). American software engineering today seems as averse to paradigms, processes, methodologies, as ever. However, the "Lower CASE" tools did spread throughout the US in the 1990s via IDEs, object-oriented languages, etc. Visual Basic via Visual Studio stands out to me as a combination of Rapid Application Development with many of the "Lower CASE" functions built into Visual Studio. Outside of interface builders, code generation seems to have fallen by the wayside. After the rise of OOP and UML, diagramming fell out of favor entirely and is a common punchline in developer jokes.

Despite the industry aversion to "armchair astronauts," the FAANG companies have gradually formalized and automated their own processes, but it's unclear if there was much influence from the literature that Yourdon advocated studying via his reading list. Each FAANG seems to have high custom automation around its code reflecting its bespoke process, but occasionally former engineers cross-pollinate some of these practices via open-source and managed SaaS offerings.

After 30 years, the "Upper CASE" dream of code generation from planning documents (via prompts, not docs) has finally had a resurgence due to LLMs. However, the CASE literature seems unknown to the folks developing AI products, so this is mostly reinvention.

Interest in offshoring appears to have picked up again after organization got used to remote collaboration during the COVID-19 pandemic. The number of countries pursuing these roles have massively increased compared to the 1990s.

Thirty-three years later, and the US still has leadership, but the underlying stressors he identified are still present. It's an open question if Yourdon was incorrect in his assessments or if he was simply a few decades ahead of the curve.