This article takes an analytically reductionist approach in a philosophical attempt to distill the fundamental problem of achieving adaptiveness in software engineering.

We are always striving to build software better and faster. We have terms and commonly recognized words for the symptoms of these problems—complexity, technical debt, pain points, developer friction. The list goes on. But perhaps it is worthwhile to dive deeper into this introspection. We need to analyze these symptoms and seek their root cause, if one exists. Understanding the fundamental cause can help to construct the right mental model and approach, not just to increase productivity in engineering organizations but to completely revolutionize the field of software engineering.

“Software Systems” are the scope of this discourse. This includes not just the silicon-based hardware and the data stored in 1s and 0s but also the technical builders who participate in activities that enhance these systems. Within this scope, every instance of Software System has two crucial parts: artifacts and processes. It is the constant interaction between these parts that drives the evolution of software systems, for better or worse.

Processes Link to heading

At the heart of software engineering, we codify business rules and processes into scalable and repeatable self-services through the collaboration of multi-disciplinary and skilled individuals. Disciplines and methods are established to ensure robustness in the processes which in term should delivery outcomes of a similar quality. This process transforms product visions and requirements into tangible assets that operate repeatedly on hardware with minimal manual supervision. These assets are stored and iterated upon in the form of artifacts.

The symptoms of poorly optimized software development processes are well-known: development friction. This occurs when the transformation of product requirements into tangible assets slows down or becomes more costly. The result is a tar pit for innovation, competitiveness and value creation.

Artifacts Link to heading

Artifacts encompass all static and tangible components of software systems—things we can put on a screen, where nothing changes without a trigger or an action. The most familiar type of artifact is application source code. However, artifacts also include documentation, infrastructure-as-code, user data, frameworks, programming languages, and the myriad of development tools.

The symptoms of problematic artifacts are equally familiar: complexity, unmaintainable codebases, and high inertia to change. These artifacts accumulate unoptimized cyclomatic complexity quickly, resulting in a viscous cycle, a downward spiral.

The Common Demon-inator Link to heading

What’s common between development friction and artifact complexity? The impact can be descirbed as the reduction of useful output given the same level of input. This phenomenon has a name: Entropy.

In thermodynamics, entropy measures the level of disorder in a system. The effect of entropy is that energy supplied to a system results in reduced meaningful work, dissipating in other forms. This is the Hydra we fight every day in software systems.

Entropy in Software Systems Link to heading

Entropy manifests differently in the two parts of software systems—processes and artifacts.

Entropy in Processes: Development Friction Link to heading

Entropy in processes is fundamentally about knowledge. Information ingestion creates knowledge, and applying this knowledge produces tangible artifacts. Entropy arises in two broad ways:

  1. Ingestion of Information:

    • You don’t know what you don’t know. A perfect solution may exist, but you are unaware of it. Alternatively, retrieving the required information for a specific scenario may have a high barrier, whether in parts or as a whole.

  2. Application of Knowledge:

    • Even with the correct knowledge, one may fail to apply correcly and efficiently. This imperfection is inevitable. Disciplines and practices aim to enforce standards at scale, but they cannot entirely eradicate this source of entropy due to our own perfect imperfections.

Entropy in Artifacts: Technical Debt Link to heading

Entropy in artifacts can be observed through the right measurements. Developers often encounter a psychological complexity threshold beyond which they struggle to maintain adaptiveness in any given code base. Entropy in artifacts increases through two primary mechanisms:

  1. Improper Clustering of Distributed Artifacts:

    • Software systems must be modularized to accommodate human cognitive limits and facilitate collaboration. However, as systems evolve and team compositions change, clustering can either become overly concentrated (monolithic) or excessively distributed (nano services), both of which introduce inefficiencies and create innertia for future change.

  2. Overly Complex Integrations:

    • Distributed artifacts require integration, both within modules in the same architectural layer (e.g., events passing between backend services) and across such layers (e.g., backend application connecting to a database). Excessive integration complexity contributes to entropy. Thus, the simplest system that achieves the goal is often the best solution.

Inverse Relationship Between Adaptiveness and Entropy Link to heading

As entropy increases, adaptiveness decreases. Systems with high entropy are less capable of evolving, pivoting, or scaling effectively.

Heuristics and Solution Model Link to heading

How can we contain or even reduce entropy in software systems?

  1. Reducing Entropy in Processes:

    • High throughput retrieval of highly relevant information to make informed decisions with trade-offs.
    • Minimize the surface area of human operations in all processes related to creating, enhancing, and maintaining software systems.

  2. Reducing Entropy in Artifacts:

    • Adopt “just enough” and “just in time” design at pivotal moments in a software’s lifecycle.
    • Strive for a singularity of abstraction in integrations to reduce complexity.

By addressing entropy in both processes and artifacts, we can create software systems that are adaptive to change.