Business managers and application developers face many of the same fundamental challenges. Whether the entity is a business unit, or a software application serving that business unit, the entity must be cost-effective to create and to subsequently maintain. If the entity is to endure, it must be able to adapt to unforeseen changes in a cost-effective manner.
From an external perspective, as a consumer we are interested in the service(s) offered by the entity and, if several equivalent services are available, how their properties compare. Is a service reliable? Is it a green/ethical service? Is the service geographically local to me, and/or in the correct jurisdiction? Is it competitively priced?
Figure 1: A consumer of a service.
As a consumer, as long as the selected service honors its description, I have no interest in the internal implementation details.
|As the consumer of a service I am interested in the advertised capabilities of a service. Do these capabilities meet my requirements?|
The Service Provider
As the service provider (a business unit or an application), the internal implementation details are fundamental.
Let us consider a modern application. Today, an application is no longer a single block of monolithic code deployed to a single large computer, but rather a set of small, interconnected software components that may span many physical or virtual computing resources, (the reason for this shift will become increasingly apparent as the reader continues). To meet service availability targets, this structure must be maintained. To meet new business objectives, the structure needs to be adapted. To achieve each of these, the structure must be understood.
This graph of interconnected software components is similar to the classic business organizational chart.
Figure 2: The service provider/system maintainer
Org charts allow us to quickly gain a basic understanding of the structure of a business unit. For example, from Figure 2 we can deduce that:
- The entity is composed of 15 components.
- We know the names of the components.
- We also know the dependencies that exist between these components, though we don’t know why those dependencies exist.
- While we do not know the responsibilities of the individual components, from the degree of inter-connectedness, we can infer relative importance. For example, component Tom is probably more critical than Dick.
Note that while we are responsible for the management and maintenance of this entity (the business unit or the application), it is unlikely that we created many, or indeed any, of the individual components used. Just as our customer is primarily interested in the capabilities of the service we offer, with no understanding of the implementation, we likewise have little or no understanding of the internal construction of the components we use. We simply require their capabilities.
Requirements & Capabilities
While we know dependencies exist, we have no idea as to why those specific dependencies exist. Also, if nodes are changed, how might this change affect those dependencies, the overall structure, and the resultant service to our customers?
Figure 3: How do we track structural change over time? The earlier system functioned correctly; the later system – with an upgraded component – fails. Why is this?
With respect to managing change, one might initially resort to versioning the node names. Changes in the structure would be indicated by version change on the affected nodes.
However as shown in Figure 3, versioned names, while indicating change, fail to communicate the impact of change, to explain why Susan 1.0 can work with Tom 2.1, but Susan 2.0 cannot!
It is only when we look at the capabilities and requirements of the nodes participating in the graph that we understand the problem.
Figure 4: An organizational structure: effectiveness of versioned names, capabilities/requirements and the use of semantic versioning.
Now we understand that Tom 2.1 requires a Manager capability. This capability was initially provided by Susan 1.0. However, at the later point in time, Susan 2.0, having reflected upon her career, decided to re-train. Susan 2.0 no longer advertises a Manager capability, but instead advertises a Plumber 1.0 capability.
|Once dependencies are expressed in terms of requirements and capabilities, then flexible substitution is possible. A node in the graph may be replaced by any other node whose capabilities satisfy the requirements of its neighbors.|
Communicating Change – The Role of Semantic Versioning
Hence capabilities and requirements provide the mechanism via which we understand structural dependencies. However, we are still left with the problem of understanding the level of impact caused by a degree of change; i.e. the Tom, Susan scenario just discussed. Via simple versioning we can see that changes have occurred; however, we do not understand the consequences of these changes.
- If an employee is promoted and/or re-trained (capabilities enhanced), are the dependencies shown in the org chart still valid?
- If we re-factor a software component (changing/not changing, internal implementation and/or a public interface), to what degree are the dependencies still valid?
If, however, semantic versioning is used (see http://www.osgi.org/wiki/uploads/Links/SemanticVersioning.pdf), the impact of a change can be communicated; the importance of this is increasingly being realized by the industry. For example, the Apache Maven project has recently (2013) discussed adoption of version ranges for artifact names.4 (While a welcome improvement, the concept is still flawed as the dependencies are still described in terms of the entities’ names.)
Semantic versioning achieves this in a completely generic manner via the following mechanism:
- Advertised capabilities are versioned with a minor.micro versioning scheme.
- We then collectively agree that – minor or micro version changes represent non-breaking changes for third parties; e.g. 2.7.1 ➞ 2.8.7. In contrast, major version changes; e.g. 2.7.1. ➞3.0.0., represent breaking changes, which may affect the users of our component.
- The interpretation of micro, minor and major are domain and context specific.
- Requirements are now specified in terms of a range of acceptable capabilities. Square brackets ( ‘[’ and ‘]’) are used to indicate inclusive and parentheses (‘(’ and ‘)’) to indicate exclusive. Hence a range [2.7.1, 3.0.0) means any Capability with version at or above 7.1 is acceptable up to, but not including, 3.0.0.
If these semantic versioning rules are now used with the previous organization chart, we can immediately deduce that:
- If Joe is substituted for Helen, Tom’s Requirements are still met.
- However Harry, while having a Manager Capability, cannot meet Tom’s Requirements as Harry’s 7 skill set is outside of the acceptable range specified by Tom i.e. [2,3).
Semantic versioning used with requirements and capabilities provides sufficient information to enable substitution of components while ensuring that structural dependencies continue to be met: our simple system, whether a business unit or an application is agile, maintainable and evolvable!
Turtles – All the Way Down
What happens as the size and sophistication of our business unit or service provider increases? The number of constituent components and the inter-dependencies between these components also increases. Usually, if left unchecked, the number of inter-dependencies increases much more rapidly than the number of components. The structure becomes increasingly complex.
Those of you who have already noticed the degree of self-similarity5 arising in the previous example may already have guessed the appropriate response to this situation. To make the larger system manageable we introduce a new level of structural abstraction.
Figure 5: An agile hierarchy: Each layer only exposes necessary information. Each layer is composite with the dependencies between the participating components expressed in terms of their requirements and capabilities.
As shown in figure 5, it is possible for an individual to be a consumer at one level of a structural hierarchy while being a maintainer at the next logical layer above.
For enterprise software, this process started in the mid to late 1990s as organizations started to adopt coarse-grain modularity as embodied by Service Oriented Architectures (SOA) and Enterprise Service Buses (ESBs). These approaches allowed legacy business applications to be loosely coupled, interacting via well-defined service interfaces or message types. SOA advocates promised more “agile” IT environments as business systems would be easier to upgrade and/or replace.
However, the core applications never actually changed; the existing application interfaces simply exposed as SOA Services, and/or message Consumers/Publishers. As the degree of modularity introduced was only one level deep, each post-SOA application was as internally inflexible as its pre-SOA predecessor.
Hence, with hindsight it should not be surprising that SOA – by itself – failed to deliver the promised cost savings and business agility.6 Yet, while not an end in itself, traditional SOA is a valuable step on the journey towards modular systems.
To be Agile
Agile systems need to exhibit the following characteristics:
- A hierarchical structure: Each layer in the hierarchy composed from components from the underlying layer.
- Abstraction: For each layer, the behavior of participating components is exposed via stated requirements and capabilities relevant to that layer.
- Isolation: Strong isolation ensures that the internal composition of each participating component is masked at each layer.
- Self-Describing: Within each layer the relationship between the participating components is self-describing; i.e. dependencies are described in terms of published requirements and capabilities.
- Impact of Change: Via semantic versioning the impact of a change on dependencies can be expressed.
Systems built upon these principles are:
- Understandable: The system’s structure may be understood at each layer in the structural hierarchy.
- Changeable: At each layer in the hierarchy, structural modularity ensures that changes remain localized to the affected components; the boundaries created by strong structural modularity shield the rest of the system from these changes.
- Evolvable: Within each layer components may be substituted; therefore, the system supports diversity and is evolvable.
|The conclusion is as simple as it is profound. Systems achieve agility through structural modularity.|