Agile SOA?

Benefits of SOA

It seems that everyone is talking about Service Oriented Architecture (SOA). All sorts of benefits are claimed for SOA, including:

• Cost savings by increasing reuse. The idea is that common services can be created and then reused whenever this type of service is required within the enterprise. Typical examples include a common authentication service and a common payment service. Those who have been involved in software development for a while will recognise that this was one of the objectives of object oriented technology. Object technology has a good record for achieving reuse within projects or systems but a bad record for achieving inter-project reuse.
• Cost savings by reducing the skill required to develop enterprise applications. If a palette of ready built common services is available, maybe they can be wired together by less highly skilled developers or even directly by business people?
• An increasing number of vendors of commercial off-the-shelf (COTS) products already have SOA or have stated that they will have SOA in the future. Maybe it will be easier to integrate these products with each other and with custom developed services?

I would suggest that the jury is still out on whether or not these benefits are actually realised in practice.

Isn't agile software development supposed to deliver some of these benefits anyway?

Agile software development directly addresses reuse in an even stronger manner than a non-agile object oriented approach. Agile takes a rigorous approach to avoiding duplication in code. This typically applies not only to the version of a system that is currently under development but also to future systems, built on its basis.

Agilists tend to frown upon a large amount of architecture and design work being done up front, instead regarding design as something that is part of what software developers do every day and architecture as an emergent property of a system. At first sight, this might seem to preclude the development of an SOA based system, which surely requires considerable up-front and architecture and design work? Well, as we will see later, maybe the two are not as incompatible as they might first appear.

Agile doesn't directly address the remaining supposed benefits of SOA. Is it possible to combine agile and SOA to create a best of both worlds approach? Well, the answer is a qualified maybe. Here are some possible approaches:

Strategies for agile SOA

1. Design an initial service model up-front. Do a small amount of design up-front to define the service architecture (including means of communication between components) and an initial partitioning of the problem space into services. Build the software using an agile approach, in business scenario focussed vertical slices, adding code to the appropriate service as you go. When necessary, introduce new services or remove ones which didn't pan out. Use a "do the simplest possible thing that works" followed by aggressive refactoring approach, to keep the flow of business functionality coming whilst still preserving the integrity of the SOA.
2. Let a service model emerge. Do virtually no design up front, except to decide upon an outline architecture, including the the means of communication between components. Develop potential shippable system increments based on vertical slices of functionality. Periodically refactor the functionality that has been developed into services.

A case study

I was recently hired to agile enable a new strategic programme for a UK based company. The company concerned was insistent that a services based approach should be adopted. They provide a large range of services, most of which have overlapping functionality. They had previously expended a large amount of effort reinventing existing functionality, in some cases, several times. Business leaders in the company perceived that the development team was taking too long to respond to the market by releasing new systems. The development team was frustrated by continual drip-feed of requirements and by the apparent low esteem in which they were sometimes held by the business side of the company. All of this adds up to a classic case for agile. I was hired by the development manager who had been convinced by senior technical people to give agile a try. To add to the mix, the company wished to outsource the development of the service layer. The internal development team would handle an orchestration layer and the user interface. Strategy 1, identified above, was broadly adopted, as follows:

• Pre-iterations. Four two-week iterations were used before the main software construction iterations, during which:
• A service design was created, which identified the main common services that should be developed, the major system compoents and the means by which they should communicate.
• Interface definitions for the major services were developed as Java interfaces.
• An existing proof-of-concept system, originally conceived as a "spike" was brought up to production quality by judicious refactoring to fit in with the service model and the addition of appropriate automated acceptance and unit test cases. Agile purists may flinch at this, suggesting the PoC should have been scrapped but I usually find that a pragmatic approach to issues like this avoids alientating clients!
• Release and iteration planning. During release and iteration planning, both the internal (presentation and orchestration layers) and external (services) teams were present. Since the external team was actually offshore, the majority of the team were present virtually using video conferencing. However, representatives of the external team were also present in person.
• User story based vertical slices. When planning, the customer presented vertical slices of functionality using user stories - several weeks of coaching prior to the first planning meeting, included a number of dry runs during which the customers practiced writing stories, drawing heavily on Mike Cohn's story writing material. The internal and external teams then estimated the work necessary to implement their part of each story separately. This resulted in most stories needing to be split. Since delivery of the functionality described by the story required both teams to complete their part of the story, particular attention was placed on synchronising the outputs of the teams when release planning.

Observations

The approach taken has certainly worked and the organisation now has a number of well implemented services available and end user systems based on these services are online. However, progress has been slower than would be expected from a pure agile approach. Some of this can be put down to the initial inexperience of the teams involved but the majority of it is likely to be due to the contraints imposed by the service approach. The requirement for the two teams to coordinate their activites around the service interfaces is less efficient in creating business functionality than a purer agile approach would have resulted in.

An emergent approach to SOA

A pure agile approach implies an almost completely emergent approach to architecture and design and would make the development of SOA based systems practically impossible.

SOA as currently applied by many organisations includes a large amount of up-front architecture and design and is traditionally seen as a largely deliberate approach. As agile practitioners we know that this is best avoided, particularly because it limits our ability to respond to change.

A reasonable compromise is to do some up-front architecture and design by developing the service model and interfaces. This can be usually achieved as part of "iteration 0" activities. The result will perhaps not be the best of both worlds but will deliver the major benefits of both agile and SOA.

Just as it is now acccepted business wisdom that a combination of deliberate and emergent approaches to stragegy is most effective, a combined deliberate/emergent approach to agile SOA is likely to deliver effective results.

technorati tags: agile, soa