Observations from the Field – The Hard Part of SOA

Where have I been? It’s been over a month since I posted. I’d wish I could say I’ve merely been enjoying the first springtime in Maine I’ve witnessed in the dozen or so years I’ve lived here. But, in actuality I’ve been working – consulting, chatting with architects and lead technologists, writing, reading, tinkering and thinking. It’s the last one (thinking) combined with a period of silence that used to simultaneously intrigue and frighten my old (awesome) team at LLB. It was hard to predict the outcome of such a cycle, but it was bound to be interesting! At the moment, even I’m not sure what will transpire, so I’ll refrain from wasting more space on that here (for now).

In the meantime, I want to share some information and insights I’ve collected over the last month. Instead of dumping everything into one post, I thought I’d start with an excerpt from the Observations from the Field: SOA report I just wrote for PSGroup.

In this excerpt, I share the “Hard Part of SOA” as identified by enterprise practitioners and lead thinkers/technologists (not marketers) at application infrastructure vendors.

The illustration I’ve included is one I first drew for a SOA workshop gig in 2004. I’d be curious if readers see the relationship between re-use and granularity differently.

Since real-world insights are invaluable, I invite enterprises to share their SOA tips, concerns, and stories with me. In turn, I’ll share that information back to the broader community. Together, we can cut through the hype, and develop pragmatic practices for SOA success.

Excerpt: The Hard Part of SOA

Enterprise architects and technical leaders consistently state the hardest part of SOA isn’t the technology. Rather, the real work is in service definition, semantics, and establishing a SOA program.

1. Service Definition

Many enterprises find it difficult to determine the correct bounds (job and tasks) and granularity (collaborations) of business services. Correctness is viewed in terms of both re-use and agility. Business services should be reusable (shared) within a line of business, across lines, and in some cases, at the enterprise level. Business services should be easily changeable (replace, augment, compose) to meet business demands.

Enterprises have erred equally on creating services at too fine a grain, and too large a grain. Erring on the fine-grained side still allows for re-use, through service collaboration, but creates performance inefficiencies to perform actual work (bounds). Erring on the large-grained side creates application- or use-specific services, rather than common business services.

Business Service Re-Use Diagram [click on diagram to enlarge]

 

This illustration shows the general relationship between granularity and service re-use. As service granularity increases, the opportunity for re-use of the resulting service decreases. However, what’s important to note is the reliance on collaborators when creating a large-grained service. A large-grained service, such as a process-oriented service, should be composed of multiple function and information-oriented services. The proper use of appropriately bounded (defined) collaborators is the re-use payoff.

Industry Specifications. Architects most confident in their business service definition have been able to leverage industry-specific specifications, such as the Open Travel Alliance (OTA). OTA specifications include schemas and WSDL implementation guidelines for common travel-related interactions, such as Request Rail Availability.

Unfortunately, industry specifications for true service interactions, rather than information hand-offs, are the exception rather than the norm. In most cases, architects and business analysts are relying on service definition practices from SOA technology and/or solution providers, or adopting best practices from experience, gut feel, and a wide range of published methods, including PSGroup’s Service Discovery.

Lack of Methodology. What’s missing—and desperately needed—is a publicly available, cohesive, services definition methodology. Cohesive doesn’t refer to degree of documentation. [I believe a methodology should contain the right amount of methodology to accomplish the task, no more, no less.] Rather, with the word “cohesive”, I am referring to the service definition activities that span business modeling, analysis, and design. You find a service (and the need for it) during business modeling. You define and refine the service during analysis. You further refine and provision the service during design.

In such, the business definition of a service is derived independent of technology implementation—current or future. As notation and tooling are used to transition the business service definition to technical implementation, the business intent must be retained.

Since the emergence of a good methodology happens over time, from real-world experience, I encourage practitioners to share their tips and challenges publicly, and join or create practitioner-led forums to advance real-world SOA.

2. Semantic Understanding

For humans, machines, and the combination, semantics has always been a challenge. While it is relatively easy to receive information (conversation, message, data exchange) it is not always easy to discern the sender’s true meaning. For example, when the word “customer” is used, does it mean end consumer, prospect, or purchaser?

As information exchange between applications and businesses became prevalent, industry and company dialects were defined to assign common terms, syntax and meaning to information elements. Given the initial purpose (technical integration) of these terms, the definition and administration has largely been the purview of technologists (geeks). As such, many of the terms are cryptic, derived from the application and information sources, rather than a true business dialect.

With SOA, the need for semantic understanding becomes even more pronounced, because of the multitude of service interactions, and the self-describing nature of service-oriented languages for service descriptions, contracts, policies, and orchestrations.

Use Business Terms. Similar to the service definition (above), semantics must be expressed in business terms. These terms must be recognized and understood by both business and technology professionals and machines, within an enterprise, and often between enterprises.

Most architects are approaching the challenge of semantics incrementally. A first priority is to achieve a common language between human parties in the enterprise, to enable proper business service definition. Another is to adopt a common XML dialect to enable semantic interoperability between services.

A common practice is to describe interfaces and message payloads with a common XML dialect, and leave the underlying providers (applications, data stores) in their current formats.

The adoption of industry standard dialects varies by enterprise. Some enterprises adopt industry standards such as ACORD for insurance, throughout the enterprise. Others limit the adoption to customer- and partner-facing services. A common complaint from architects is the industry dialects are overly complex. And the problem worsens as functional dialects (human resources, banking) are added to the mix.

Vendors agree. The semantics problem is very difficult to resolve. While most offer support for industry dialects in their SOA products, none have tackled the big semantic problem. Although the problem is difficult, it can’t be ignored. To ease your way, be sure to define and/or adopt a common business lexicon for semantic compatibility of services, applications, events, processes, information stores, and of course, people.

3. SOA Program

The mission of an SOA program is to articulate and execute an SOA strategy that makes sense for your business. Common activities include evangelizing SOA, setting and enforcing standards (technical, methods), selecting application infrastructure and tooling, building out common (system-oriented) services, finding and conducting pilot projects, and stewarding the service catalog.

Architects, technical leaders, and business solution project leaders say the biggest challenge in developing a SOA program is balancing the natural tension of executing business projects (today, on-time, on-budget) with the need for architectural integrity.

Obviously, the simplest thing to do is give your architecture team a head start. Fund, staff, and start the SOA program three to six months ahead of the first business project that needs it. The shorter the head start, the less critical that first business project should be.

Barring the luxury of a head start, build iteration, tolerance, and refactoring into the execution of your architecture and business project. Tips from the field include:

• Re-Use. Be smart about re-use mandates. While every service has re-use potential, focus initial development efforts on business and infrastructure critical services. Everywhere else, use good design practices—well-defined interfaces, separation of concerns—to allow for non-invasive replacement of “non-enterprise class” assets.

• Increments. Plan and deliver your SOA environment—practices, tools, technology, and infrastructure—incrementally. Plot your plan to balance business project delivery and architecture delivery. Better to build an incremental architecture, than a bottleneck, or bypass route.

• Refactor. For tolerant re-use and incremental architecture, you must plan and allocate time and resources to refactor and re-release code/assets.

• Simplify. Trust your architect instincts. Think about the standards, technology, and products you need in a service-oriented manner, and then map options to them. Don’t buy a box of SOA. Leverage tools and practices you have in place today.

• Communicate. Early and often.

Comments

[Anjan Bacchu]

hi brenda,

nice blog -- subscribed.

nice post -- i discovered you from O'Reilly Radar.

BR,
~A

[JT]

Hey Brenda,
Good to see you back and posting!
This was a well thought out post...both practical and pragmatic.
Two questions:
(1) If you have taken the steps to adopt SOA but have a large portfolio (and therein partial bene's) and are looking to take it to the next level...what challenges do you see for architects supporting that scenario?
(2) Who owns the taxonomy for "Business Terms"? Business? Architecture? Both?
Cheers!
-JT

[Mike Herrick]

I've seen the industry standards you mention result in a lot of challenges. I think that a Canonical Message Format is very important to success in SOA/EDA.

The trick is to keep it simple - there is a sweet spot.

I posted some more details on my blog.

[Matthew Adams]

Good read. I was particularly interested in point number 2, semantic understanding. This begins to touch on the concept of something that my company, Xcalia, calls a "service metadata repository" (SMR). We have a product that leverages our version of an SMR to enable the truly dynamic composition of both traditional databases as well as services (web and otherwise).

Your article focuses on the identification, partitioning, and creation of services, which is obviously very important. The industry, however, is largely overlooking another issue in SOA -- the consumption of services, especially when those services run the gamut from today's web services back to EJB session beans, CORBA services, and even mainframes.

With a machine-readable description of
* what services are available,
* their implementation technologies,
and, most importantly,
* a semantic description of what the services do in terms of the business dialect,
tools like ours can be used to completely encapsulate a composite application from the underlying services that are used in the application's course of execution, instead being built only against the object model that comes directly from the business lexicon.

I'd like to discuss it further with you -- please drop me a line if you're interested. I think it would be well worth your time.

Thanks for the stimulating reading!

--matthew

[Pano Anthos]

Brenda,
We certainly concur on point #2. Semantic reconciliation is key to scaling SOA. But current approaches won't scale:

(1) Canonicals by themselves won't scale SOA's if you don't have tooling to manage their complete lifecyle. Manual governance and universal consensus surrounding the definition and versioning of canonicals won't scale. All that's been done is move the reconciliations to the edge and disconnected from any sort of end-end impact analysis.

(2) EAI tooling can't do this. They are essentially point-point mapping and transform tools and don't manage the abstraction needed in a many-many architecture.

(3) The very abstraction we find in the business process (BPM's) and the transport layers (ESB's) needs to occur in the semantic tier. A sort of semantic broker without the proprietary server or single point of failure.

(4) Industry taxonomies are coming to the fore as they have grown in richness and abstraction. The consequence of this development, is however, that the models are getting more complex and hard to handle. Advanced tooling is going to be essential to govern and operationalize them.

(5) We might claim that this semantic problem is getting solved in the area of structured data for application integration. We would of course welcome feedback and criticism. https://users.pantero.com/display/blogs/Home

[brenda michelson]

I'm glad this post has generated a lot of interest. Here's my quick attempt to answer comments to date.

Anjan: Thanks for the feedback.

JT: Good questions.

(1) This one deserves a long answer. But for now, (IMO) the short answer is "portfolio management". With the "composable architectures" (SOA, EDA, etc) it's critical that Application Portfolio owners and Project Managers think about what they are producing at the asset, deliverable (project, application) and portfolio levels. Folks need to start considering the value of the individual assets. Are those assets valuable beyond the current initiative? What is the value relative to other portfolio assets? How might that change design and implementation decisions?

Too often a project or application is a labeled “quick and dirty” and then all assets are treated that way. Same on the other side. Just because a project or application is a “strategic enterprise whatever” it doesn’t mean every asset needs to be “enterprise class”.

The architect will be critical to this shift... portfolio planning, rationalization, valuation, refresh/replace, and release planning. This is one of my "business-driven architecture" soapbox items. I'll clarify and augment this in a future post. Hopefully, the gist comes through.

(2) I think the business is responsible for the definition (terms, meaning), architecture is responsible for the management, and business and architecture are jointly responsible for the communication and adoption.

Mike: Thanks. Good Post. Consistent with the folks I spoke with.

Matthew: Sounds interesting. To date, the practitioners I speak with are doing "discovery" at design time, rather than run-time. I'd be interested in hearing your story.

Pano: Good to hear from you. I'll take a look at your blog.

-Brenda

[Hugh Madden]

(as posted on TSS):
How about the data architecture???

Personally I'm fairly comfortable with levels of granularity, defining cononical forms, and communicating domain models and symantics.

The huge unknow for me is how exactly the data architecture _really_ works.

If anyone knows of a book or some online literature that would help me learn, please, please point me in the right direction :)

The unknowns for me:
When your business functions are represented as services (instead of monolithic applications), what happens to your data integration? Do you really have to throw out centralised databases and use EII/ data gateways to access all of your data?

IMHO the realities of volumes/ performance and transactional complexities make this a pie in the sky approach.

How about reporting?
Do we:
-throw away our traditional reporting tools (crystal/ actuate) and build reports manually, calling web services?

-use the traditioal reporting tools, but source them to web services/ EII platforms instead of the databases directly? (performance and productivity again precludes this as a serious option)

-use the traditional tools but have multiple data sources (losing the benefits of loose coupling - each time a service changes, your reporting investment breaks)

-build OLAP data stores (data marts/ warehouses) with feeds from the decentralised databases (where-as the EII vendors claim their tools make OLAP a thing of the past)

-have a centralised database providing the data for your services (in which case we again lose the loose coupling benefits, as the services are now tightly coupled through the shared database schema)

There is a lot of hand wavy marketing material floating around talking about how EII / data gateways are going to solve all these problems, but the detail is lacking and I find myself seriously doubting that this problem space is well understood (let alone resolved).

I've done a lot of SOA based on pub sub in the past (which I believe is vastly superior to point-to-point models). I've always had to compromise with the data architecture. In some cases the data is sourced from a service, which is elegant. In other cases (high volume or reporting), I've had to tightly couple services to a shared database schema.

In large environments, OLAP is required, but comes with a loss of agility, and high operational/ maintenance costs.

Anyway - there's my rant... I'm not interested in any "holier than thou" flames, but if anyone can point me at resources to improve my understanding of the data architecture in relation to SOA - thanks :)