elemental links

Off to Dallas...

I'm heading off to Dallas this afternoon for a client gig Tuesday afternoon.  I might sneak over to RFID World 2006 while I'm there, but if the weather forecast is true (80 degrees), I might end up reading/writing outside.  That would be +60 from my current location in Maine!  If you are in the Dallas area and want to connect: bmichelson at psgroup dot com.

Event-Driven Architecture – Thoughts from Practitioners

I've been meaning to point out a couple of good posts by practitioners related to my EDA overview post.

In the first post, MarkG over at darth.net excerpted a whitepaper he wrote on real-time integration challenges:

Real time integration brings forth a host of challenges driven by the need to consume business events as they happen. These challenges can be categorized into the following topics: Event Notification, Event Delivery, Event Persistence, Event Recovery, Event Consumption, and Event Monitoring.

Mark brings up some important design considerations. I found myself nodding my head and recalling design sessions with my (at the time) lead architect about how to find lost events. The biggest gap is between the event generator and the initial event persistence. Adding to our challenge, we used the event architecture for system events.

So, what happens when the event code (generator, persistence, processing engine) fails? How would we be informed if we couldn’t generate, persist, or process a system event? Good stuff to think about. There are always answers (bypasses), but the key is to make the right trade-offs. How much are you willing to spend (time, money, overhead) for perfection? What is the risk (business) if you don’t?

In the second post, an anonymous consultant over at Controlled Agility, points out the criticality of traceability in an event system – specifically in a chain of events (parent-child).

To enable traceability you need two more fields in each event header, you need an array of ParentEventIds and an array of SibblingEventIds.

This makes sense. I only caution that event relationships should reflect business relationships (order caused low inventory) rather than technical implementation (order is also high value order).

On a related note, I was pleased to see Joe McKendrick’s supporting post; particularly his agreement

“...that SOA is part of a constellation of strategies she calls the 'business-driven architecture’.”

The traffic generated by Joe’s post and the strong response to my IT Linchpin post, is a slap in the head for me, it’s time to post more about business-driven architecture. In a perfect world, my BDA book would be done by now, but I got distracted working with Patty…

Net Neutrality – An Important Topic for National Conversation

Last week, I strayed from my typical IT architecture and strategy coverage, to what I believe is a critical topic for national conversation – Net Neutrality.  As I saw the topic quickly referenced in numerous articles the week of the hearing, I couldn’t stop myself from digging in.  The more I read, the more I read. As a result, I put together a perspective for the PSGroup research service that attempts to frame the issue – how we got here and the positions of each side.  I do offer my own opinion at the end, but my goal in writing the piece was to present the issue neutrally.  With the intent to engage more people in the national conversation. 

What follows is that perspective. This is a long post (even for me).  You can also download (free with registration) the perspective at the PSGroup site.  I’d like to thank Patty for supporting my inclination to take this tangent.  The perspective:

Net Neutrality – An Important Topic for National Conversation

The United States Congress is currently exploring revisions to the nation’s communications laws, namely the 1996 Telecommunications Act. One of the most important, and contested, issues under consideration is Internet Neutrality (Net Neutrality). The concept of Net Neutrality is straightforward.

In a neutral network, all network traffic is treated neutrally (not discriminated against) regardless of origin or destination. In a neutral network, all endpoints (content, applications, equipment) are treated neutrally (not discriminated against), regardless of function, ownership, or implementation. Since the Internet’s inception, the network has been neutral.

From the consumer’s point of view, a neutral network allows for unfettered access to any (legal) content and applications, using the equipment of their choice.

From the application and content provider’s point of view, a neutral network supports the offering of any (legal) content and application, using the platforms of their choice.

From the network operator’s point of view, a neutral network dictates the implementation and management of a standards-based network, to transmit information from origin to destination, without consideration for the usage patterns, payload, and volume generated by the endpoints.

In essence, a neutral network is a dumb pipe. The intelligence resides at the endpoints. And there lies the conflict.

In a neutral model, the network operators claim the application and content providers get a free ride on the network, with great financial returns, on the network operator’s investment, while the network operators themselves are prohibited from offering new services (access tiers, prioritized traffic) to realize a return on their investment. Without a return on investment, network operators claim no incentive to continue their investment, specifically in broadband deployment.

In response, the application and content providers claim a discriminatory model would limit consumers’ choice on the Internet, and create high barriers to entry for new, innovative, application, and content providers. Without accessible and compelling content and applications, broadband adoption and usage will decline. Not to mention, the network operators do collect fees from the endpoints (consumers and providers).

As you can see, the Net Neutrality issue is circular. The network needs compelling applications and content. The applications and content need a viable network.

However, the good news is both sides, and the government, are in agreement that further broadband deployment and adoption are critical for innovation, social causes, economic growth, and global competitiveness.

The challenge, then, is setting forth policies that encourage both network and endpoint investment and innovation today, without placing unforeseen restrictions on future innovation.

Given this, the Senate Commerce Committee held a Net Neutrality hearing on February 7, 2006, inviting experts from both sides of the debate to share their perspectives. The hearing was important not only for the information shared, but also because it stimulated a much-needed national conversation on the topic.

In the spirit of participating in, and encouraging, that conversation, I compiled this Net Neutrality perspective. The perspective presents the Net Neutrality issues, provides important context, and shares each side’s position, along with the intersections and divergences. In an attempt for neutrality, I limit my own views to the closing.

This is an issue best understood in historical context, so I begin with background information on policy and the marketplace.

Background – Telecommunications Policy and Marketplace

In this section, I take a look at significant policy and related marketplace developments that led up to today’s Net Neutrality conversation. Keep in mind, the U.S. government has taken the stance that a limited regulatory atmosphere is best for Internet innovation and growth. A vibrant, competitive, free marketplace should in most instances be self policing and self correcting.

Since the 1996 Telecommunications Act evolved from the Communications Act of 1934, I start there.

First, There Were Telephones

The 1934 Act—through a series of revisions—split carrier services into two types, basic and enhanced. Basic services included transmission services over any medium (copper, microwave, fiber optic cable, wireless), and voice phone calls. Enhanced services are value-added information services, such as caller ID, call blocking, and call forwarding.

Basic services were regulated. Enhanced services were not. Basic service regulation requires all common carriers to be non-discriminatory in: providing service upon reasonable request, allowing the attachment of devices to the network, and connecting their networks to other operators. Basic services are regulated under Title II.

Then, There was Dial-Up Internet

Back in the days of dial-up Internet access, the Internet fell under common carrier policy. The basic and enhanced service definitions migrated into the Telecommunications Act of 1996 as telecommunications services and information services. Therefore, telecommunications services were regulated, and information services were not.

Telephone carriers providing telecommunications services were required to be non-discriminatory. This led to an explosion of ISPs and spurred the creation of the Internet we enjoy today.

Next, There was Broadband Internet

Over time, applications and content became richer, directed at a wide range of audiences—consumers, students, educators, business professionals, scientists, technologists, social groups, sports enthusiasts, gamers, you name it. This expansion required bandwidth: broadband networks over cable lines and digital subscriber lines (DSL).

Information Service Classification. To encourage investment and deployment in broadband cable networks, in 2002, the FCC classified cable modem service as an information service, rather than a telecommunications service. This removed the non-discriminatory carrier code policy from cable network providers. As a result, the FCC only has ancillary authority (Title 1) over the cable network providers. The lack of definition of this ancillary authority is a topic of discussion for Net Neutrality supporters.

The “information service” classification was overturned by the Ninth Circuit Court of Appeals, but was then affirmed by the Supreme Court in 2005.

Internet Freedoms—February 2004. In the interim period, while the information service classification was overturned, then FCC Chairman Michael K. Powell delivered his consumer-oriented Internet Freedoms in a speech at the Silicon Flatirons Symposium on Feb 8, 2004. Powell challenged the broadband network industry to preserve the following Internet freedoms:

"Freedom to Access Content. First, consumers should have access to their choice of legal content.

Consumers have come to expect to be able to go where they want on high-speed connections, and those who have migrated from dial-up would presumably object to paying a premium for broadband if certain content were blocked. Thus, I challenge all facets of the industry to commit to allowing consumers to reach the content of their choice. I recognize that network operators have a legitimate need to manage their networks and ensure a quality experience, thus reasonable limits sometimes must be placed in service contracts. Such restraints, however, should be clearly spelled out and should be as minimal as necessary.

Freedom to Use Applications. Second, consumers should be able to run applications of their choice.

As with access to content, consumers have come to expect that they can generally run whatever applications they want. Again, such applications are critical to continuing the digital broadband migration because they can drive the demand that fuels deployment. Applications developers must remain confident that their products will continue to work without interference from other companies. No one can know for sure which “killer” applications will emerge to drive deployment of the next generation high-speed technologies. Thus, I challenge all facets of the industry to let the market work and allow consumers to run applications unless they exceed service plan limitations or harm the provider’s network.

Freedom to Attach Personal Devices. Third, consumers should be permitted to attach any devices they choose to the connection in their homes.

Because devices give consumers more choice, value and personalization with respect to how they use their high-speed connections, they are critical to the future of broadband. Thus, I challenge all facets of the industry to permit consumers to attach any devices they choose to their broadband connection, so long as the devices operate within service plan limitations and do not harm the provider’s network or enable theft of service.

Freedom to Obtain Service Plan Information. Fourth, consumers should receive meaningful information regarding their service plans.

Simply put, such information is necessary to ensure that the market is working. Providers have every right to offer a variety of service tiers with varying bandwidth and feature options.

Consumers need to know about these choices as well as whether and how their service plans protect them against spam, spyware and other potential invasions of privacy.”

The Internet freedoms were presented as a code of fair conduct, and an FCC enforcement strategy. In March 2005, this enforcement strategy was demonstrated in the most noted case of blocking by a network operator. Madison River Communications settled a dispute with the FCC for alleged blocking of Vonage’s VOIP service. Many Net Neutrality supporters point to Powell’s Internet freedoms as the basis for a go-forward policy.  [In his testimony, Lawrence Lessig calls for the ratification of Powell’s Internet Freedoms with an additional point to protect application and content providers from barriers to entry, brought on by practices such as access-tiering.]

Summer 2005. On June 27, 2005, the Supreme Court reversed the Ninth Circuit of Appeals’ ruling, thus, the FCC’s classification of broadband cable modem service as an information service took effect.

On August 5, 2005, the FCC adopted the Wireline Broadband order which eliminated facilities sharing requirements on facilities-based wireline broadband Internet access service providers. In other words, DSL was granted the same information services classification as cable broadband. Now cable broadband and DSL would be treated equally. Equality stimulates fair competition.

On the same day, the FCC, under the chairmanship of Chairman Kevin J. Martin, adopted Internet principles to ensure that broadband networks are widely deployed, open, affordable, and accessible to all consumers. The Internet principles are:

“To encourage broadband deployment and preserve and promote the open and interconnected nature of the public Internet, consumers are entitled to access the lawful Internet content of their choice.

To encourage broadband deployment and preserve and promote the open and interconnected nature of the public Internet, consumers are entitled to run applications and use services of their choice, subject to the needs of law enforcement.

To encourage broadband deployment and preserve and promote the open and interconnected nature of the public Internet, consumers are entitled to connect their choice of legal devices that do not harm the network.

To encourage broadband deployment and preserve and promote the open and interconnected nature of the public Internet, consumers are entitled to competition among network providers, application and service providers, and content providers.”

At the conclusion of the Policy Statement (FCC-05-151), is a statement the principles will be incorporated into ongoing policymaking activities. However, that statement was accompanied by a footnote: “Accordingly, we are not adopting rules in this policy statement. The principles we adopt are subject to reasonable network management.” The looseness of these statements concerns Net Neutrality supporters.

Freedoms vs. Principles. A quick comparison of the Internet freedoms and Internet principles shows the former giving more rights to the consumer, and the latter providing more latitude to the network operator. The fourth Internet principle shows the current FCC chairman’s tendency to trust the dynamics of the marketplace to prevent discrimination.

October 2005—Merger Mania. On October 31, 2005, the FCC approved the mergers of SBC and AT&T, and Verizon and MCI. As a condition of the approval, the merged companies had to agree to enforceable compliance with the Internet Principles (FCC-05-151) for a two-year period.

Interestingly, the companies were asked to comply with policies that depend on a competitive environment, when the merger approvals essentially created a duopoly.

Right Now—Telecommunications Act Review. As mentioned in the opening, the United States Congress is currently exploring revisions to the nation’s communications laws, including the important topic of Net Neutrality. During the February 7, 2006 committee hearing, Senator Ted Stevens, the Commerce committee chairman, made two notable comments. First, in his opening, Senator Stevens stated: “The FCC has announced Net Neutrality principles, but Chairman Martin has stated that regulation is not needed and that it will not be needed.” Second, in his opening remarks to Panel 2, Senator Stevens asked the lawyers present to consider (for subsequent conversation) the work required on the Act itself. Did it need to be revised, replaced, or amended? Do the Net Neutrality issues stem from the Act itself?

These comments show a legislative course has not been predetermined. There is ample time for a national conversation, and of course, lobbying.

Pro Net Neutrality-Who, What, Why

Net Neutrality advocates want the Internet to remain, as it has always been, neutral. No network traffic should be discriminated against, regardless of origin or destination. No endpoint (content, applications, equipment) should be discriminated against regardless of function, ownership, or implementation. Keeping the Internet neutral is imperative for innovation and growth.

This position is articulated by Vint Cerf, father of the Internet, and Google Chief Internet evangelist, in his Feb 7, 2006 testimony, “The Internet’s open, neutral architecture has proven to be an enormous engine for market innovation, economic growth, social discourse, and the free flow of ideas.”

Vint Cerf continues:

“Because the network is neutral, the creators of new Internet content and services need not seek permission from carriers or pay special fees to be seen online. As a result, we have seen an array of unpredictable new offerings—from Voice over IP to wireless home networks to blogging—that might never have evolved had central control of the network been required by design. Allowing broadband carriers to control what people see and do online would fundamentally undermine the principles that have made the Internet such a success.”

Who Supports Net Neutrality?

The concept of Net Neutrality is supported by content and application providers (Google, Amazon, eBay), service providers (Vonage, Skype), retail equipment manufacturers, network equipment manufacturers, and consumer groups (Consumer Federation of America, Consumers Union and Free Press). I use the term “concept,” because the “how” of Net Neutrality remains under discussion.

What Do Net Neutrality Supporters Want?

• User Access. User rights to access the legal content and applications of their choice. The network should not block content or applications, such as VOIP service. The network should not re-route a request to a different destination based on network operator partnerships. For example, a request to Amazon should not be re-routed to Barnes and Noble.

• User Connection. User rights to connect using the equipment of their choice. Network operators should not mandate proprietary standards which favor specific equipment manufacturers.

• Service Plan Information. User rights to access service plan information. The service plan should clearly articulate fees, usage terms, bandwidth limitations and general quality of service. The broadband consumer may be offered tiers of service based on usage patterns. The standard tier must provide sufficient bandwidth—the lower-end consumer must not be squeezed out.

• Provider Offering. Provider rights to offer the legal content and applications of their choice. Providers need not ask permission of the network operator to deploy new content and applications.

• Provider Access. Provider rights to fair access and entry into the Internet marketplace. Providers should not have to make additional payments, beyond access fees, to ensure consumer requests will be received, and provider responses will be sent. Network operators should not employ access tiering (fast lane, slow lane), toll booths, and/or private network strategies.

• Internet Design Protection. Retain the end-to-end design of the Internet, in which, the intelligence and control is at the endpoints, in the hands of users.

• Internet Spirit Protection. Retain the freedom, creativity, innovation, and entrepreneurial spirit of the Internet with lightweight enforceable policies, rather than complicated and litigious legislation.

Why Now?

• Regulatory Changes. The FCC’s classification of cable broadband and DSL as information services, rather than telecommunication services, removes the carriers from the non-discriminatory enforcement rules under Title II. The applicable ancillary Title 1 rules are undefined and untested.

• Service Blocking. Recorded instances of service blocking, most notably the Madison River example cited in the background section.

• Consolidation of Carriers. The mergers of SBC and AT&T, and Verizon and MCI, have effectively resulted in a duopoly. The current FCC regulations hinge on a competitive carrier environment.

• “Free Ride” Statements. Recent statements from executives at AT&T (nee SBC) and Verizon make strong overtures to new surcharges for application and content providers, in the form of access tiering, or network taxes/tolls.

- AT&T. In a November 7, 2005 interview with Business Week, SBC CEO Edward Whitacre Jr. responds to a question regarding competition:

Business Week: “How concerned are you about Internet upstarts like Google (GOOG), MSN, Vonage, and others?”

Mr. Whitacre:

“How do you think they're going to get to customers? Through a broadband pipe. Cable companies have them. We have them. Now what they would like to do is use my pipes free, but I ain't going to let them do that because we have spent this capital and we have to have a return on it. So there's going to have to be some mechanism for these people who use these pipes to pay for the portion they're using. Why should they be allowed to use my pipes?”

Verizon. A February 7, 2006 Washington Post Article on Net Neutrality, quotes John Thorne, a Verizon senior vice president and deputy general counsel, as follows:

“The network builders are spending a fortune constructing and maintaining the networks that Google intends to ride on with nothing but cheap servers," Thorne told a conference marking the 10th anniversary of the Telecommunications Act of 1996. "It is enjoying a free lunch that should, by any rational account, be the lunch of the facilities providers.”

con Net Neutrality—Who, What, Why

Net Neutrality opponents are marketplace dynamics proponents, rather than pro-discrimination. This group is essentially saying three things. First, network carriers have no incentive to discriminate against user access and choice. Second, network carriers should have the right to explore new business models and pricing plans to recoup sunk investment, and expand broadband deployment. Third, the marketplace, consisting of consumers voting with dollars and carrier competition, along with existing regulation (FCC, anti-trust) will prove self policing and self correcting.

These positions are articulated in the testimony of J. Gregory Sidak, visiting professor of law, Georgetown University Law Center:

“A telecommunications carrier already lacks the incentive to block a consumer’s access to lawful content, because content and carriage are complementary goods, not substitute goods. A telecommunications carrier also lacks the incentive to degrade the quality of packets for VoIP services, because that degradation would be quickly detected and could trigger antitrust or business tort litigation. Finally, the overarching reason why anticompetitive behavior of any sort is implausible is that competition will constrain the market power of any given carrier.”

And, the testimony of Kyle McSlarrow, president and CEO, National Cable & Telecommunications Association:

"With bandwidth usage growing at a rapid pace, continued investment will be needed to keep broadband services robust. If broadband providers are to continue to make these investments, and if consumers are going to be given the levels of services and innovative new products and features they desire, all at prices they can afford, broadband providers need to have continuing flexibility to innovate in the business models and pricing plans they employ. Likewise, websites and content providers also need the flexibility to experiment with business models, and to partner with broadband providers in doing so.”

Who Opposes Net Neutrality?

The marketplace dynamics position is supported by the major broadband carriers (AT&T, Verizon), the United States Telecom Association, the National Cable & Telecommunications Association, and some individual policy experts, such as Kyle D. Dixon of The Progress and Freedom Foundation.  [Mr. Dixon’s testimony contains the following disclaimer: “The views expressed here are my own and may not reflect those of The Progress & Freedom Foundation, its Board, or its supporters.”]

Companies, such as TimeWarner, with both content and broadband interests have yet to take a public position on Net Neutrality.

What Do Net Neutrality Opponents Want?

• Return on Infrastructure Investment. Network operators want the ability to explore new business models and pricing plans to achieve a return on investment, and provide shareholder value. A proven ROI will stimulate new investment to expand broadband deployment.

• Accessible Consumer Pricing. The consumer alone should not bear the burden of the network operator’s infrastructure investment.

• Network Protection. Network operators want the ability to protect the network from harm—viruses, spam, congestion, incompatible devices.

• Network Traffic Controls. Network operators want the ability to dedicate bandwidth for their own intensive services, such as video delivery and for the services of partners.

• Government Non-Interference Stance. The government should stick to its non-interference stance, which has stimulated the growth of the Internet to date. Technology is advancing more rapidly than can be reasonably predicted. New legislation may have unforeseen future consequences.

• Competitive Marketplace. A competitive marketplace will keep the broadband carriers from self-damaging discriminatory practices. A competitive marketplace includes alternative to cable and DSL networks, such as satellite, wireless and power over broadband (PBL).

• Evidence of Discrimination. Net Neutrality opponents claim: “Net Neutrality is a solution looking for a problem.” Instances of discrimination have been limited and resolved. Market power abuse cases should be prosecuted under existing laws.

Why Now?

• Policy Encourages Broadband Deployment. The FCC’s action to classify broadband cable modem and DSL as information services was to encourage broadband deployment and adoption. The
United States is behind in broadband deployment, adoption, and bandwidth delivered to residences.

• Marketplace Opportunity. Network operators want to be able to leverage their investment to exploit opportunities in video delivery, VOIP, advertising revenue, and content and application partnerships.

• Financial Demands. Broadband deployment, particularly in less dense areas, requires significant investments.

• Network Threats. Spam, phishing, and viruses continue to threaten the network and its users.

Intersection and Divergence

As in any debate, there are points of both agreement and disagreement. Not surprisingly, some of the disagreements are as a result of assumption and inference, rather than a direct statement made by the other side. In this section, I highlight important points of agreement and disagreement.

Points of Agreement-

• Governmental non-interference has spurred investment, innovation, and growth.

• Further broadband deployment is critical to stimulate innovation, economic growth, and remain competitive globally.

• The United States is behind in broadband adoption, and bandwidth delivered to residences.

• The customer must receive value based on price and performance.

• Carriers have the right to protect the network from harm.

• Don’t Break Internet!

Points of Disagreement-

State of Competition:

• Carriers argue there are viable alternatives to cable broadband and DSL.

• Net Neutrality supporters argue any real penetration by these options (wireless, satellite, BPL) is several years away, requiring significant investment.

• Carriers argue the average consumer has multiple broadband provider choices.

• Net Neutrality supporters argue most consumers only have a choice between cable or DSL.

FCC Enforcement Power:

• Net Neutrality supporters question the enforceability of Title I for network discrimination cases.

• Carriers point to the Madison River case—however, that case was enforced under then FCC Chairman Martin’s Internet Freedoms.

Consumer Pricing Model:

• Carriers interpret the Net Neutrality stance on consumer pricing as one price fits all.

• Net Neutrality supporters accept a multiple tier consumer price model, as long as service plan information is accessible, and the standard service provides sufficient (not slow lane) access.

Bandwidth Scarcity:

• Net Neutrality supporters advocate network investment to increase bandwidth to all users, similar to the Internet2 model.

• The testimony of the President and CEO of the United States Telecom Association states “bandwidth is a finite resource.”

Rights of Network Operators:

• The network operators want to explore new business models and pricing plans that may lead to a tiered or segregated network.

• The Net Neutrality supporters oppose this.

Rights of the Application/Content Provider:

• The network operators claim popular application and content providers are freeloading.

• Net Neutrality supporters claim intelligence at the endpoints is the intrinsic design of the Internet.

The State of  Network Operators Financials:

• The network operators claim providing a neutral network is a financial burden.

• Net Neutrality supporters claim these are overstatements:

- The network operators collect access fees from both ends—consumer and application/content endpoint.

- Network operators are taking rapid depreciation on their network infrastructure investment.

Clash of Titans? As you can see, there is an impending clash between major network operators, and large application and content providers. For the sake of illustration, assuming neither side compromises:

If the network operators prevail, more consumers will have reasonably-priced broadband access to a narrower selection of content and applications. This is similar to the network television model—broad reach, limited programming.

If the Net Neutrality advocates prevail, a smaller population of consumers will have broadband access to a wide selection of content and applications. Access fees will be commensurate with the consumer’s bandwidth appetite. This is similar to the Internet model today—broad reach, varying access levels, broad choice.

closing thoughts

Staying true to my roots, I am pro-consumer and pro-innovation. As well, I appreciate the need to achieve profitability and create shareholder value.

When marketplaces are vibrant and competitive, there is an element of self policing and self correction. Participants won’t take actions that risk their marketplace position. As a result, customers have choices, innovators have opportunities, and infrastructure providers are fairly compensated. In a scenario such as this, formal Net Neutrality policies are not required.

However, recent carrier consolidation, and the absence of penetrated alternatives at the network transport layer (satellite, wireless, broadband over powerline), tell us marketplace dynamics alone are insufficient.

If the network operators are allowed to impose surcharges and create access tiers, it will be increasingly difficult for consumers to find new application and content providers, preventing consumers from voting with their dollars. As a result, innovation suffers, and the Internet and U.S. economy stagnate.

In the absence of true competition—I support a simple, enforceable, non-discrimination policy.

My position is just that, mine. I recommend readers educate themselves on this important issue, and if compelled, take action. Join in the national conversation. Contact your Congressperson. Speak with representatives from consumer groups and trade groups.

For Further Reading

Doc Searls’ provocative essay on Saving the Net

John Windhausen, Jr.: “Good Fences Make Bad Broadband,” Public Knowledge Whitepaper

Adam D. Thierer: “Net Neutrality: Digital Discrimination or Regulatory Gamesmanship in Cyberspace?” Cato Institute Policy Analysis

Vint Cerf : “Vint Cerf Speaks Out on Net Neutrality,” Official Google Blog

 

Event-Driven Architecture Overview

[Notice February 6, 2011: To celebrate the 5th Anniversary of the Event-Driven Architecture Overview, I've re-issued the original content in a new report. Learn more here.]

Continuing my business-driven architecture series, I just published an Event-Driven Architecture Overview for the PSGroup Research Service. In the overview, I explain key event concepts, walk through event processing flows, and identify the major implementation components of an event-driven architecture. This post excerpts the core of the paper. If you would prefer the full report, go here (free with registration).

For those unfamiliar with business-driven architecture, I believe the most viable, agile architectures will be comprised of a blend of architecture strategies, including (but not limited to) service-oriented architecture, event-driven architecture, process-based architecture, federated information, enterprise integration and open source adoption. How you blend, depends on your business.

Before I jump into the excerpt, I want to touch on the relationship between service-oriented architecture and event-driven architecture.

<soapbox>

I believe that SOA and EDA are peers and complements.

So, I disagree with the SOA evangelists who say EDA is merely a subset (child) of SOA.

On the SOA and EDA are complements point, I see two distinct interactions:

Event-Driven SOA. In the first interaction, the occurrence of an event (a notable thing that happens inside or outside your business) can trigger the invocation of one or many services. Those services may perform simple functions, or entire business processes. This interaction between events and services is commonly referred to as event-driven SOA.   

Service as Event Generator. In the second interaction, a service may generate an event. The event may signify a problem or impending problem, an opportunity, a threshold, or a deviation. Upon generation, the event is immediately disseminated to all interested parties (human or automated). The interested parties evaluate the event, and optionally take action. The event-driven action may include the invocation of a service, the triggering of a business process, and/or further information publication/syndication. In this interaction, the service is purely one of many event sources in a broader event-driven architecture.

A broader event-driven architecture stretches beyond event-driven SOA, to include real-time information flow and analysis, and complex event processing.

</soapbox>

And now, the excerpt: key concepts, event processing flows and implementation components.

EVENT-DRIVEN ARCHITECTURE BASICS

What is an Event?

An event is a notable thing that happens inside or outside your business. An event (business or system) may signify a problem or impending problem, an opportunity, a threshold, or a deviation.

Specification and Occurrence. The term event is often used interchangeably to refer to both the specification (definition) of the event, and each individual occurrence (instance) of the event.

Define in Business Terms. For an event to be meaningful to downstream subscribers (human and automated) it is imperative that the event (name and body) is specified in business terms, not data or application terms.

What is in an Event?

Each event occurrence has an event header and event body. The event header contains elements describing the event occurrence, such as the event specification ID, event type, event name, event timestamp, event occurrence number, and event creator. These elements are consistent, across event specifications.

Event Body. The event body describes what happened. For example, if a retailer specified a low inventory threshold event, the event body would contain the information to communicate which product fell below the allowable threshold.

The event body must be fully described so any interested party can use the information without having to go back to the source system. For the low inventory threshold event, the event body would contain not only the product identifier, but also the product description, and the point in time inventory and threshold levels. To ensure events are understood by all consumers, a clear business lexicon or ontology should be used.

What is an Event-Driven Architecture?

In an event-driven architecture, a notable thing happens inside or outside your business, which disseminates immediately to all interested parties (human or automated). The interested parties evaluate the event, and optionally take action. The event-driven action may include the invocation of a service, the triggering of a business process, and/or further information publication/syndication.

Extreme Loose Coupling. By its nature, an event-driven architecture is extremely loosely coupled, and highly distributed. The creator (source) of the event only knows the event transpired. The creator has no knowledge of the event’s subsequent processing, or the interested parties. The traceability of an event through a dynamic multipath event network can be difficult. Thus, event-driven architectures are best used for asynchronous flows of work and information.

Event Processing Styles

There are three general styles of event processing: simple, stream, and complex. The three styles are often used together in a mature event-driven architecture.

Simple Event Processing. In simple event processing, a notable event happens, initiating downstream action(s). Simple event processing is commonly used to drive the real-time flow of work—taking lag time and cost out of a business.

Stream Event Processing. In stream event processing, both ordinary and notable events happen. Ordinary events (orders, RFID transmissions) are both screened for notability and streamed to information subscribers. Stream event processing is commonly used to drive the real-time flow of information in and around the enterprise––enabling in-time decision making.

Complex Event Processing. Complex event processing (CEP) deals with evaluating a confluence of events and then taking action. The events (notable or ordinary) may cross event types and occur over a long period of time. The event correlation may be causal, temporal, or spatial. CEP requires the employment of sophisticated event interpreters, event-pattern definition and matching, and correlation techniques. CEP is commonly used to detect and respond to business anomalies, threats, and opportunities.

EVENT PROCESSING

In this section, I present example event flows for each style of event processing. The flows are represented logically, broken out into four layers. Before jumping into the event flows, I describe the logical layers.

Event Flow Layers

An event flow starts with the event being generated and culminates with the execution of any downstream (event-driven) activity. The four logical layers are as follows:

Event Generators. Every event is generated from a source. The source might be an application, data store, service, business process, transmitter, sensor, or collaboration tool (IM, email). An ordinary event may be evaluated for notability by an event preprocessor (router, filter), resulting in the generation of a new notable event.

Because of the variety of event generators, not all events will be generated in the required format for event processing. In those cases, the events need to be transformed to the required (enterprise standard) format prior to being deposited in the event channel. [a]

Event Channel. The event channel, typically a messaging backbone, transports standard formatted events between event generators, event processing engines, and downstream subscribers.

Event Processing. In the event processing layer, upon receipt, events are evaluated against event processing rules, and actions are initiated. The event processing rules and actions are defined in accordance to the needs of the interested parties, not of the event generators.

The actions include invoking a service, initiating a business process, publishing the event out to a subscription hub, directly notifying humans or systems, generating a new event, and/or capturing the event for historical purposes.

Events are processed by engines. A simple engine processes each event occurrence independently. A complex engine processes new event occurrences in context of prior and future events.

Downstream Event-Driven Activity. A single event, or event correlation, may initiate numerous downstream activities. The invocation of the activity might be a push by the event processing engine (service invocation, business process initiation, notification) or a pull by subscribers of event publications. Subscribers might be humans, applications, active business processes, data warehouses, performance dashboards, and/or automated agents. Events should be published in the standard event format. Transformation to subscriber-specific formats is typically done by an enterprise integration backbone. [b]

Example Event Flows

I am presenting simple examples to illustrate the concepts and capabilities of an event-driven architecture. The inherent flexibility at each layer—what events are generated, how they are processed, and who receives them—is the real power of event. As you review our examples, think of your own business and potential event flows that can streamline work and information delivery.

Please note, for the sake of completeness, my event flow illustrations (1-3) include more actions in the processing layer, and more subscriber types than I reference in the examples. The illustrations flow from left to right, starting with event generators.

Illustration 1 - Simple Event Flow [click ON picture to enlage]

Simple Event Processing. In Illustration 1, I show a simple event processing flow for an online bookseller’s inventory position optimization. When a customer places an order for books, the bookseller immediately commits inventory via a commit inventory service. The commit inventory service shifts inventory from available to reserved, and then checks the remaining available inventory against the optimal inventory thresholds. If the stock falls below the inventory threshold, the commit inventory service generates a Low Inventory Threshold event. This event is shown in Illustration 1 as Event Q, occurrence 27.

The Low Inventory Threshold event is deposited into the event channel and received by the simple event processing engine. The processing rules for this event type (Low Inventory Threshold) dictate two actions. A re-order inventory process is initiated, and the event is published for subscription. The subscribers are the inventory buyer, and the inventory manager’s performance dashboard. The re-order inventory process could be a straight-through process, or it could require human review and approval.

Illustration 2 - Stream Event Flow [click ON picture to enlage]

Stream Event Processing. In Illustration 2, I show three stream event processing flows for a multichannel consumer electronics retailer. In the first flow, top left hand corner of Illustration 2, an RFID sensor is emitting events every time a product leaves the warehouse. The consumer electronics retailer wants to be informed when high-end products leave the warehouse. To meet this requirement, a local event filter has rules to filter out events for items priced less than $4,000. One event, Event A, occurrence 2, is for a $5,000 plasma TV. This event is reformatted to a standard event format, and placed in the event channel. [c] The simple event processing engine receives the event, and following the rules for high-end products leaving the warehouse, publishes the event. The event is subscribed to by the inventory manager’s performance dashboard.

The second and third flows originate from the order entry application shown at the bottom left hand corner of Illustration 2. For every customer order placed, an ordinary Order Placed event is generated. All ordinary Order Placed events are streamed to a data warehouse, through the event channel, and the publication action of the simple event engine. These events are shown in Illustration 2 as Event B, occurrences 1 and 2.

For high value customer orders, an order totaling more than $1,500, the consumer electronics retailer wants to immediately reward the customer with a promotion in status. To meet this requirement, a local event router evaluates the order total of each ordinary Order Placed event, and generates a new notable High Value Order event when the $1,500 condition is met. This event is shown in Illustration 2 as Event C, occurrence 1.

The local router is employed because the burden of event routing or redistribution is on the event processing system, not the source application. The source of the event should only have to say once “this thing happened,” regardless of how the event is used in the enterprise.

Once the High Value Order event is generated, it is deposited into the event channel, and processed by the simple event engine. The action associated with the processing rules for this event is to invoke the Evaluate Customer Level service. This flow is also an example of event-driven SOA.

Illustration 3 - Complex Event Flow [click ON picture to enlage]

Complex Event Processing. In Illustration 3, I show three complex event processing flows for the same multichannel consumer electronics retailer. In the first flow, top left hand corner of Illustration 3, a business-to-business order gateway is supposed to be emitting System Heartbeat events every 15 minutes. The System Heartbeat events inform IT operations the gateway is up and running. The absence of a heartbeat event indicates a failure. If the order gateway is down, business customers are likely to place an order with a competitor.

The complex event engine tracks the timestamp of the last received System Heartbeat event. If 15 minutes have elapsed, the event processing actions associated with non-arrival are initiated. In this case, the order gateway IT guy is immediately notified (paged) and a new B2B Order Gateway Failure event is generated.

This new B2B Order Gateway event is deposited into the event channel. When the B2B Order Gateway Failure event is received by the simple event engine, a publishing action is performed. The enterprise problem management system subscribes to this event for resolution tracking.

The second and third flows show two variations of fraud detection. Both flows originate with the point-of-sale application shown at the bottom left hand corner of Illustration 3. Similar to the customer order example above, for every store sale, an ordinary Store Sale event is generated (Event Y, occurrences 1-3). These events are evaluated by a local event router, producing High Value Store Sale events (Event Z, occurrence 1) for transactions more than $1,500. All of the Store Sale events, ordinary and notable (high value), are deposited in the event channel.

In the first fraud detection flow, the complex event engine checks for multiple transactions (ordinary store sale events) by the same customer (credit card), over a short amount of time (10 minutes), at different locations over a large distance (20 miles). If these conditions are met, the place account in fraudulent status service is invoked.

In the second fraud detection flow, on receipt of a High Value Store Sale event, the complex event engine does an inquiry on the customer’s past purchases to determine if this purchase should be marked suspicious. [d] If the current purchase amount deviates more than 50 percent from largest historical purchase, the event (Z, occurrence 1) is published as suspicious. A customer advocate team subscribes to these events and places calls to registered cardholders.

As you can see from my examples, the possibilities are almost limitless. I find it particularly powerful to combine services and business processes with events. The services and business processes can be event generators, event-driven actions, or both.

EVENT-DRIVEN ARCHITECTURE IMPLEMENTATION COMPONENTS

Illustration 4 - EDA Implementation Components [click ON picture to enlage]

In the event basics and event processing sections, I surfaced many of the implementation components required for an event-driven architecture. Illustration 4 shows these components in a layered architecture manner. The components can be broken out in five categories:

• Event Metadata. A good event-driven architecture has a strong metadata architecture. Event metadata includes event specifications and event processing rules. Event specifications must be made available to event generators, event format transformers, event processing engines, and subscribers. While no standards currently exist for event definition and processing notation, it is only a matter of time.

• Event Processing. The cores of event processing are the engine and the event occurrence data. Simple event engines are often homegrown. Complex event engines should be acquired from a CEP engine provider. [e] Event occurrence data is typically persisted and retained for audit and trend analysis.

• Event Tooling. Event development tools are required to define event specifications and processing rules, and to manage subscriptions. Event management tools provide administration and monitoring of the event processing infrastructure, monitoring of event flows, and visibility into event generation and processing statistics.

• Enterprise Integration. An enterprise integration backbone plays a large role in event-driven architecture. Some of the required integration services are: event preprocessing (filters, routes, and transformations), event channel transport, service invocation, business process invocation, publication and subscription, and enterprise information access.

• Sources and Targets. These are the resources of the enterprise—applications, services, business processes, data stores, people, and automated agents—that generate events and/or perform an event-driven action.

The topology configurations for these components will vary for each enterprise, based on the event flows, event occurrence volumes, enterprise integration backbone, and location/distribution of sources and targets.

---

[a] To date, no industry standards exist for business event definition or notation. IBM has defined a common base event for systems events. The common base event is included in the Web Services Distributed Management (WSDM) specification. The common base event format evolved from IBM’s Tivoli (systems management) product line.

[b] For human event subscribers, consider the use of Really Simple Syndication (RSS).

[c] It is an architectural decision if the local event filter should be aware of the proprietary RFID format, or if all RFID events should be transformed to a standard format (event formatter), and then evaluated by a local event filter.

[d] To facilitate the enterprise database query (customer order history), complex event processing systems have the capability to cache relevant enterprise information.

[e] Some of the providers offering event processing engines are:  AptSoft, IBM, KnowNow, Progress Software (parent of Sonic), StreamBase, and Tibco.

CAIT at Washington University in St. Louis – SOA Briefing

Last week I went to St. Louis to give a presentation on Insights from Early SOA Practitioners. [My luggage went to Detroit, but that’s a whole other story, replete with whining and late night speed shopping.] The members-only event was sponsored by Washington University’s Center for the Application of Information Technology.

I agreed to participate in the event for two reasons. First, I’m a fan of programs that strengthen IT communities in order to stimulate regional and state economic development. As an offshoot of my participation, CAIT’s director offered to share her insights with the director of Maine’s Software and IT Association (MESDA) – my own local IT community.

Second, CAIT’s membership is comprised of enterprises from a variety of industries, employing a multitude of technologies, spanning paradigms. In other words, CAIT’s membership reflects enterprise IT reality. So, I saw the event as a two-way opportunity. Not only would the audience (hopefully) learn a little about SOA, but I could learn from their questions and concerns. I believe interactions like these strengthen all of our work.

My presentation was based on a handful of my published papers/posts (SOA Cheat Sheet, SOA Forum Takeaways, Vanilla Layer Cake Theory, L.L. Bean Architecture Case Study) and conversations I’ve had with SOA practitioners. Fear not, all non-disclosure agreements were (and always will be) respected.

Instead of repeating material here (my presentation), or rehashing the complete Q&A, I wanted to share some of the audience’s questions that could benefit from wider discussion. If you have insights on any of the following, please comment on this post, or send me an email.

Before jumping into the questions, I wanted to call out a few bonuses from the day, including the opportunity to work with Fred Domke, the chance to chat in person with Todd Biske, and a great lunch in the Lafayette Square section of St. Louis with CAIT members.

The Questions:

1. At what point can/should an organization be considered service-oriented?

The gentleman asking this question was trying to understand just how much SOA, the early SOA practitioners had in place. In other words, how far behind were companies that hadn’t started?

At the core of his question, was how is the industry counting? Is it by the presence of small number of service-oriented solutions? Or, based on the percentage of an application (business solutions) portfolio that is service-oriented?

I would say the former. Which I think is fine – at this early stage – as long as the numbers are being used in context. So, returning to the bigger question, you aren’t by any means a “laggard” if you haven’t started SOA.

However, I find the percentage of portfolio aspect intriguing. This is a much better indicator than numbers of applications and/or services.  I'd like to explore this further, which leads me to some questions for my readers:

a. For early SOA adopters, what percentage of your application (business solution) portfolio is (today) service-oriented? Composed of services? Service enabled?

b. Over time, what percentage of your application (business solution) portfolio will be service-oriented? Composed of services? Service enabled?

2. Is Service-Oriented Architecture being used for Compliance? (SOX, HIPAA)

I’m sure it is. But, I found myself unable to cite a specific example, so I did the next best thing, referred the audience to the guys at Redmonk and their Compliance Oriented Architecture.

If readers have specific SOA and Compliance examples – that can be shared – let me know.

3. What is the Relationship between SOA and ITIL (Information Technology Infrastructure Library)?

Huh. On this question, I had two options. Make something up. Tell the truth – I’m aware of ITIL, but to answer the question, I’d have to make something up!

So, I went with the latter. However, I would like to learn the answer, for the gentleman that asked, and for my own (ever continuing) education. If you have worked with both SOA and ITIL and/or have some insights you’d like to share, I welcome them. As well, I’ll be digging into some ITIL materials myself.