@ Gartner AADI: Mastermind Interview on Event Processing – Roy Schulte & Mani Chandy

Session Abstract: Roy Schulte and Dr. Chandy released a new book on Event Processing in October 2009, aimed at business and system analysts, architects, application managers, CIOs, and technically-oriented business managers. This session will highlight the key points of the book and explain why and how mainstream IT departments will ramp their use of event processing up during the next ten years.

Dr. Chandy is the Simon Ramo Professor at the California Institute of Technology in Pasadena, California. He has received numerous awards including the CMG Michelson Award, the IEEE Kobayashi award, and the Babbage Award.

W. Roy Schulte is Vice President and Distinguished Analyst at Gartner. He was the lead author of the 1996 Gartner report that introduced the term SOA to the industry. Mr. Schulte originated the research in the field of message brokers, coined the term business activity monitoring (BAM), and wrote the first analyst reports on the zero-latency enterprise and the enterprise service bus (ESB).

Yesterday, Roy did a rapid-fire session on Event Processing.  A few nuggets of his that I tweeted:

• "traditional systems give 'slow-motion' situational awareness"
• "Need to move management reporting and decision-making to continuous intelligence
• "continuous intelligence gives the opportunity for continuous corrections"

Today, Yefim Natis is interviewing Roy Schulte and Dr. Mani Chandy on event processing.  Here goes…

YN: Why this book, why now?

RS: A lot of books on agile business and real-time business, and a lot of event programming books, but no book that brought the two together.  How you would go about creating an agile business, using sense and respond, creating real-time visibility and responsiveness.

MC: There are a lot of event systems in practice, but across a variety of industries – cow pasture management, intelligence community, financial services.  This book brings the common ideas across those implementations together.

YN: What is an Event?

MC: Event is a change.  That change can be just a passage of time. 

RS: Two big ideas behind event processing.  Simple event processing – reacting to one event at a time.  Complex event processing – processing and potentially reacting to a plethora of events.  [My Event Driven Architecture Overview has similar definitions]

MC: There is an EAI aspect of event processing, and the sense and respond aspect of business event processing.  Both make sense.  Both have been programmed before.  Business events (and processing) drive agile business.

YN: Let’s be real about event processing.  How much is hype, how much is real?

RS: Right now, there is not enough hype on event processing.  Speaks to google trends on event processing.  Fear that EP becomes overhyped, over promised.  Wants enough hype to create awareness for business change, but not too much. 

MC: Event Processing won’t replace SOA, complements SOA.  Event Processing has tie to cloud computing, event objects (instances) may reside/pass thru event clouds, in the “cloud”.

RS: Event Processing and SOA, there is an event-driven style of SOA.  But again, not a replacement.

MC: A lot invested in SOA.  A lot invested in defining the right business services.  In Event Processing a big piece of work is teasing out the Business Events.  If you have an SOA in place with well defined business services, it will be easier to tease out those events, placing an event layer on top.

YN: Is SOA a first step in Event processing?

MC: SOA is decoupled distributed architecture.  SOA needs to be done regardless. 

RS: Event-driven architecture is not a goal in itself.  Need to have supporting business goals.  Ask: Does adding event sense and respond enable optimization of your business processes?  Does having real-time, or near real-time, visibility to particular sets of business information or activity allow you to make better decisions?  If so, as you work on those business processes, services or information sets, you need to identify the related events as first class citizens in your design process.

YN: How do you make sense of all these events?  External events?  Events from different systems?  Do you need event-specific application integration?

RS: Dispel the notion that you have to buy software to do event processing.  You can do event processing via programming.  You might want to purchase tools to deliver faster.  Also, many packaged applications are sending events out now.  When you configure the package, you specify the events you want.

MC: When working with external events, you need to do some degree of integration, such as data (schema) mapping and data quality checks.

YN: How do BPM and Event Processing relate?

RS: Nice thing about architecture, and BPM is they have so many definitions.  BPM, the business discipline, is very compatible with event processing.  As you model a business process, you identify the relevant events (triggers, outputs).  Also, the monitoring of business processes (BAM) is event based.  BPM tools, if they don’t already, need to add event awareness.

MC: Another integration point between Events and BPM is that an Event can trigger a business process, or activity within a business process.

YN: What tools should people look at?  At what point do you need to look into tools?

RS: Simple event processing is about creating and conveying an event object, you can do that with standard communications software in place.  If you have high volume and/or low latency requirements, you might need to buy an additional tool.  Another investment might be a CEP engine, or rules engine, to correlate events and execute the associated next steps.

YN: Do you see opportunity to generate events out in the cloud?  [I do.]

MC: Gives an example of earthquake detection.  If there were sensors on mobile devices, buildings, etc to detect a “shake”, events could be generated and sent to the cloud, the cloud based application could then generate an earthquake warning.  This same application could be used for earthquake detection around the world. This is an actual example, the cloud based application is done in GoogleAppEngine.

RS: Aware of cloud based CEP engine services.  Also data feeds from financial markets.  EP and Cloud Computing will increase ties overtime.

…

RS: If you are going to do event processing, you should be aware of common practices and patterns.  Sharing this information was a goal of the book.

YN: Is event processing crystallizing old ideas?

RS: Event processing happens today, situational awareness happens today, in your head.  It is different to automate this processing.  The vastness of information processing, and the speed required, needs automation.  Event processing is not new at all.  The automation of EP provides significant business (and social) advantage.

MC: There is a discipline of event processing.  If you understand the discipline, you can build sense and respond, situational awareness applications, better..  [I agree]

YN: How do organizations sell the need for Event Processing to the business executives?

RS: Don’t do justification for event processing.  Do justification for business application (business solution) that will incorporate aspects of event processing.  A good method is to create a mock-up of a dashboard showing the data that would be available for decision-making with event processing.

Yefim is advising audience to read the book.  [I agree]  Asks, what should attendees do on Monday?

MC: Focus on the business and ask what situations they need to respond to quicker?  Identify the data sources.  Make an inventory the tools you already have to enable event processing.  With that inventory, you’ll see how close you already are.

RS: Start learning.  There aren’t a lot of people that know how to do this.  Educate yourself for your business.  Educate yourself to be in high demand. 

[As an EP consultant, I’d be remiss not to add, or contact me.]

Comments

[Chris Bird]

I am glad to see both CEP kinds of event handling and simple (?!) event handling one at a time both addressed. Much of the time the "event hype" is around CEP, and yet I think there is huge value to be obtained through handling the events one at a time too.

I am also glad to see that there is a downplaying of tools. Perhaps this will help us avoid some of the early SOA horrors. Oh, let's go and buy a SOA....

For me the "event at a time" or simple event processing model doesn't require a whole lot of new tooling. It does require some new patterns though. The most interesting pattern IMHO is ClaimCheck. In this pattern the information that would be submitted along with the event is, instead, written to a store. This is a simple write - doesn't require a massive database engine. A claim check is also generated. That provides an identifier to the data. When a subscriber to the event gets the event, the only data that it gets is the claim check. The subscriber uses the claim check to locate the written data and pull it as it requires. A benefit of such an approach is that we can use a very lightweight mechanism (REST anyone?) for implementing the content management and delivery without gumming up the works of the infrastructure. I refer to this as the pushmi-pullyu pattern (name taken from Dr. Dolittle). The push part is the delivery of the event/claim check. The pull part is the pulling of the appropriate information. Sure there's a bit more to it than that, but that kind of hybrid pattern has tremendous power in manbaging throughput through an Event Network, while providing for the necessary richness.

Could we have done it with EAI tools of the past? Yes! Why didn't we? EAI was about connecting applications and not more finely grained components. In EAI we typically responded to application events not business events. Here in simple event processing we need to be focused on the business events. There is of course a whole lot more to think about, but I wanted to get this into the thinking stream...

Chris

[Paul Vincent]

Hi Brenda - nice report.

Hi Chris - simple event-at-a-time processing is of course the "norm" for many event-driven systems in the past - TIBCO for example has over 1,000 customers for its BusinessWorks (SEP) tool. However CEP is interesting because events are naturally related to other events, and this aggregation of information across events and event types is often significant...

Cheers

[John Clark]

An example of a BPM based, ITIL v3 aligned Event Management process that is included in our BPM based Service Management solution called e-Service Desk;

http://blog.iccm.co.uk/2009/07/11/a-bpm-example-using-the-itil-v3-event-management-process/