2010: The Rise of Event Processing

I don’t typically engage in predictions, but here’s mine for 2010, fresh from my tweet stream:

2010: Event Processing transcends niche status, to well-recognized & adopted business technique for real-time visibility & responsiveness.

I can list tons of reasons why, but it boils down to this:  you can’t change what you can’t see.

Comments

[Mark Palmer]

Hi Brenda -

Well I certainly agree with this prediction. Interestingly, Marc Adler provoked me to provide an update in his blog that I thought might make sense to re-post here. I responded by listing the top 9 things going on in CEP in the last year, and these might provide some insight into your prediction. Clearly they are from the StreamBase point of view, and perhaps a bit biased, but since I hold a unique view from running 2 of the top companies, I think it's fairly well balanced. So here they are:

The CEP market is cranking - watch for a steady rhythm of very important news from StreamBase over the next 2 months, but even just recapping the LAST few months, there have been several watershed moments for us, and I think for the industry. For fun, I'll put my "top 10" together:

1) Roy Schulte / Mani Chandry's Book Published This is a big deal because Roy has been proactively involved in promoting CEP for a long time, and the presence of this book and Gartner's coverage should spread CEP more deeply in the consciousness of IT outside of the capital markets. Here's a link to the book on Amazon.

2) CME Group announcement I think this is THE customer announcement of the year - and another in a series of endorsements that show that CEP is going more mainstream and broader in terms of the applications it's being used for. Here was my blog post on what CME Group is doing, why they went CEP

3) StreamBase Continuing to Rise CMC Markets, RBC, BNY ConvergEx, PhaseCapital, BlueCrest, Kairos all announce StreamBase selections. The volume of announcements reflect great progress from StreamBase. And, event better, CMC Markets, who handle over a trillion dollars of trading traffic, was public about switching from Apama to StreamBase, which I examined in this post called "Why the CEP Tide is Turning To StreamBase"

The CMC Q&A with CIO Asif Adatia was remarkable and informative.

[Mark Palmer]

5) Adoption of CEP in Brazil CEP adoption has exploded in Brazil, mostly led by Apama (they have the strongest local support). We had a nice announcement that was covered by the FT. Here was my blog post on it

6) Microsoft Stream Insight buzz There has been some smoke about the Microsoft CEP product, but no fire. But in general the buzz from Microsoft, IBM, and Oracle has increased.

7) Some, but limited expansion outside of capital markets There have been a few public endorsements / signs of adoption of CEP outside of capital markets. Apama did a few, but they were hardly mainstream names or applications, unfortunately. I think a cruise ship and a location based services app.

8) Forrester issued a report on CEP It's great to see Forrester cover CEP as a space. The report was purely paper based and only mild utility for anyone in capital markets, but at least outside of capital markets it helps raise awareness.

9) Watch this space in the next 2 months 3 weeks ago, I tweeted that StreamBase will announce a series of news that will be remarkable. It started with CMC, and is to be followed with more big news about the progress of StreamBase in particular, but also CEP in general. 3 of the announcements, in my estimation, will rank in the top 5 on this list, so maybe I'll revisit later if you wish.

Hope this is helpful.

Mark Palmer, CEO, StreamBase Systems

[chris bird]

Brenda,
I am delighted to see that event "stuff" is becoming more mainstream too. I think it is so important, so necessary for cleaner decoupling of system components. I have to admit that I am less interested in the CEP activities at the moment than I am in simply oiling the wheels of "the business." Too often I see to much complexity when we try to control things that aren't our responsibility.

There is a slippery slope here, though. In an article on my blog I talk about the dangers of attempting to orchestrate the event network from the perspective of an event publisher. John Schlesinger is quite eloquent in his comments - well worth a read, I think.

There has been a lot of chat about whether when we send an event the content is transmitted to subscribers with the event. There is a kind of mix-up of concerns here too. I would argue that the content doesn't travel with the event - puts a big tax on the infrastructure, but that the content that would have been transmited with the event is stored somewhere else so it can be pulled as desired. Looks to me to be a convenient hybrid push/pull model with a RESTful capability for the pull part. So think "Claim Check" pattern where the claimcheck for the content within the event is a URI.

Its funny how things come around. The first big airline rservation system was Sabre. A heavily message oriented system. The whole basis of getting work done was "queueing" things. So much so that employees at American Airlines used to (and possibly still do) use Sabre as a messaging system among themselves. This was of course years before corporate email. Disclaimer - I now work for sabre, but these views are mine not its). So early large sale computing - message oriented. Then a decline into command/control kinds of systems. It's good to see the rebirth of event/message systems. The worm has turned!

[Paul Vincent]

Hi Brenda: I can't disagree with your prediction, of course! Although I suspect the entrenched practices in IT won't be easily persuaded to move to event-centric approaches.

However I will append a comment below after reading Mark's comments (from financial services CEP vendor Streambase), which although as usual very interesting, could cause some of your readers to get the impression that CEP remains only relevant and/or successful in financial services.

While it is true that many niche CEP vendors remain focused (and successfully so) on "trading system" and financial systems event processing, others (such as ourselves at TIBCO) are finding CEP success in other domains too, such as transport and logistics, government, and telco.

Cheers!

[brenda]

Hi Paul,

Absolutely. By "transcends niche status", I was referring to adoption use cases and markets. There is tremendous potential for EP in all domains, as your early adopting customers well know.

Best,
Brenda

[brenda]

Hi Mark,

Nice list. Points 1, 2, 6 - 8 played into my "prediction". As did the sudden EP (AEP?) interest of traditional analytics players and pundits, Progress' SOA/EDA positioning as Operational Responsiveness, and Informatica's purchase of Agent Logic.

Plus, one other thing, which I can't talk about (yet). 2010 bodes to a great year for EP.

Have fun in Davos!

Brenda

[brenda]

Hi Chris,

Would that be a cloud resident "claim check"? That way, events are free to cross borders. All part of the "fluid enterprise" I wrote about forever ago.

I look forward to chatting further with you on event processing and "oiling the wheels of business".

Best,
Brenda

[Chris Bird]

Brenda,
May be "Cloud Resident", but I tend to think it will be resident on a web server accessible using HTTP by the subscribers. So there are trust boundaries at work here.

[Mark Palmer]

Hi Paul -

To clarify on your "clarification" - StreamBase is not a "capital markets CEP vendor." Our platform is general purpose and has many applications both inside, and outside, of capital markets.

For example, as you may know, StreamBase is used by the federal government for a variety of classified applications - indeed, our company was funded by In-Q-Tel, which is the investment arm of the U.S. intelligence agencies. And we have public internet / ecommerce customers such as Orbitz, and Linden Labs / Second Life, etc. And, as you might know, our customers in the capital markets do a lot more than trading. BNY ConvergEx, for example, announced monitoring applications built on StreamBase, and more. And our real-time web news that swirled around our Twitter adapter announcement earlier this year shows yet another element of the broad application of CEP and StreamBase's activity outside of capital markets. Twitter, indeed, is one of the other recipients of the WEF innovator honor. There is no doubt that we focus on the capital markets, it's by far and away the dominant market for CEP today.

That said, our broad impact beyond capital markets was a big part of why the World Economic Forum selection process - it was an award for the global impact across all industries StreamBase has made, and will continue to make.

- Mark Palmer, CEO, StreamBase

[Mark Palmer]

Hi Brenda -

I'm looking forward to reading your take on SOA and event processing. When I was at IONA leading their ESB group, and at Progress with Apama and Sonic, I fought against SOA <-> event platform unification product strategy. Simply put, most ESB customers don't need it - event processing and SOA are complementary, but they don't need each other.

The existence proof for this is are the hundreds of applications of event processing in capital markets, where you will very rarely find SOA. TIBCO and Progress will market SOA + event processing product strategy because that's what they sell, not because it's what customers need.

Don't get me wrong, SOA is a fine technology, but it's got little or nothing to do with event processing. I'd be happy to hear your independent view on this debate - and, even more valuable, some specific examples of where SOA needs event processing, or vice versa.

- Mark

[Paul Vincent]

Hi Mark - Au contraire, SOA can be a "source of events" for event processing (eg track and trace of service usage, sense and respond to erroneous service invocation patterns, etc). Also, for some people, "complex event processing services" and EDA are part of a wider SOA definition. And even if one thinks of SOA as "just" the WS-* stack, then excluding service requests being treated as events excludes a whole set of potential (and indeed in-production :) ) CEP applications - even in capital markets, I'm sure!

The Event Processing Technical Society definition of "event processing" refers to "operations" on events - which for high performance and low latency algorithmic trading will usually exclude synchronous service invocations of course. However, other event processing applications may do things like event enrichment (database service calls), service-based analysis of complex events (invocations of predictive analytics services) etc.

Incidentally I recommend the EPTS Glossary definition for readers of Brenda's blog, rather than any particular vendors' definition - it can be downloaded from http://www.ep-ts.com

PS: I too would be interested in seeing an example of the marketing of an "SOA+event processing joint strategy" you credit to us at TIBCO. We certainly do provide customers with both SOA and event processing solutions, and quite often use the latter to help monitor and manage the former. For references, the TIBCO CEP blog mentions 1 or 2 telco and cloud examples by the way.

Cheers

[brenda]

Hi Mark,

My reference to Progress positioning SOA/EDA as operational responsiveness was to note the transition of SOA as the marketing lead, to EDA/EP as the marketing lead.

As for SOA & EDA, I wrote my Event Processing Overview back in 2006 as a (frustrated) response to then SOA pundits saying EDA was merely a subset of SOA.

I hold firm to my stance, in that paper and even earlier in practice, that SOA and EDA are peers and complements. Services can be event generators (sources) and/or downstream actions (event-driven SOA).

However, and I'll just quote the 2006 paper "event-driven architecture stretches beyond event-driven SOA, to include real-time information flow and analysis, and complex event processing."

As I work with practitioners now, the business scenarios are a mix of SOA alone, Event-Driven SOA and EP/CEP alone. None are under the impression that SOA is required for EDA, or vice versa.

Best,
Brenda