The two types of Event Processing
I was thinking this morning about two types of Event Processing: “Detection Oriented” and “Operations Oriented”. It looks like these are the main categories of Event Processing, both in terms of business value and requirements for a COTS product. Today the Event Processing Technical Society does not make a big deal out of this distinction, but I think that will change.
Detection Oriented Event Processing applications want to locate interesting information in a flow of data. Operations Oriented Event Processing applications want to take an action (which involves making decisions) based on incoming events.
Detection Oriented Event Processing is the Event Processing equivalent of data analysis. It is driven by a need to detect faster than was previously possible and is important to understand that Event Processing adoption is not driven by the need to detect more accurately. More accurate detection requires better detection methods. Detection methods are not part of Event Processing, although Event Processing may make it easier to implement certain methods.
Detection Oriented Event Processing suffers from a significant disadvantage compared to traditional data analysis: the data to be analyzed is not all available at once. Rather, the data arrives incrementally as events.
Detection Oriented applications are developed by data analysts. They require a good understanding of what is being detected and how the events contribute data. They may benefit from statistics, data mining or machine learning. Their challenge is to balance the goal of detecting with their performance requirements. They are tested by running various real-life or simulated data scenarios through the Event Processing Network.
Operations Oriented Event Processing is driven by a need to react faster (with lower latency or higher throughput) than was previously possible. Again we must distinguish faster from “better”. Better reaction requires better decision-making. And better decision-making is not a part of Event Processing, although Event Processing may help make decisions with lower latency or higher throughput.
Operations Oriented Event Processing applications are developed by logic coders. They require a good understanding of how the business should react and what the events mean to the operation of the business. They may benefit from decision management, decision-making under uncertainty or other areas of operations research. Their challenge is to balance the goal of deciding with their performance requirements. They are tested by running specific sequences of events through the Event Processing Network.
The two types of event processing put different requirements on Event Processing software. For example:
Detection Oriented applications will change as the data changes, as the detection methods change or as the detection goals change. It is common for these applications to regularly add more detection logic, while keeping the old logic.
Operations Oriented applications will change as the business optimizes or changes its decision-making strategies. It is more common for these applications to update their decision-making logic and get rid of the old logic.
Of course, every Detection Oriented application will do something with its detection results. And every Operations Oriented application will involve some amount of detection. But the focus of the application remains on one or the other goal.
There are applications that require both Detection Oriented and Operations Oriented Event Processing. These are applications with both complex detection and complex reaction requirements. In my experience, is common for these applications to be cleanly divided between the two goals. This division is often so complete that the two goals are implemented essentially by separate systems and often by separate teams. And when this division is not so complete, everyone usually wishes that it were.
So if these types of Event Processing are separate in terms of their value and their implementation, maybe they should each use separate Event Processing software?
[...] 23, 2009 I recently began a line of thinking about breaking Event Processing into two categories: Detection Oriented and Operations Oriented. In my last post (part 1), I began to talk about the business value that can be generated by [...]
I think the key point here is not identifying different types of event processing, but decoupling different components.
In my articles for the CBDI Forum, I have advocated two critical decouplings.
Decoupling the event capture from the event response (We may develop more automated or more accurate ways of capturing or detecting business events, but this should not affect the way these events are processed.)
Decoupling the event from the event capture mechanism. (The event is the same, regardless of how we discover the event.)
http://cbdi.wikispaces.com/Event-Driven+SOA
Hi Richard. It seems that there’s more to the story. The division between types of EP is usable at many levels. For example, to expand the decoupling idea:
There is decoupling in the Operations Oriented logic which leads to intermediate, business events, based on existing event contracts. All of the events involved here follow business level contracts.
Then there is decoupling between Detection Oriented and Operations Oriented events. This means separating the logic that deals with chaotic events that follow a loose contract from the logic that deals with the events that follow a strict contract.
Give me a few more posts to convince you – it’s hard to paint the complete picture quickly.
From an architectural point of view, we need to understand what various kinds of decoupling bring in terms of system flexibility. For example, given the expectation that detection devices will improve over time, we can build a loosely coupled EP system that allows the detection subsystem to be modified without affecting the operational subsystem.
Bottom line: it seems that decoupling is only part of the story. It’s a consideration for architecture, of course.
But there are other considerations:
For operations oriented EP, the system might need to support refactoring out intermediate events as well as debugging through many intermediate events. You want to evolve the existing business logic over time.
For detections oriented EP, this might be less of a concern. It’s not so much of a problem to develop new detection logic in parallel to the existing logic. There are concerns like software reuse and resource utilization, but still in the area of evolving logic, detection oriented applications are often more flexible than operations oriented.
This is just an example, I hope you’ll see other compelling differences over the coming posts.
Hans,
you seam to be contradicting yourself when you say
“… every Detection Oriented application will do something with its detection results. And every Operations Oriented application will involve some amount of detection …”
I don’t see how these are not two different components of a CEP system (detection & operation/execution) rather than two different types.
The way you connect these two components can differ depending on complexity (that’s where the decoupling logic comes in), but I fail to see how these are two different types of an event processing systems.
True – some use cases may require more complex detection, while others may require more complex executions (e.g Smart Order Router in a trading system), but these are implementation details.
Anyway, these implementation details are mostly abstracted if you use a vendor product such as Streambase or Esper, or if you roll your own, can be addressed with existing patterns.
BTW, I think your argument would carry more weight if you offered some examples, otherwise it’s too abstract.
Indeed these may be components of a CEP system. But it is quite illuminating to analyze the requirements of each of these types component – you will often find that the two types of components place very different requirements on the CEP system.
In trading, signal generation and risk calculation are the Detection Oriented parts. This includes liquidity signals for smart order routing.
Order and position/P&L management and order routing are the Operations Oriented parts.
When we start to look closely at these two areas of trading, we find different requirements and usage scenarios for a CEP system. In face, we find that different CEP systems have very different strengths and weaknesses when it comes to each of these types.
Whether the same CEP system is used for both parts (they are components of a CEP system) or a different system is used for each part – that is an implementation detail. But by breaking down our thinking in this way, we have an effective framework to analyze the strengths and weaknesses of a CEP system for our current and anticipated requirements.
I understand about the examples. I put up a post analyzing two other blog posts in terms of the two types of EP. I am working on better examples, hope to have some illustration in future posts.
Hi Hans – Glad to hear from you again! Your thoughts are always quite cogent and helpful.
So I generally agree with your dividing a couple of different classes of event processing with respect to some of the use cases we see. That said, I’m unclear about what useful work the classification might provide. That is, what are you trying to accomplish with the taxonomy? At one point you postulated that perhaps different types of event processing would be useful for your two types. However, at StreamBase, we parse no distinction between Detection Oriented applications and Operations Oriented applications at StreamBase; nor did we parse this distinction at Apama. So it’s not clear to me that work sis done there.
The other point I wonder about is the hard line on whether one type of EP application tends to be a replacement for an existing system or, I guess, a brand new system.
So I guess I’m wondering out loud the goal of the taxonomy exercise we’re taking on here. For me, both “classes” of applications are well within the sweet spot of what those of us building event processing platforms are focused to support?
Cheers,
- Mark Palmer, CEO, StreamBase
Hi Mark, the goal is just to show a good way to think about EP. I think about it as a practitioner, so mostly I write practical advice. Over time, I think you’ll see that this way of looking at EP results in very good practical insight.
In terms of the benefit to you – I’m not doing it for you.
There may be some good for you. It would probably be good for practitioners to get that ah-ha where they not only see the strategic vision but also a path to actually using EP.
WHAT? You didn’t write this post for ME?
Sincerely, I was trying to understand your mission, and now that I understand (thanks), perhaps I can contribute – even though my current blogging is on the strategic / high level stuff, the bulk of what we do is to educate about use cases, so I’ll provide a few comments and examples that should add texture to your post.
“Detection Oriented” CEP applications, in my experience, are, conceptually, the first ones new users grok.Often, this type of application is the first one customers build. Your description of them is excellent. Concrete examples include basic performance monitoring (is my system running well?) system health management (is my system healthy?), pre-trade risk management (are trades I’m doing violating my risk profile?), surveillance (is anything illegal going on right now?). One of the best examples of a Detection Oriented system is BNY ConvergEx, here’s more detail on what they did with StreamBase – their first CEP application was a massive monitoring system that actually Detected multiple levels of events in their U.S. trading operations: system health, market feed latency, market execution latency, and a whole host of business analytics, which, as you rightly say, grew over time – they kept adding more and more:
http://www.streambase.com/finance-customers.htm
Another good example that we published tons of detail on was CMC Markets. One of the first applications they talked about was FX pricing, which is a very sophisticated Detection Oriented application:
http://streambase.typepad.com/streambase_stream_process/2009/11/trading-trillions-on-cep.html
Operations Oriented systems – wow, where to start? Now you’re adding action – which is where event processing really adds big value. Instead of just watching or seeing what’s going on, you’re doing something about it. Operations Oriented apps, in my eyes, are the most fascinating applications of CEP. If I understand your characterization of this class of application, Hans, would you agree that Operations Oriented systems are a superset of Detection Oriented systems? That is, in order to perform operational tasks, first you have to detect them. To make them smart, you have to create, monitor, and control actions. State management becomes more important here.
Examples of Operations Oriented applications are many. Smart Order Routing, trading of all types, control systems in the government and telecom. Intelligent monitoring systems that take corrective actions once they detect anomalies. Risk management, surveillance, and defense systems that take corrective action.
There are tons of detailed case studies on the StreamBase blog about Operations oriented applications, including:
Bluecrest (which uses Vertica in interesting ways to manage real-time state on ‘everything that ticks’ on the financial markets):
http://streambase.typepad.com/streambase_stream_process/2008/12/case-study-bluecrest-capital-management.html
PhaseCapital (built their entire trading infrastructure on event processing:
http://streambase.typepad.com/streambase_stream_process/2009/04/8-lessons-for-entrepreneurs-a-case-study-based-on-phasecapital.html
CME Group (this one is higher level but very good)
http://streambase.typepad.com/streambase_stream_process/2009/06/innovation-by-the-numbers-at-cme-group.html:
Kairos (another full trading system):
http://streambase.typepad.com/streambase_stream_process/2009/10/streambase-in-brazil.html
More recently, Gain Capital, City Index, RBC, and others can be found in the case studies section:
http://streambase.typepad.com/streambase_stream_process/case-study/
So as a lens through which to view different types of applications, the classification is helpful, hopefully these specific examples might make the framework more concrete.
- Mark
Mark thanks, great stuff.
I would not say that Operations Oriented are a superset of Detection Oriented. I was too quick to mention different types of applications. They are either types of applications or components within a single application.
The most interesting (to me, and I think to you) applications are the ones that combine detection and operation.
But there can be operation without detection. For example, plain order management. You’ve got all kinds of operational features and requirements, but not really any detection. Of course, it gets more interesting when you’re managing orders that are being traded as a result of signal generation, which is the detection part. And when you validate orders against calculated risk – more detection. Or when you scan for fraud – detection again.
Trading applications all separate out into detection or operation (sometimes combined into the same process, but separate in a coding sense). But no one calls it by those names. I did a bit of a breakdown for trading in a subsequent post.
The way trading applications have evolved to separate these two types of EP – that is a pattern that can be used by practitioners looking to get a handle on EP. And recognizing the breakdown can also help plan and even think up new trading software projects.
I posted an example non-trading use case: Foot Traffic Management. This is an application that uses both types of EP – take a look at the breakdown and see what you think.
Now that we’re getting more specific, the discussion is getting more interesting too – thanks to Mark for contributing and Hans to responding.
However, I still don’t see anything that would warrant such a classification.
I think Mark hit the nail on the head with the comment that Operations Oriented systems are a superset of Detection Oriented systems.
More so:
even the examples of Detection Oriented systems have “operations”; they may be trivial (notification alerts), but they are still there.
What is the point of event processing if there’s no action in real time? It would be then analysis of stored historical data, and it is not event processing.
Hans, your examples of operations without detection (order management, position management) again sound more like possible components of a CEP system, or something that can be used without event processing at all – such as in batch processes.
My view is that that was interesting exercise to try to introduce such classifications for people to grok the use cases for (C)EP, but the real examples (Mark’s Streambase cast studies and yours in other newer posts) seem much more helpful in achieving such a goal.
- Ed Y.
Hi Ed, I’m glad that you’re interested in at least part of this. I agree that we should thank Mark, who’s always been a driving force behind the (C)EP community.
There is still more work to do to do in terms of clear definitions for the classification and even the names of the types. In the mean time, read on – and hopefully you’ll find more of interest, even if you don’t agree with the classification.
You’re concerned that this classification is blurry. These kinds of analytical patterns often seem to start out blurry. Martin Fowler has commented on this many times in his own analysis and patterns work. So for the moment, I’ll take that as room for improvement and not a fatal flaw.
Currently, this classification is useful for analysis; maybe there’s some work to do before it’s useful for implementation. But I can tell you that it helps with tasks ranging from understanding strategic value, finding specific use cases, evaluating requirements, and even to a certain extent thinking about software architecture. As you said, every EP solution has components of detection and operation. It is quite useful to think about these parts separately, no matter how the implementation ends up.
Both position and order management are used both as independent systems and components of a trading system, depending on specific trading requirements. But in modern trading they are certainly used in real-time rather than batch mode.
To give you a little taste of one dimension of EP that may be analyzed with this method: consider two OSS EP projects Esper and Drools Fusion. These two projects offer quite different functionality, yet are both targeted to EP. So you can see that indeed EP software does not really abstract out functional concerns. You must still choose between different models of functionality.
You might be interested to analyze where each of these models are best. You can start this analysis by breaking the problem into detection and operational components. But of course you will find that additional classification is required – it’s not as if one type of functionality is best for all detection and one for all operation. However – the classification can serve as a good start. I will eventually get to a post on this, but probably not for quite a while – that choice is one of the most tough topics to tackle.
Since I’ve already discussed operation without detection, here is an example of detection without operation: Military and Intelligence.
Although it sounds good to have machines making operational decisions in this area, most people agree that this is not likely to happen in the near future.
But Detection Oriented EP is breaking ground in providing faster and thus better situation awareness. Humans still make all of the interesting operational decisions – the operational aspects of the EP are limited to data storage, alerting, display, etc.
In any case, I am interested in practical analysis of EP and to that end, I am not restricted to the classification of detection and operation. If I find a better starting classification, I will be equally enthusiastic about that.
[...] is exactly the kind of thing that lead me to describe the two types of Event Processing: Detection Oriented vs. Operations Oriented. I see this pattern over and over in pretty much every use of Event Processing that I have worked [...]