Public Sector Event Processing

IBM has a web ad spot called The Smarter City, showing how IT can make a difference in the urban public sector. The web campaign associated ideas like e-Govornment and Intelligent Transportation Systems (ITS) with improvements in IT for healthcare, energy and education.

The short clips that introduce these topics make many references to gathering and using data in a more timely manner. I noticed this particularly in the Transportation, Public Safety and Energy & Utilities sections. This seems like a good fit for Event Processing.

Many years ago, I worked in intelligent transportation (ITS) and had the pleasure of attending a few ITS conferences. The big technical challenge at that time (the biggest overall challenge being political) was infrastructure: aside from the cost of all the sensors and cameras, the amount of data that streams in from truly comprehensive monitoring of roads and public transportation, seemed daunting to say the least.

The infrastructure problem is much less daunting these days, although still expensive. To justify the expensive infrastructure, we need to get started on the next technical challenge: the software.  Google already has some traffic speeds on its maps. But that’s just the beginning, right? Looks like Event Processing has a bright future in the public sector.

Separating streaming SQL from CEP

Julian Hyde has interesting things to say about streaming SQL and Complex Event Processing. It’s also a good introduction to SSQL.

Event Processing in elevators

Elevator makers are calling it Destination Dispatch – the process of scheduling elevators when the destination of each rider is known in advance, rather than after they enter and press the button. This is an example of the most common Event Processing value proposition – to make adjustments to business processes on the fly.

Event Processing example: Foot Traffic Management

This post continues the idea that Event Processing (EP) is more easily understood after dividing the field into two distinct areas. I started  with a couple of posts using this perspective to analyzing the business value of EP. Those posts are available from my EP reference page.

The most common request since then is to better explain the Two Types of EP concept. To that end, I first posted an analysis of two blog posts using the Two Types of EP.

Now I will introduce an Event Processing use case and then analyze it from the perspectives of Detection Oriented vs. Operations Oriented Event Processing. This is not a use case from an implemented project, but it’s plausible and a good example. Real use cases are currently hard to analyze because no one is willing to publish a detailed description of their use of Event Processing.

My background is in statistics, not marketing, and there are flaws in the marketing side of this use case. Anyone who has really looked at this particular marketing scenario will find many problems and questions to be addressed before the idea could be considered for implementation.

The use case: Let’s say that a chain of coffee shops wants to boost its work-hours sales to the urban business crowd. They have many chain stores in urban business centers and they want strategies to increase sales during normal work hours.

The chain has two large data sets available in-house to drive this effort:

• the (electronic) register tape  from every store
• data captured by a retail traffic counter (Google the term if it’s unfamiliar), which counts foot traffic entering and leaving each store

Let’s say that analyzing this data using basic queuing theory, produces the following results (this is not real data, I’m making it up to suit the example):

• Even during “peak times”, the length of the lines at most stores fluctuates greatly during any 30 minute period. There are 10 minute periods with very long lines followed by 10 minute periods with short lines.
• During peak periods, customers seem put off by long lines
  • They may walk out before making it to the register
  • Foot traffic may tail off when the line gets too long, and it seems likely that interested customers see the long line and never enter the store.

The coffee chain sees a group of customers that would like to make  a purchase, but are put off by long lines. Some of these customers may  go to another of the chain’s stores, but many won’t because there are now so many choices for the urban working crowd.

To sell to these impatient customers, the chain can come up with some traditional marketing ideas:

• They can open more shops in the area. Now the customers can spill over from a crowded chain shop into a less crowded one. But it may not be practical to open a new store just to capture peak traffic spillover.
• They can offer price incentives (discounts) during off-peak times. But this has a very negative effect. If the incentives are low, customers pay no attention. If the incentives are high, the result is to move the peak time but lower profit.

Since neither of these ideas works, let’s find a different and creative way to sell to these impatient customers.

We want to help customers avoid lines. This is a valuable service, but doesn’t have negative profit impact of price incentives. Maybe we can do this in a way that will “smooth out” the foot traffic to each store. In other words, try to get customers to pick just the right moment when the store has a shorter line, rather than a longer line. We know that even during peak selling times, a customer arriving at any random time has a chance to find either a long or a short line – can we direct some customers to the short lines?

To summarize, we would like to:

• Help interested customers to find a store with a shorter line
• Help interested customers who would like some coffee, but are willing to come at any point within a half hour window, to find the best time to arrive at a store
• Attract undecided customers by giving them the confidence and convenience of quickly finding a store with a short line
• As much as possible, prevent the very longest lines and wait times that discourage some customers

We will be creating some kind information service to direct customers, as much as possible, to shorter lines. That information might be delivered on a query basis or through push notifications.

Since this concept relies on customers believing in the system, we need to be reasonably accurate with any help we give. We don’t need to be perfect, but if customers don’t believe the information, they will quickly abandon the service. And this is where Event Processing comes in. Without some element of on-the-fly data analysis, there is no way to give good advice on line length. Because line length at any particular time relies on a very random effect of foot traffic, it’s not enough to consider seasonality (even combining annual, monthly, weekly and intra-day components of seasonality) combined with day-to-day time series. The extra step of real-time data analysis is required.

The meat of this use case will have to wait for a future post. But here are the roles of each type of EP.

Detection Oriented Event Processing

• Inputs are real-time information from registers and foot traffic counters
• Output events an estimate of the current line length and wait time
• Outputs short term prediction of line length and wait time by combining real-time data, with stored data that results from analyzed historical data:  multiple components of seasonality and day-to-day time series
• Considers the effects of this marketing campaign on predictions

Operations Oriented Event Processing

• Inputs are events containing line length and wait time info (current and predicted) from Detection Oriented EP components
• Integrate this information with the marketing strategy
  • Stores data for access by query-based systems like web or smart phone apps
  • Distribute the proper alerts and information to interested parties (customers, managers, etc)
• Interacts with customers, so is part of the customer relationship as well

We’ll see that it is interesting to look at this use case through the lens of Detection vs. Operations Oriented components is for a few reasons:

• The accuracy or effectiveness of the components are measured differently. The detection might work perfectly, but the operational rules make poor use of the data.
• The components are tightly integrated to the value of the overall concept, but still the business value of the two types can be viewed independently. The detection logic provides a data source that can be valuable to customers. The operations logic drives foot traffic and even becomes a part of the customer’s relationship with the brand.
• The development process is different. Developing the detection portion involves tweaking an algorithm by back-testing using large amounts of historical data. Development of the operational portion involves planning out a business strategy and analyzing the impact of various decisions, but no historical data is required to test the logic.
• The logic itself is different. One is focussed on analyzing raw data, the other on making business decisions.
• The maintenance and evolution over time will be different. The operational decision making will evolve over time, it might grow to consider (or at least not interfere with) many components of the marketing strategy. The detection logic will evolve if the model for foot traffic and line prediction changes, but the chain may find additional information that can be mined from the same real-time data sources.

Trading with the Two Types of EP

After writing about the Two Types of Event Processing (EP), I have mostly gotten requests for examples. And many people want trading examples. I will try to eventually write a detailed post about trading and EP, but for now I will have to settle for a post targeted at front office developers.

Here is the basic breakdown of EP for trading:

Detection Oriented Event Processing

• Signal generation – this includes liquidity signals for smart order routing
• A lot of (but not all) real-time risk calculations – but using the calculations is Operations Oriented, see below
• A lot of (but not all) real-time trade monitoring

Operations Oriented Event Processing

• Order management
• Position management
• Real-time P&L
• Order routing and checking against limits or calculated risk
• Responding to trade exceptions and events from real-time trade monitoring
• Old style smart order routing (e.g. route to each venue, cancel if not executed in a short time)
• Some or most crossing and trade matching – this might be surprising. I need to analyze whether combinatorial matching falls into Operations or Detection Oriented EP.
• Market data aggregation

Think about that breakdown and picture how they place different requirements on an EP (CEP) system.

I will elaborate more in the future. We will find different requirements and usage scenarios for an EP system. In fact, we find that different EP systems have very different strengths and weaknesses for the two different types of EP.

My links page for Event Processing

I’ve put together a summary page for my own writing on Event Processing. It’s not a big page now, but a reader can return to over time to find an organized view that is not so visible from a stream of posts.

Applying the “Two Types of EP” to recent blog posts

To help illustrate my discussion of Two Types of Event Processing, I will analyze two recent blog posts.

First blog post: an EP application

Take a look at the post So, What is This CEP Good For? by Marco Seiriö of ruleCore. Among other things, this post illustrates an awesome looking Event Processing application called FleetNotice.

At first glance, it may not be obvious how FleetNotice breaks down between Detection Oriented and Operations Oriented Event Processing. Hopefully I am about to shed some light.

FleetNotice receives events that show the periodic global position of couriers, as they navigate Manhattan. One assumes that the events come from cell phone GPS monitoring. As Marco explains: when you have hundreds of such moving objects, it is simply not practical for a person to watch them all.

So Marco has developed a system for watching all this moving data and trigger business actions using rules. To quote his example rule:

Alert me if Couriers Ben, Jerry and Mr Smith diverts from their designated route for more than five minutes and they take more than 20 minutes between the waypoints (40 minutes is ok if it’s between 4pm and 6pm on fridays, except last friday of month), but only on working days. Also alert me if the courier does not arrive before 6pm to it’s destination and alert me also if he does not leave the starting point before 1pm

From this rule, we can see evidence of both Detection Oriented and Operations Oriented Event Processing. But it may be surprising to learn that the rule statement itself is purely Operations Oriented Event Processing. The Detection Oriented Event Processing is hidden under the covers. This is a common scenario for a mix between Detection Oriented and Operations Oriented Event Processing.

The above rule statement is Operations Oriented because it takes in events with clear business meaning such as “courier diverts from their designated route for more than five minutes” and “courier takes more than 20 minutes between waypoints” and “courier does not arrive before 6pm”.

The focus of this rule is not on analyzing data to produce these informational events. The focus is on using these well-defined events to trigger some business activity: an alert.

So maybe now it is clear where the Detection Oriented Event Processing is hiding. It is logic developed by Marco (and/or his team) to take in a bunch of GPS signals and convert them to events like “courier diverts from their designated route for more than five minutes”.

Here are two brief points about why the distinction between Detection Oriented and Operations Oriented matters. There is a lot more to say on this topic, which I may get to in the future.

First of all, in this case the Detection Oriented and Operations Oriented Event Processing are developed by different groups. The Detection Oriented Event Processing is under the covers and developed by Marco. The Operations Oriented logic is developed by the business user (or business IT person). When Marco developed the Detection Oriented logic, he had some idea of how it would be used, but was not aware of every possible business scenario. When the end-user develops the rules, they could not care less about how the data is processed, they specify rules using business-level events.

If there is a bug in the Detection Oriented logic, the result will be an incorrectly detected business situation or activity.

If there is a bug in the Operations Oriented logic, the result will be that the wrong business situation or activity was referenced or the wrong business response was triggered.

Second, the Detection Oriented logic handles more complex edge or outlier cases than the Operations Oriented logic. The Operations Oriented Event Processing handles events that follow a nice contract. You will only get an event signalling “courier does not arrive before 6pm” under very specific conditions, which have direct meaning to the business. In contrast, the Detection Oriented Event Processing gets events that follow a much looser contract (the couriers may do what they are supposed to, but they may not). The Detection Oriented logic imposes a higher level of understanding and order on this chaotic set of GPS events.

Second blog post: a high level description of EP

Now take a look at 9 Predictions for the Future of Event Processing by Mark Palmer of StreamBase. This post is at a higher level, so to speak, than the post above. Meaning that it doesn’t go into technology specifics, but sticks to the broad concepts. Now StreamBase recently won very prestigious recognition as a technology pioneer. Since Mark is the CEO, we can be sure that Mark is very good at communicating the value of Event Processing.

Analyzing only the middle section of this post, titled What’s the problem? Why event processing? What is event processing? we see a very eloquent description of Detection Oriented Event Processing. Mark describes high-speed data analysis, which pulls relevant information (some might call knowledge) from a mass of moving data.

Do notice the contrast between Mark’s description and Operations Oriented Event Processing. Mark does not talk about what gets done with this information once it is detected. The eventual business activities that are triggered by this kind of high-speed data analysis – is the domain of Operations Oriented Event Processing.

Of course, it should go without saying that Mark is a pioneer in Event Processing and is fully aware of all of its aspects. This post clearly articulates the Detection Oriented aspect, and he’s written more than enough about the Operations Oriented aspect in the past.

Event Processing vs. EDA vs. event-driven

I’ve been writing a series of posts on Event Processing, structured by a breakdown of the topic into two broad types. I’ve done two (here and here) posts on the business value of what I’m calling Detection Oriented Event Processing.

This post will be about Operations Oriented Event Processing. I started on a post about business value, but decided to start by clarifying the role of Event Processing in IT.

Recall that “Operations Oriented Event Processing applications want to take trigger an action based on incoming events.” In other words, in this post I treat Event Processing as a tool for implementing event-driven logic. But not just any event-driven logic: here we are interested in ”events” that represent real-world activities and conditions that are relevant to the business. I’ll develop this last idea later in the post.

First, I want to distinguish Event Processing from Event Driven Architecture (EDA). Operations Oriented Event Processing applications are the typical components of an EDA. Specifically, they are a type of Event Processing Agent (EPA) that receive incoming events and trigger business actions based on those events. You can read about EPAs at the blog of Opher Etzion.

But it is not necessary to adopt an entire IT architecture style, just to use Event Processing. In fact Event Processing has a definable value all its own, in additional to its own patterns, best practices and tools. Since an EDA is made up of lots of Event Processing Agents, everything about Event Processing can be projected in some way on to EDA. An Event Driven Architecture is what you get when you set out to use Event Processing in (almost) every IT service.

Also, I want to separate Event Processing from event-driven programming. Ask a pundit about the key benefits of an Event Driven Architecture, and a large part will boil down to decoupling. Although this benefit is highlighted in seemingly every book on EDA, it is not mentioned in the Wikipedia article on EDA. The upcoming book Event Processing in Action has a discussion of decoupling.

The decoupling that comes from an Event Driven Architecture, comes from the use of event-driven programming. As demonstrated by this article, every use of event-driven programming, from GUIs to email, gets the benefit of decoupling.  But, to get a little technical, there is more to the picture than event-driven programming.

There are logic patterns that arise when using events that represent real-world business activities and conditions. The patterns are the subject of an upcoming book called Event Processing in Action. These patterns could probably be applied to any event-driven computing scenario, but they are particularly useful when dealing with real-world, business-level events.

So Event Processing is a layer on top of event-driven programming. It captures patterns, practices and tools that may not be useful for every type of event-driven programming task, but are very useful for “real-world” events.

Now here is an example of how real-world, business-level events differ from other event-driven programming.

Most GUIs are event-driven, in that input is delivered as events and the component mostly communicate via events. But the usual logic of a GUI component is not concerned with understanding the meaning of multiple, related events. Rather each event is applied, in order, to the state of each component. The component is not at all stateless, but is generally unconcerned with events that happened before or events that will happen later. It may be the case that certain components are, in fact, worried about multiple events. But their concern is usually fairly simple: “wait for a button click” or similar.

Compare the GUI scenario to the kinds of events that are seen at a business level. Let’s say that an Enterprise Resource Planning system places orders as events. This is a fairly common scenario, all it means is that the ERP system sends a message to each vendor when it would like to place an order. But the systems that handle this “Order” event, use logic that would not enter into a GUI. Every system that touches this event considers multiple types of events that came before or after. For example, they may check for duplicate orders, they may wait for a statement of credit, they may ask for human approval. And the ERP system is also waiting for future events: it would like the supplier to confirm that the order will be delivered on time and it would like the receiving department to show that the order has indeed been received.

This consideration of multiple types of events that came before or will arrive in the future, is a pattern that regularly arises in the processing of business-level events. And it is a key focus of Operations Oriented Event Processing.

Also, underpinning all networked, event-based systems are very technical issues like load management, quality of service, failure recovery, threading, caching, persistence, algorithmic performance/complexity and inter-process communication. And all of this stuff needs to be put together properly in order for any event-based system to work properly. In fact, different use cases show radically different requirements in almost all of these technical areas.

Event Processing also looks at the tools that are used to implement event-based systems. The aim is to reduce the barrier to entry by allowing the tools to handle as much of the complex technical work as possible. This helps the IT architects and developers to focus on their main task: creating business value.