Business value for Detection Oriented Event Processing (part 2)

I recently began a line of thinking about breaking Event Processing into two categories: Detection Oriented and Operations Oriented. In my last post (part 1), I began to talk about the business value of Detection Oriented Event Processing. Here I will continue that topic.

This post should provide ideas for researching uses for Event Processing. I am currently discussing the data analysis side of Event Processing, so all the ideas here involve data analysis. In other words, the Event Processing referred to in this post is treated as a tool for data analysis.

There are many complimentary technologies for Detection Oriented Event Processing, and I will get into the specific technologies in a future post. For now, I am focussed on business value and I will only go into the very surface of how Event Processing can help.  That may be frustrating for some readers, but it follows a certain pattern: 1) identify value and then 2) find the technical means to deliver it. We are currently on step (1).

In fact, a lot of this stuff can (and often should) be done without using Event Processing. And just because a scenario falls into one of these categories, does not mean that Event Processing can definitely help. There is no magic.

Ideas for uses of Detection Oriented Event Processing:

Lower the latency of existing data analysis

Lowering the latency of existing data analysis just means getting data where it is needed, faster. But as I mentioned in part 1 of this post, you only get value if the business can adjust to use the data at the new speed. Here are some ideas for how Event Processing can help lower latency of existing data analysis.

Rather than processing data in a batch, collect and process it incrementally as it is generated. By processing the data incrementally, the results are available immediately after the last of the data is available. Contrast this to batch-based processing, where the analysis only starts when the last of the data is available.

Deliver periodic, incremental results for existing data analysis. For example, some analysis may currently take into account one entire day’s worth of data; it might deliver more value by producing results throughout the day showing the analysis up until that point (including a result showing the analysis over the whole day).

Apply existing analysis over shorter periods of time. The same Performance Indicator that applied to daily operations might also apply to hourly operations.

Automate analysis that is conducted by people. Although often a challenge, it is increasingly possible and valuable to automate on-the-fly analysis that is currently done by people. Think about this: not too long ago, loan applications were approved by hand. There was no such thing as instant credit, a major driver of revenue for lenders today.

Develop new data analysis that requires lower latency

This area can be particularly interesting. Think of analysis that works much better when acted upon quickly. For example, predicting the immediate future based on very recent history (the past hour, minute, second or less) can be very accurate. But that accuracy degrades very quickly with time; a prediction (or optimization) based on the previous thirty minutes may be highly accurate for the next thirty minutes but almost useless a day later.

Find decisions or operational parameters that deliver value when they are adjusted quickly, then trace back their data requirements. If the data can be provided faster, perhaps adjustments can be made faster or can be allowed to take into account more data. The actual process of fast decision-making falls into Operations Oriented Event Processing, which I intend to cover in the future.

For example: the delivery of materials to a factory may be optimized by quickly adding or cancelling materials shipments from a storage facility, depending on the recent factory production level and materials requirements.

Increase the throughput of existing data analysis

Find and address bottlenecks in existing data processing. There are no miracles in Event Processing, but two common bottlenecks are disk storage and memory. Event Processing techniques are usually meant to process lots of data in limited memory and with little disk storage. Once these bottlenecks are eliminated, it may be possible to push through more analysis in the same amount of time.

For example: processing of high volume logs. There are many technologies that can help with this task, including map/reduce and other distributed computing. But those solutions can be (but not necessarily) even more disk intensive than just loading everything into a database. By treating each log entry as an event, it may be possible to leverage inherently resource constrained Event Processing techniques to process more log data at once.

Develop new analysis that requires high throughput

Sometimes there is just too much data to analyze; it would be nice to comb through all of it, but the return just doesn’t justify the cost. Again, without expecting miracles, Event Processing may help process data in a resource-efficient way. With lower resource usage, you may be able to finally analyze all that data that was otherwise going to waste.

For example: It might be nice to install RFID sensors throughout a store to record product movement. But the amount of data generated by a busy store would mean prohibitive storage costs. By treating RFID readers as event sources, it may be possible to pre-process all that data before it ever gets to storage, so that the summary data that is eventually stored remains at a manageable level.