EP can help the “core crisis”?

So I was reading some posts about whether CEP (Complex Event Processing) is mature, when Mark Palmer’s excellent post linked to this blog entry on WS&T. And while reading that post, I noticed a link to this article, also on WS&T. The article describes how hard it is to find programmers who can write highly parallel code that would take advantage of new multicore architectures. The first comment on the article calls it the biggest crisis facing computing since… something.

Whether or not “CEP is mature”, let’s look at the software available right now, and I’ll call it EP for Event Processing so as not to even get near the question of what Complex means. You have here an ecosystem of software from StreamBase, Progress Apama, Coral8, Tibco, Aleri and more, much of which has been battle tested for years now. They are mature in the sense that they do what they claim and they don’t crash or hit you with lots of bugs.

And what do they do? They each provide a high level language that helps a programmer write high performance applications without spending much time thinking about (or even knowing about) the performance plumbing. Here you have a bunch of players that are all in the game of bringing down the skill set needed to build high performance applications. Some of them automatically thread and provide a framework that guides the user to write an application that is inherently highly scalable, without ever knowing what it is that makes it scalable.

So if we have an impending crisis brought on by the growing number of cores, we also have a potential solution available today! Several potential solutions, even. And are these solutions mature enough to be deployed in banks? YES! They are already deployed in banks. I’ve seen StreamBase churn through literally all the electronic trading messages in a firm, in real-time, and stay up for full weeks without a hiccup, just as it’s supposed to. All the major vendors have such stories. Their software works. It could be used to make many applications more scalable right now (see my previous post on the subject).

In the future, I hope that the people worried about the impending core crisis go talk to the EP people (on the CEP forum) about how these two ideas can result in not just a problem but a solution. And I’d hope that any organization looking to evaluate CEP understands that if they need to lower latency, then this can be a place to start, before investing in the next generation of hardware. Again, see this previous post on creating a uniform architecture that ensures scalability.

I’ll write about maturity and such in a later post. Sufficed to say that whether or not modern “CEP” products meet everyone’s needs or could be improved…they work well for what they do, and that is to bring down the cost and the skill set required to build scalable and high performance applications. Even if they don’t have the one feature that this or that person believes will make them officially “CEP”, they do have many features that were driven by customer demand, integrated into stable platforms that can drop the TCO and time to market for low latency, scalable software. And that is something, IMO, to pay attention to.