Horizontally scalable Workflow

One of the things I have been pondering over is horizontal scalability for workflow engines. Horizontally scalable systems have disrupted other parts of the middleware market, starting with database systems and, for some time now, messaging. So it seems only natural that workflow will be the next segment of middleware systems to be disrupted by the rise of distributed systems.

Why are horizontally scalable systems so appealing? OK, everybody has to admit that Netflix running 2700+ Cassandra nodes is impressive. But for most users it is not the netflix-like processing volumes that spike their interest. Instead, I believe it is the flexibility to scale as need arises and to be able to do so without having to change anything substantial in your architecture. So even if you do not have Netflix-like request loads to process, horizontal scalability is still a very valuable property.

Sometimes I get asked: “What is the number of Workflow instances per second at which I need a horizontally scalable process engine?”

The beauty of horizontal scalability is exactly the fact that you do not need to have such a number. You just look at what kind of load you need to sustain and put the appropriate hardware in place. And whatever your future processing requirements are, you will be able to adjust by adding more hardware.

Think about it, what is better: Going to your boss and saying: “Remember the servers we bought last year? Good. I need 3 more of those.” or “Remember the expensive DB system we bought last year when I promised it would be powerful enough for the next 3 years? Yeah… we need to throw that out and replace it with something even more expensive”.

Well, I guess that second guy is getting fired…

Workflow has many of the characteristics that fit horizontally scalable architecture well. Many users use workflow engines in order to automate their core business processes, as a result their workflow platform needs to scale with their business. You do not want to be limited in growing your business by a workflow technology that does not scale.

Many users also have peaks and lows in their processing requirements. For instance, German insurance companies need to do something fairly complicated at the end of the year which I never fully understand but which requires them to “iterate through all their customer records and start one or multiple process instances for each of them in order to do something or else” (I am paraphrasing here). Wouldn’t it be great if they could, just for that period of time, acquire more processing power and then scale down when they are done?

It sounds obvious, but then, all the best things do.

Zalando, one of our customers at Camunda is already using our current technology in a way which is more horizontally scalable. (Read about it here: http://tech.zalando.com/posts/camunda-meets-cassandra-at-zalando.html )

I believe that horizontally scalable BPM products will disrupt the BPM market in the years to come. Such systems will break the current status quo and I, for one, am definitely looking forward to it :)