Faster and Furiouser

Evel OrangeThe Flowable team have been busy inspecting how the Flowable engines operate at high loads in real-world configurations, the results being discussed in my last post, concluding that the upcoming release is faster allround. As hinted there, this was still just a baseline comparison with previous versions and there was another recent feature that was going to be added to the benchmarking. The results of that are now available here.

This benchmarking compares the upcoming version with a recent feature (“async history”) disabled, as it was in the previous benchmarks, and then with it enabled, which has not been benchmarked before now.  Sadly, this doesn’t tell us directly how the very best performance compares with older releases.  We can do some simple sums to give us a combined percentage improvement, which is shown in the table below:

Test

% Faster than v5.23

% Faster than v6.0/6.1/6.2

StartToEnd

53%

130%

AllSequentialTasks

109%

185%

ManyVariables

95%

55%

ParallelSubprocesses

47%

92%

TerminateUserTasks

71%

78%

What’s most rewarding about these results is that they allow you to evaluate more complex process structures and configurations that reflect more realistic uses of the Flowable engines. The tests are all available as open source, and we have run them against a couple of other BPM engines, giving us even more confidence of our leadership in performance and scale.

Of course, speed isn’t everything, which is why the next release will have some additional capabilities to help cover wider business use cases. Those are topics I’ll cover in future posts!

Advertisements

Keeping ahead of the zombies

Zombie_1Competition is great, it gives you a marker against which to compare yourself as well as a drive to innovate to keep at the front. The difficulty with competition in the software industry is that everyone starts at different times; it’s very rare to have all the competition starting at the same time (or may be not so rare recently with the cryptocurrency and AI hypes). This means it can be difficult to compare vendors at different stages in their maturity. We’re all used to the beauty parade of feature lists, which tells you what you’ll get now, but may be no indicator of what you might get next year or the year after. It’s very easy to get swayed by all the stuff happening around the outside without noticing the core has gone cold: dead on the inside. It might be a very well-dressed and mobile zombie, but it will start shuffling and groaning before too long.

That’s why the good old performance benchmark can be a great way to show that the heart is beating stronger and stronger. Of course, benchmarks can be gamed, but if you make the benchmarks public and open source with a range of complex scenarios, it makes everything clearer and available for meaningful discussion. Now, some of you will say “making stuff faster doesn’t make it better in the grand scheme of business needs,” and in some cases you may be right. However, in my experience, when I’ve seen a team thinking about how to re-architect, adapt and adjust to get more out of the engine, then they gain new insights that ripple out through the product. I’ve seen it can also help you focus on what’s really important for the product in the real world, not what sounds good in a marketing tweet.

Why am I talking about this now? Obviously, we’ve done some serious conceptual and technical work to boost the performance of the Flowable engines that leaves the opposition shuffling in our wake.

The deep details for the technical types, as well as the easy graphs for types like me, are in this Flowable blog post. There’s more to come after it too, so this isn’t the end of the performance story before the next software release. We think it’s important to push the boundaries of performance for BPM, all the time retaining transactionality and dynamicism. Of course, we also watch what’s happening with technology: Spring Boot 2? Of course we do.

Zombie_2

Is your Business Process provider putting all their efforts into trying to dress up to look contemporary? Are they dead on the inside? Are they a zombie?

 

Machine Learned Decisions in Business Process Management

cyborg-smile2

With all the fluff and buzz made about AI, it’s very easy to either hype or dismiss the benefits of the technology for use in enterprise and government applications.  Business Process Management is a well-established technology that is used to model the flow of work between systems and people: collecting data along the way, branching down different paths, running business rules, and very often relying on human decisions.  With so much context available to a process, you would hope machine learning could offer something of real value.

This is precisely what some of our customers have been looking for.  Remove the mundane human decision-making activities, leaving people to work on more valuable tasks.  Take this scenario: you have a process where someone has to review some citizen or client information and make a decision about what should happen next.  Now imagine a machine is watching this; patiently analysing possible correlations between what is known and what is decided.  Eventually, the machine reaches a level of confidence that you allow it to advise the users (I don’t necessarily mean like 2001, HAL: “I’m afraid I can’t let you do that Dave”).  Take things a step further and it may reach a level of confidence that you just let the machine make the decision alone.

All sounds wonderful, but what might happen if the machine makes the wrong decision in one instance? When processes are audited, how can you explain what happened and why?  Well, it’s hard if the machine learning is a black box that you can’t look into or understand.  What’s really needed is that the machine learned decisions are represented and executed in a well-understood way, both by systems and humans.  Sounds fantastic.

That’s what you can do with Flowable.

Look at the simple loan request process below.  There’s some initial data gathering and then a user task, “Loan Review”, to check the information and decide whether to accept, reject, or pass for further review.

ml-process

 

You can use Flowable to automatically locate human decisions in a process and record all the data reviewed by users, along with the decision made.  This historical data is fed to a machine learning algorithm that analyses it and ultimately generates a decision tree.  We can now use this to predict what decision should be taken.  You can show or use that in many different ways with user.

 

ml-decide

To take it to the next level, we can convert the decision tree into rules in a decision table (DMN standard, or course).  This is something that can be readily understood by business users, as well as executed by systems.  At this stage, you might just get the derived rules validated and signed off as part of a process improvement lifecycle, and a new process defined using the table instead of a human task.  Alternatively, you could use one of Flowable’s unique capabilities to dynamically modify a running process and switch the user task out automatically.  You could define confidence levels in predictions to trigger a transform of the menial decision-making from human to machine.

You can see such a system being run live with full technical detail in front of an audience at Devoxx 2017
.

Engines of Disruption

FlowPistols_4No, not another punk or metal band, but another open source engine that joins the BPMN (Business Process Model Notation) and DMN (Decision Model Notation) engines already in the Flowable Project. There was a gap between B and D acronyms, so we’ve filled it with a C. The CMMN (Case Management Model and Notation) standard is now at version 1.1 and we’ve seen it being used increasingly at customers for business critical solutions for a while.  This prodded us to develop a completely new CMMN engine written from the ground up with the same principles as the BPMN engine that has served many so well.  Those principles are to be lightweight, fast and efficient processing at scale.

Before the techno-babble: why is any of this interesting to business people trying to solve the digital transformation project that their CEO/CTO/CIO/CDO has dumped on them? Well, CMMN offers a different way of looking at things – a more declarative, or data-driven view. Many business problems are case-like: an insurance claim, recruiting an employee, on-boarding a client, dealing with a citizen request. There’s an event, person or thing that a range of different activities and data are focused around. Those activities can be approval, data collection, validation and communication, either through processes, tasks or integrations with external systems. There may be any number or complexity of conditions for starting an activity, or dependencies on previous life-cycle stages of a case. CMMN gives you the option to express your problem naturally. You can achieve the same with BPMN, but there’s extra work or overheads that make it less effective (also, the BPMN diagram would be an impenetrable mess compared to the CMMN one). A light, scalable and fast native CMMN engine gives you the option to express your problem more naturally, at the same time giving you excellent execution performance.

Because CMMN takes a different perspective to the world than BPMN, instead of trying to mangle its execution on a BPMN or rules engine, we knew doing it properly meant a specialized engine. Working with a large number of sentries (potential triggers for processes, tasks and other activities) in a case model needs an implementation that efficiently scales and performs.

Of course, there’s a huge amount of interplay between processes, cases and decisions (BPMN, CMMN and DMN), each potentially driving or being driven by another. For this reason, the Task, Variable, Job and Identity Link Services in the BPMN engine were refactored to be separate services that could be used by any of the engines independently or shared when operating together. Now that these are true services, it also means that other custom engines and applications can use them as well. More importantly for us, it doesn’t matter if it’s a BPMN User Task or a CMMN Task, it goes into the same Tasklist for the user or team. The services even manage their own schema independently. If you’re into microservices, we’re giving you even more fine-grain service actions that you won’t find in other implementations!

With the open source Flowable engines, you have what you need to power your digital transformation or customer engagement solutions: the engines to power your ability to disrupt your market and do things differently.  Download the new Flowable 6.2 bundle and take a look at the release notes to learn more about using BPMN, CMMN and DMN.

We’re putting the band back together

BluesBrothers

Mrs. HH: Don’t you “Don’t get riled, sugar” me! You ain’t goin’ back on the road no more, and you ain’t playin’ them ol’ two-bit sleazy dives. You’re livin’ with me now, and you not gonna go slidin’ around witcho ol’ hoodlum friends.

PHH: But babes, this is Joram and Tijs, the Flows Brothers.

Mrs. HH: The Flows Brothers? Shoot! They still owe you money, fool.

Joram: Ma’am, would it make you feel any better if you knew that what we’re asking PHH here to do is a BPM thing?

Finally, the band is back together.  It’s been a while, but at last Joram is out of the cooler.  We’ve been making good progress while he’s been in stasis, and now things will really start to accelerate as Tijs gets back his technical sparring partner.  We’re not expecting it to take long for Joram to get up to speed, so look out for new things in the coming weeks.

I’ve been managing engineering teams for a while now, and have had conversations with other managers about the different types of engineer: the loud personalities; the quiet ones working silently in dark corners; the principled; the hacker; and occasionally the well-rounded human.  Which ones are the best?  Which types produce the most innovative or the most stable product?

Well, as always with people, there isn’t a simple answer.  I’ve seen the quietest individual come up with an exceptional insight and the most annoyingly loud ego produce something where you have to say, yes, this time you were as smart as you thought you were.  Ultimately, it isn’t about the individual, because they don’t scale.  A soloist can be really impressive – total mastery of their instrument – but your breath is only taken away when the full band tightly kicks in. It’s about how people work together that makes the difference, and finding complementary skills and personalities is hard.  Fortunately, people will often find each other, deliberately or accidentally.  This is the case with Joram & Tijs.  Individually, they are among the best engineers I’ve ever worked with, however, together they’re an awesome combination that inspires others.  A good band needs a front man or two, to drive them to new places in an ever changing music landscape.  The band’s back together and we’re going to make people listen.

We were awarded Rising Star at bpmNEXT 2017 last month.  I was privileged to present some of the groundbreaking technology created by this team, which you can watch here. All bpmNEXT talks follow the format of 20 slides automatically moving forward every 15 seconds followed by 15 minutes of live demo. Let’s just say it’s a “stimulating” format the first time you do it!

The Flow Pistols: Anarchy in the BPM

FlowPistols_3Last week I showed some of the new dynamic process capabilities of Flowable V6 at the bpmNEXT conference and I thought it might be nice to give a taste of that demo here.  The demo scenario was that we’re organizing a tour for the Flow Pistols and like any well designed processes, it can’t always cope with what the real world throws at it.  I showed Flowabe 6 running within the edoras one Case Management platform, with a case representing the overall tour and a process to run for each gig that needed arranging in the tour.

The process is quite simple: gather some details about the gig, and if the expected audience is above a certain size, then organize stadium-scale sound and light equipment; otherwise, just arrange transport for the band’s own gear.  The process diagram is:

arrange-gigInstead of just allowing random dynamic actions anywhere in the process, we can tag user tasks as being able to allow the user to insert dynamic tasks or processes at that point.  I’m going to tag two of the tasks as allowing either dynamic tasks or processes.

config-dyn

Once the process is deployed, I create a case for the tour and kick it off with a gig in Paris.  The case view allows me to start any appropriate processes associated with the case.

caseview

If I click the Start process action on the right-hand side, I can pick from the available processes for the case and get its start form – in the Arrange a gig process, this is just a dropdown to select the region for the event.

start formOnce the process is started, the first task appears, requesting the gig details.

task1

We can complete the details needed here, including the expected size of the audience.  We’re expecting a relatively small crowd in Paris, only about 4000.

form1

Because it’s going to be a small gig, we’ll be taking the band’s own equipment and the process takes us to the task to arrange transport for the kit, getting details of the vehicle carrying it.  Notice the additional actions to add dynamic tasks or sub processes.

form2

Inject a parallel task in a live instance

Now, imagine I had got to this point in the process, when I heard from the company that usually transports the gear, saying that because of Brexit, the French government was insisting on UK commercial transport taking out an additional insurance.  Luckily, we already have forms we use for non-EU countries, so we can just reuse one of those.  Of course, I could just create an ad hoc task that’s linked to the case (or as a child of a user task), but in this case, I really do need it as an independent task that can be done in parallel to booking the transport.  Also, I do need both tasks to be completed before the process continues.  What I really want is to magically change this instance to a process more like the following:

arrange-and

This is exactly what I can get with dynamic tasks.  If I click the Add dynamic task, I get a form to define the task.  I can give the task a name and select its form.  It could have been configured to more specific, predefined tasks, but I’m keeping it simple.

add-task

Now, I have two tasks in my list:

tasklist1

I can now complete the insurance task and booking transport task to allow the process to continue, eventually to completion.

Inject a complete process into a live instance

Well, that was fun, but not exhilarating.  Let’s arrange a gig in the US.  I’ll start the Arrange a gig process again and this time take a different path.  If I arrange a gig for New York and say I’m expecting a crowd of 15,000, then we go down the path needing stadium equipment to be booked.

stadium-form

Just at this point, I see that there has been a presidential tweet early this morning saying that some planned energy tax legislation is being brought forward, effective from today!  Luckily, I had been planning for the tax, so already had someone define the process that modeled it.  What do I do?  Try and create a new process model that includes this, deploy it and them migrate active instances to it? If I had enough time to learn the intricacies of BPMN models and migrations (plus some developers ready to handle the cases where it just doesn’t seem to want to play nice), then I might.  Alternatively, I can just click the Add a sub process action.

add-dynproc

I can give a name to use and then select the USA Energy Levy process.  This time, the entire process will be inserted in parallel to the current task and started, so, as you’d expect, an additional user task has been opened.

tasklist2

Before we continue, let’s take a peek inside the engine to see what’s going on.  By starting the Flowable Admin app, I can inspect all kinds of interesting things about the engine, but I’m just going to look at that active process instance.

admin-inst1

If I click the Show process diagram action, I can see the familiar Arrange a gig process, but with one difference – the Book light and sound systems task isn’t a user task.

dynproc-root

If I click on it, I zoom down a level and can see a parallel gateway with the open task and a sub process.

dynproc-sub1

If I then click on the subprocess, I can see the energy levy process and its open task.

dynproc-sub2

Back in the case view, I can complete the tasks for the energy tax, giving details about the consumption of the equipment.

energy-form

Sadly, because we’re using low energy gear, we end up on the path that gets hit by the tax.

levy-form

And it looks like we have to pay an additional tax, because being concerned about global-warning means we’re probably communists.

wall-form

Of course, demos like that can look amazing, but you never know how hard it was to do behind the scenes, and what corners were cut.  So, here’s the code that was used to insert a subprocess into a running instance:

DynamicEmbeddedSubProcessBuilder subProcessBuilder = new DynamicEmbeddedSubProcessBuilder()

.id("customSubprocess")

.processDefinitionId(processDefinition.getId());

dynamicBpmnService.injectParallelEmbeddedSubProcess(task.getId(), subProcessBuilder);

Inject a process into an injected process in a live instance

Yes, I did just say that.  If you can inject a process into a running process instance, why not inject a process into that injected process as well? Sounds twisted, but think again about dealing with the real world – you never know what you might have to deal with at any time.  Also, think about it in the context of having many process snippets that can be used as needed by the user or by a smart subsystem.  Then think about a chatbot that follows a process snippet when the user asks something, and while following that path, the user suddenly has something else come to mind that makes their conversation take a diversion. The user journey could be created using process injection.  Here’s an injected process to update the Accounts Payable system in the injected process that used the new tax for the first time – see the path in the top of the view.3levelinsert

Where does it go from here?

As we got a hint of when looking under the covers, we can visualize, audit and even reuse these one-off instance variations.  Existing analytics can continue to be used, and there’s an exciting potential to use Machine Learning to recognize the cases when exceptions should be considered for a case in hand.

This was just intended to illustrate something we can do now with Flowable 6.  As we move forward, we will continue to add features that exploit the dynamic manipulation of process state in user-friendly ways.  When looking at other BPM engines, see if they currently can do this.  A big thank you to the guys who put in the hours to make the magic happen for the demo – I was the dancing monkey.

FlowPistols_4

This Cat’s Out of the Bag

And so the next stage of my professional life begins.  When I left Alfresco, most people expected me to start a new company around the Flowable project; most people, including myself and the core team.  Certainly, it was possible, and I had a great response from the VCs I spoke with, to the point where we had decided one to go with.  Then “stuff” happened.

It saddened me that many people thought this “stuff” was not surprising.  Certainly, some of it might have been expected, but I was taken aback.  I’m not going to go into this now.  My approach is always to win by being better.

edoraswareMeantime, another opportunity surfaced: edorasware.  Right from the beginnings of ACTIVITI, we’d had a great relationship with people from edorasware working on the project; they even contributed a complete subproject for process simulation.  They built a Case Management platform based on ACTIVITI and have been selling it very successfully, so were keen to understand what the implications of Flowable were for them: we started talking.  Our ideas about where to take process technology forward meshed and we all got a bit giddy (if you don’t get giddy about your work occasionally, I feel sorry for you).  With our departure from Alfresco, they were now interested in working with us on Flowable, to the point of merging.

This gave us a conundrum: following the VC model with “full control”, and building up a company and infrastructure from scratch, as well as developing enterprise product; or, a bit less control, but straight to market with a BPM and mature Case Management offering.  Their customers also intersected nicely with where we’d seen a lot of interest in process applications.  Also, importantly, the BPM world is a busy place and there are definite pivots happening, so we decided that time-to-market was critical: we didn’t have a couple of years to build ground up from a new open source project.  You can get support and maintenance from an experienced team on the open source version of Flowable today.

Over this year we’ll transform the company from its core outwards (including the brand), organically building on what are already mature products and organizations.  We may still take investment from VCs, but it’ll be from a far more advanced position.  Most of all, it’s going to be great working with people who have real vision and passion.  Expect great things from us.