You asked, we answered – FlowFest Q&A

FlowFest 2022 was a big success. People from all over the world gathered online to discuss business process automation and Flowable. Considering the many questions we have received, we decided to create a post answering all your questions that came up in the Tech and Architect stream.

Here they are:

Tech Stream Q&A

Tech Stream | Session 1: What's new in Flowable? | Tijs Rademakers

When will the new Flowable Design application be available to Flowable users?

Before releasing the full Flowable Design application, we will make a version available in our new Flowable Cloud. The initial version will focus on CMMN, BPMN, and DMN. After its release, we will work on launching the full, new Flowable Design application. We cannot name a specific date yet, but it should be sometime in 2023.

Any plans to support MongoDB?

There is a proof of concept to implement the BPMN engine persistence using MongoDB which you can find here: https://github.com/flowable/flowable-mongodb. Currently, there are no plans to extend this support, but this also depends on the demand for our user base.

How do you scale Flowable Work horizontally? Do the sessions make it impossible to scale?

Flowable Work is by default stateless and therefore can be scaled horizontally without any issues. When using an authentication mechanism like OAuth, sessions are currently needed due to the Spring framework. But with this approach, the scaling can still be done horizontally by distributing the sessions evenly across the cluster.

Are these new features/designs available in Flowable Open Source?

The new Flowable Design application will be available in the Flowable Work enterprise product, but once available, you will be able to get a sneak peek of it in Flowable Cloud.

Any plans to migrate to the new Jakarta EE 9 and 10 java package names? (from javax.* to jakarta.*)

Yes, we are working on an Open Source release that will include this migration.

Any plans to combine both, the Flowable Work UI and the Flowable Control UI?

Flowable Work UI is an end-user application, while Flowable Control is a system administration application. As they serve different purposes, complementing each other, we do not plan to combine them.

Are there any improvements to the modeling flow for multiple process analysts collaborating on the same application? UI Performance, excluding WIP definitions?

There will be improvements in this area, with more export and import options and the usage of the already existing model locking functionality.

Tech Stream | Session 2: Taking forms to another level | Orlando de Mauro

How can you adjust the Flowable host URL and how is authentication handled in the FlowApp integration of another react application?

This can be adjusted by providing a base URL with the following javascript variable:

window.flowable.endpoints = { baseUrl: 'myHost'}

The authentication should be handled by the react application and therefore it needs to work with the REST service authentication that is used.

How do Flowable Forms compare to jsonforms 3.0?

Three aspects make our form more powerful than jsonforms 3.0:

Are any other events planned, e.g. onLeave, onHover?

Yes, we are always looking for ways to add more event support based on customer feedback.

What's the plan with the form editor and the new design presented earlier?

The form editor will be part of the initial public Flowable Cloud with the Flowable Design application that will be made available as part of it.

Do forms have conditional question capabilities? e.g. skip questions 5 - 10 if the answer to question 4 = "Male"

Yes, this can be implemented using expressions in the form model.

Will the Flowable library be available in Flowable Open Source later?

The current plan is to keep this as part of the Flowable Work enterprise version.

Tech Stream | Session 3: Scripting for smart solutions | Joram Barrez

Scripts are powerful. Scripts can change history/auditrail, deployed definitions, ++. Is there support to limit its power?

Yes, with scripts there are security risks, and we will be working on features to sandbox the script execution.

How are script exceptions handled in Flowable?

In general, a script exception would lead to a Java exception. But there are also options to use BPMN errors in the script to handle a script exception and throw a logical BPMN error that can be handled in the process model.

How to test scripts? In IntelliJ or VScode we have immediate feedback on a syntax error, we also have easy creation of Unit tests.

Validating and testing a script is indeed something that needs specific functionality. In Flowable Inspect we already have the option to evaluate expressions and this could be extended to scripts as well. But options need to be analyzed for this.

How to invoke a soap-xml request instead of JSON REST request into a script?

Currently, the focus is to support JSON REST requests in a script, and this can be extended to SOAP requests when the demand is there.

Is a script debugger support planned? At least for Groovy the debugger support is not complicated.

A script debugger is not yet planned, but is being validated and analyzed to see what the options are.

Do we have flw.setTransientOutput variant?

Yes, this is supported.

Any plans to better support exact decimal types as a native Flowable data type (e.g. Java BigDecimal instead of Doubles)?

No plans, but it is an interesting idea. I assume this is to have better precision. The main problem is storage: when persisting to a database, the value would need to be 'converted' to a double which could mean losing precision. Storing it alternatively (e.g. as a String) would mean that mathematical operators won't work.

How is this new sandboxing different from flowable-secure-scripting jar?

The new sandboxing is built-in at a JDK level as a first-class feature, which gives way more means to control the security. The flowable-secure-scripting jar approach has known limitations that don't make it 'safe' in all use cases.

Is there a canonical front-end API documentation page for the Enterprise product, including these new flw.xxx methods?

Note that the flw.xxx methods shown in the presentation are backend functions. However, there exist flw functions on the frontend too. For the frontend functions, the docs are here: https://documentation.flowable.com/latest/forms/basic-expressions/#flw-functions

Tech Stream | Session 4: Event-based sharding for hyper-scaling | Filip Hrisafov

Which JMS broker are you supporting? For example, do you support AMQ Broker using AMPSSL protocol?

There is general JMS support using the features provided by Spring JMS client. ASo all supported protocols in Spring JMS are in general supported.

Are outbound channels all about fire & forget? Is there a way to correlate outbound-inbound channels for synchronous request/response messages?

Outbound channels are meant for sending messages and this can be fire & forget, but can also be waiting for a correlated response. In the event model, there's an option to define a correlation identifier and the response message can be matched against this correlation identifier to trigger the correct process / case instance.

A Question about multiple databases - Are the 3 databases need to be kept in sync? How are these configured?

The multiple database approach presented is meant for very high scale / volume use cases. In this case, the databases will contain separate data that is not kept in sync. It's possible to connect the databases to one ElasticSearch to be able to have a single view across the 3 databases.

Do the databases have to have the same models or can you do also sharding on a process definition level?

In general, the databases would have the same model, but when the use case is there to use this approach for sharding then that's also an option. Flowable doesn't limit this.

I think the current configuration options only allow one RabbitMQ connection (host) - what if we need to consume events from multiple sources?

It is possible to configure a custom org.springframework.amqp.rabbit.core.RabbitAdmin client for a specific RabbitMQ connection and thus consume events from different sources.

What happens when sending an event, but the listener is not ready yet? Can I somehow store the event until the listener is ready to listen?

There is an interface available that can be implemented for non-matching events, so that these can be stored for later processing. Flowable also provides options to make sure the listener is active before sending the event, to make sure that this case doesn’t happen.

Q&A Architect Stream

Architect Stream | Session 2: Low-code integration with Data Objects | Roman Saratz

Is there integration with MongoDb using Data Objects or is it pure REST API based?

Currently, there is no support for MongoDB with Flowable Data Objects. You can wrap a MongoDB with a REST service and call the REST service, though.

Are there limitations on Data Objects’ size and number?

There are no fixed limitations enforced by the product, but there might be some limitations provided by NFRs or network infrastructure. If the data loaded from an external REST service takes too long or the payload is too large, the network might interfere.

Is there a standard way (push event from an external source by example) to sync Data Objects from external data sources?

No, there currently is no standard way. The only place, where this could be relevant is to updated, indexed data for a specific data object. In that case, you could listen to external events via an MQ system about a data object and then trigger a reindex for the instances that reference the changed data object.

Do Data Objects have any options for caching to improve performance? Batch operations?

There is caching for data object instances that are processed within one single transaction.

Are Data Objects available with Webservices instead of REST-APIs?

Currently, we only support REST out-of-the-box. You could hook into the service registry to convert XML to JSON with the RestServiceInvokerEnhancer interface before requests and/or responses are produced/processed.

Does the REST service support the AWS Sigv4 authentication type?

Not out-of-the-box, but you can hook into the RestServiceInvokerEnhancer interface to create a signature of your request and inject it to your request header. Starting with release 3.13 [AJQ1] you will also be able to do this with a script directly in the service registry model.

Can we maintain hierarchical Data Objects like parent or child? How can that differentiate in webservice calls?

Currently, this is only possible in Database Data Objects.

Is it possible for Data Object data table to use filters and sort with REST like data table?

You can’t use the out-of-the-box sorting and filtering like the normal data table, but you can pass in filters from outside the data table. For that, you need to define an additional Data Object Source Operation with filter parameters. These filters can then be passed into the table from outside variables (See data object operation configuration in the data object data table).

Can I have an external database for a database-based data-object?

No, this is not possible at the moment.

How does this compare to the use of ElasticSearch and fetching data through the ElasticSearch query language in the process flow?

ElasticSearch is used for fast searching and filtering, but is not the source of truth. You can delete the entire ElasticSearch index and rebuild it with a reindex.

Architect Stream | Session 3: How to survive at scale | Greg Harley

How do you scale Flowable Work?

Scaling Flowable Work is typically managed by adding nodes/servers connected to the same database. Extreme scaling can be managed by sharding the event manager work, using dedicated GUI nodes and dedicated event management nodes.

Is there a way to define a duration of retention for historical data instead of deciding to store them or not?

The duration of storage is defined in the housekeeping logic. If you use the process-based housekeeping task you can set durations selectively by process case or definition.

Can we use any variable type when using transient variables?

Yes, transient variables also support binary data (such as serialized classes).

Can we perform housekeeping for active instances?

Housekeeping is in general for completed instances. If you are trying to manage active, but defunct processes that have never completed, use the Java API to collect them and complete them first. Afterwards, you can use housekeeping to clean these up. If you have defunct active processes, I would look at your process logic and perhaps add a terminate timer to close it out if it has been inactive for an extended period.

We are not using ElasticSearch, all our historical data is stored in the database. What are we missing by not using ElasticSearch?

ElasticSearch provides many advantages. The most important one is the ability to query without impacting the runtime database. ElasticSearch queries are also much faster than database searches. Also, aggregated data can be retrieved efficiently.

Is Housekeeping available in Flowable Open Source?

Housekeeping is available in Flowable Open Source but only the default setup in application.properties is available. The Housekeeping task and ad-hoc enablement in Flowable Control are only available in the Flowable Work enterprise product.

We have found that calling the java API to delete process instances is quite slow. How long should it take?

This is completely dependent on your database performance. That said, housekeeping uses batch deletes to improve performance. Using the regular delete method will search for and delete each instance individually with is significantly slower.

What is the number of concurrent tasks you can run in Flowable Open Source without a decline in performance?

This is like asking how long a piece of string is or when does a hole become half a hole. The answer is totally dependent on your environment and architecture. Flowable is known to be able to execute many thousands of concurrent tasks and processes.

From housekeeping -do we have provision to move history data to another database as a backup?

Not at this time, although this would be a nice feature. Customers can open a feature request through our support organization.

Is there a way to delete active instances e.g. to clear out a test environment regardless of where any instances are in the process?

Housekeeping does not support the deletion of active instances. In a development or test environment, I typically delete the deployment which will delete active instances.

Architect Stream | Session 4: Flowable Futures: new designs and process chains | Paul Holmes-Higgin

Do you plan to add a test module?

At some point in the future we'd like to add the capability of including test runs created using Flowable Inspect to an app model, which then could be used to execute tests as a pre-deployment stage. That way, tests could be run to validate an update before it is made live. There's no planned date for this capability yet.

Will the modeler and other parts of the open source version UI get updated as well?

The open source UIs have always been example apps that show how the Flowable REST and Java APIs can be used.  We've had very little community input to the development of these apps, so we do plan to deprecate them. Our open source focus remains, as it always has, on the Flowable execution engines, which continue to get a huge amount of improvement from Flowable staff, as well as a constant stream of pull requests from the community. We will be making the new Flowable Design available for free use with the community engines, and we'll see how things evolve from there.

How does Flowable’s future correlate with Flowable Orchestrate?

Flowable Orchestrate is primarily there to provide support and bug fix releases to organizations using the Flowable engines. Often, these are people that started building something with the open source engines and have made it so successful it's a critical part of their business. Getting support and bug fixes from the people that build the engines is plainly rational or even mandatory for some cases, and that's the role of Flowable Orchestrate. There are some other benefits, and some of those will evolve over time as makes sense to uses of Flowable Orchestrate with respect to the overall development of the future picture.

Do you have any proof of results that show flowable is best for business/clients? We will try to execute (performance etc..) same steps with other products.

What's best for one client may be completely different to what's best for another.  For some, it's performance, and Flowable has been able to win deals based on its performance when compared to others. Choosing the right configuration for the specific problem is essential if you're wanting to scale efficiently to millions of cases or processes an hour, so don't take someone else's benchmark as necessarily equivalent to your own use, (so much can depend on the database and network rather than Flowable). Sometimes it's the flexibility and capability of the execution engines, with features like dynamic process injection, CMMN, visual debugging, event integration and so on. The other reasons we're told we've won a deal include the range of automation services available - Case Management; transparent data access from DBs and web services; low code productivity; content management; easy integration; configurable UI and sophisticated forms handling. Often, it's comparing creating the total solution that wins it, in terms of the speed and least effort needed to get what's wanted up and running.

How do we handle deployments of apps with Flowable Jet?

ANSWER: The technical details are at early stages, however, a deployment approach that is secure is fundamental to it in some use cases. We're working on the basis that the execution engine should pull app models from trusted sources rather than be pushed to; also, that remote invocations of processes and cases is considered in a highly secure way. None of it is rocket science, so it's making sure we choose the right approaches, and with much of our business being with the finance sector, we have plenty of customers to validate our approaches with.

This Flowable jet sounds like some of the old BPEL solutions which vanished with the rise of BPMN, can you highlight the differences? and if a standard is used

Like a lot of IT, it can feel like we're revisiting things that have been done in the past, just in a slightly different medium or context. BPMN has become far more widely accepted and used than BPEL ever was, and architectures are machine realities (not least, mobile devices) have also changed significantly than when BPEL appeared. Team members at Flowable were building BPM engines before BPMN happened, so have a strong understanding of previous approaches, and have been at the forefront of pushing BPM engines ever forward in evolving IT environments - so, I'm sure we'll be informed by the past, but very capable of determining an innovative yet stable path for the future.

Is there some sort of a beta channel through which the enthusiastic enterprise customers can already get a taste of these innovations?

We do work with some of our major customers on early access to releases for validation, but we haven't taken it to the step of a formal beta programme. We know from experience that to do a beta programme properly takes significant resource, so we're trying to be smart about how we expend our current effort. If you're a customer with the resources to work with us on an early access basis, then it's definitely worth asking us about possibilities.