Application of workflow engine in vivo marketing automation

Author: Cheng Wangrong, vivo Internet Server Team

This article analyzes the background of introducing workflow technology into marketing automation business and the introduction of workflow engines. It also introduces the characteristics of several popular open source workflow engines in the industry, as well as the design ideas and summary thoughts during the project's self-developed development process.

1. Business Background

The marketing automation platform can support the configuration of activity journey strategies for different user life cycles, and carry out differentiated marketing reach plans based on different activity behaviors triggered by users. At the same time, there are different business processing procedures (such as approval procedures and business flows) in the specific execution of various types of activities. The business processes are complex and diverse, and the requirements change frequently. There will be the following pain points during project development:

1. Long project delivery cycle: A complete business process needs to be iterated from scratch in versions, which takes a long time to develop and is costly.
2. Repeated functional development and testing: There are many common processes between businesses, which leads to a large amount of repetitive development and testing work and low efficiency.
3. High maintenance cost: As the project business gradually develops, business processes gradually accumulate, maintainability decreases, and system changes have a ripple effect on the entire system.

How to separate business logic from control flow and let production and research personnel focus more on business realization is a key issue that needs to be solved. The traditional OA field uses the time-tested business process management solution - Workflow. Workflow is an industrial-grade solution, and a series of standards have been formulated by the Workflow Management Consortium (WfMC).

2. Workflow Introduction

2.1 Workflow Definition

Workflow - abstracts the business rules between the workflow and its various operation steps, models the work organization logic and rules in the process, and lets the computer perform automatic processing.

The essential idea of ​​workflow is to instantiate real activities through predefined workflow templates. In simple terms, it is to configure a process template (such as a running process template for sharing activities) through a preset format or visualization, and then construct a process instance object through the template when using it, and complete the activity operation tracking and backtracking through the instance object.

2.2 Workflow Reference Model

The WfMC Workflow Management Alliance has developed a reference model for workflows, the core of which is the workflow engine in the middle. The workflow engine provides process definition tools (interface 1), provides users with information query (interface 2), calls external applications (interface 3), integrates other workflows (interface 4) and monitors and manages (interface 5). For most workflow products, the focus is on the implementation of interface 1 and interface 2.

2.3 Key Features of Workflow Engine

1. Process visualization provides visual process building, process view viewing capabilities, and real-time observation of task execution capabilities.
2. Business orchestration and reuse can componentize public businesses, support the free orchestration of tasks, and freely build different processes suitable for the business.
3. The separation of business and control entrusts the task of process control (such as flow, judgment, loop, retry, etc.) to the workflow, allowing users to focus on the core business logic.

2.4 Types of Workflow Engines

There is no specific standard for the types of workflows. According to the characteristics of the process task nodes, they can be divided into:

• Sequential workflow The operation mode of sequential workflow is similar to a specific flowchart. After the previous process task is completed, it will enter the next process task in sequence, and the process is irreversible.

• State machine workflow focuses on the state of process tasks. The factors that drive the task state to change are generally external events, that is, event-driven methods, which drive the task node to run from one state to another, and the nodes are reversible.

• Rule-driven workflow focuses on the operation rules of nodes and executes workflows based on business rules. Rule-driven workflow is very useful when dealing with various projects with clear goals but different levels of "rules" or specifications.

It can be seen that different types of workflows are not completely separated. The state machine workflow can also combine conditions and rules to perform the process of operating node conversion. In software development, it is generally considered to combine state machines and rule-driven workflows.

2.5 Differences between Workflow Engines and State Machines

In the previous article, we made a simple comparison between state machines and workflow engines. In fact, the two are not completely equivalent concepts:

1. A state machine is a mathematical model of system states and the transitions and actions between these states, while a workflow is an abstract modeling of the business logic and rules between the overall workflow and its various operation steps.
2. The state machine mode is event-driven, and most of the automatic flow of states is triggered by external events; the workflow engine focuses more on describing the automatic flow after the completion of predefined process tasks, and is more predictable.
3. From the perspective of the complexity of applicable scenarios, the direct use of the state machine can clearly depict all possible states and events that lead to transitions. It is suitable for solving single-dimensional, low-complexity business problems and takes advantage of its flexibility and portability. The workflow engine is more suitable for complex business process management and can solve more complex process automation problems such as large-scale CRM, focusing on improving the efficiency of the overall business process.
4. The workflow engine can be built based on the structural model of the state machine. In fact, many open source workflow engines are also based on the implementation of the state machine.

After understanding the basic characteristics and usage scenarios of workflow, let's take a look at the more popular open source workflow engines.

3. Open Source Workflow Engine

​

4. Self-developed design of workflow engine

4.1 Problems with using open source workflow engines

1. The biggest advantage of open source workflow is that it can be used out of the box with the help of open source resources and has comprehensive functions, but it also brings with it the problems of configuration and maintenance of a large number of tables. Taking Activiti as an example, using Activiti 7.0 requires the introduction of at least 20 tables. Although it seems to be a non-invasive method, there is a certain cost in the process of system evolution and maintenance. Especially when there are many business process instances, developers need to have a deeper control over the table logic.
2. Due to the objective uniqueness of the business, as a business process component, it is generally necessary to carry out secondary development and adaptation according to its own business. For example, it is necessary to control the user role permissions of the process node according to its own organizational structure; plug-in its own business capabilities, add them to the workflow configuration, intercept callbacks, etc.

4.2 Core Design Ideas of Self-developed Engine

4.2.1 Engine core module

Returning to the essence of workflow, workflow is the process of instantiating real activities through predefined process templates. A basic workflow engine mainly consists of three core parts:

• Process template creation creates process templates based on business rules and logic, and sets the operation and change path of each node. Based on template creation, process designers, plug-in nodes, diversified template file formats, template persistence, etc. can be extended.

• Process instance publishing creates a process instance based on the process template. The relationship between the process template and the process instance is similar to the relationship between a class and an object. For example, the work order system administrator defines an approval flow template (process template), and the user clicks to create a work order (process instance). Based on process instance publishing, real-time instance observation, node transition record backtracking, instance status persistence, failure retry, transaction control, etc. can be extended.

• After the task flow execution creates the process instance, the process instance only needs to independently execute the tasks of its own instance according to the definition of the process template. Different instances do not affect each other and complete their own life cycle.

4.2.2 Engine Core Design

① When the application container starts, the process engine environment configuration is loaded, including the parser structure, process engine context, process definition file path, etc.

② Read the specified process definition file, parse the process nodes, build the execution context, and put the process nodes into the memory cache.

③ The business side creates a process, starts a new process instance, and binds the business process and the process instance.

④ Run each node of the process instance and save each process node persistently.

4.3 Specific Practice

① Engine core services.

The main external interface for engine operations includes starting process instances and obtaining services for related process definition templates, process instances, and process nodes.

 public interface FlowEngine {
 /**
 * Start the process instance according to the process definition key and parameter list
 *
 */
 FlowInstance startInstance(String processDefKey, Map<String, Object> args);
 /**
 * Execute process tasks based on process definition primary key ID and parameter list (promote automatic flow of processes)
 * Unified transaction control
 */
 void execInstance(Long instanceId, Map<String, Object> args) throws FlowAuthorityException;
 /**
 * Get the process definition process service
 *
 */
 ProcessService process();
 /**
 * Get process instance service
 *
 */
 InstanceService instance();
 /**
 * Get task node service
 *
 */
 TaskService task();
 }

② Process definition service.

It is mainly aimed at the creation and publishing of process definition templates, and can support different creation methods according to the specific implementation class.

 public interface ProcessService {
 /**
 * Create a process definition template
 *
 */
 void create(String definition);
 /**
 * Release process definition template
 *
 */
 void deploy(String fileName);
 /**
 * Get the process definition corresponding to the process key
 */
 FlowProcess getProcessByDefKey(String processDefKey);
 }

③ Process instance service.

Provides an entry point for process instance creation, persistence, and process instance execution.

 public interface InstanceService {
 /**
 * Create a process instance
 *
 */
 FlowInstance createInstance(FlowProcess process, Map<String, Object> args);
 /**
 * Execute process instance
 *
 * @param instanceId process instance id
 */
 void exec(Long instanceId);
 /**
 * Get the process instance by id
 *
 * @param instanceId
 * @return
 */
 FlowInstance getById(Long instanceId);
 }

④ Process task node service.

Provides creation and query of each specific task of the process node.

 public interface TaskService {
 /**
 * Create new tasks based on task models and execution objects
 *
 */
 FlowTask createTask(TaskModel taskModel, Execution execution);
 /**
 * Complete the task
 *
 */
 FlowTask complete(Long taskId, Map<String, Object> args);
 /**
 * Get the tasks in progress in the process instance
 *
 */
 FlowTask getActiveTask(Long instanceId);
 /**
 * Get the last completed task of the process instance
 *
 */
 FlowHistTask getLastDoneTask(Long instanceId);
 } 

The core method is

• FlowEngine#startInstanceByKey, starts a process instance. Creates a process instance object based on the process definition.
• FlowEngine#execInstance executes the process instance task, and promotes the automatic flow of the process instance according to the node tasks defined in the process through the passed-in context parameters (operator, operation variables, etc.).

4.4 Thinking and Extension

1. Process definition parsing performance. Since the current design is to load and parse all process definition files when the application starts, too many process definition files will affect the application startup speed. This can be optimized through multi-threaded parsing and lazy loading (parsing when used).
2. Process definition version compatibility. Since business processes are not static and will be continuously iterated during project development, compatibility with previous different processes is required.
3. Process node plug-in and orchestration capabilities. Extract basic services for common use to support the plug-in and orchestration capabilities of drawing different processes.
4. Process execution monitoring capability: The execution status of process task nodes is reported, and the system automatically monitors and issues alarms.

V. Conclusion

This article analyzes the background of introducing a workflow engine, driving the business logic out of the control flow, allowing the R&D team to focus more on the business and solving the problem of low R&D efficiency.

The essential idea of ​​workflow is to instantiate real activities through predefined workflow templates. Generally, it is necessary to have the basic capabilities of process visualization, business orchestration and reuse, and separation of business and control. Common workflows are divided into sequential workflows, state machine workflows, and rule-driven workflows. The most common open source workflow framework is the state machine workflow, which uses event-driven methods to drive the process operation.

At the same time, we briefly introduced the characteristics of several popular open source workflow engines in the industry. Combining the characteristics of open source workflow engines, and targeting diversified and frequently iterated business processes, we took the essential idea of ​​workflow as the starting point and developed a lightweight workflow engine. We also shared the design ideas and summary thoughts in the practice process.