Building a Powerful Workflow Engine with NestJS

Introduction

In this article, we’ll explore the architecture of a powerful and scalable workflow engine built with NestJS. This engine is designed to handle complex workflows, manage versions, and provide a secure and easy-to-use API. We’ll take a deep dive into the core modules, the database schema, and the API design.

Core Technologies

This workflow engine is built on a modern and robust technology stack:

• NestJS: A progressive Node.js framework for building efficient, reliable and scalable server-side applications.
• Prisma: A next-generation ORM for Node.js and TypeScript. It helps developers build faster and make fewer errors.
• MongoDB: A general purpose, document-based, distributed database built for modern application developers and for the cloud era.
• TypeScript: A typed superset of JavaScript that compiles to plain JavaScript.

Architecture Overview

The workflow engine is designed with a modular architecture, making it easy to extend and maintain. The core of the application is the Workflows module, which is responsible for managing the lifecycle of a workflow.

graph TB
    subgraph "Client Layer"
        UI[Web Interface]
        API_Client[API Client]
    end
    
    subgraph "Application Layer"
        WorkflowModule[Workflows Module]
        NodesModule[Nodes Module]
        EdgesModule[Edges Module]
        SecretsModule[Secrets Module]
    end
    
    subgraph "Data Layer"
        PostgreSQL[(PostgreSQL)]
        MongoDB[(MongoDB)]
    end
    
    UI --> API_Client
    API_Client -->|REST API| WorkflowModule
    WorkflowModule --> NodesModule
    WorkflowModule --> EdgesModule
    WorkflowModule --> SecretsModule
    
    NodesModule --> PostgreSQL
    EdgesModule --> PostgreSQL
    SecretsModule --> PostgreSQL
    WorkflowModule -->|Versions| MongoDB

Modules

The application is divided into several modules, each with a specific responsibility:

• Workflows: Manages the creation, execution, and versioning of workflows.
• Nodes: Represents the individual steps in a workflow.
• Edges: Defines the connections between nodes.
• Secrets: Manages sensitive data used in workflows.
• API: Exposes a RESTful API for interacting with the workflow engine.

The Workflow Lifecycle

The workflow engine manages the entire lifecycle of a workflow, from creation to execution.

1. Creation: A new workflow is created with a set of nodes and edges.
2. Versioning: The workflow can be versioned, allowing for changes to be made without affecting the production version.
3. Publishing: A specific version of the workflow can be published, making it available for execution.
4. Execution: The workflow is executed, with each node being processed in the defined order.

stateDiagram-v2
    [*] --> Draft: Create Workflow
    Draft --> Draft: Edit Nodes/Edges
    Draft --> Versioned: Save Version
    Versioned --> Published: Publish Version
    Published --> Executing: Trigger Execution
    Executing --> Completed: Success
    Executing --> Failed: Error
    Completed --> [*]
    Failed --> [*]
    Published --> Versioned: Create New Version
    Versioned --> Draft: Continue Editing

Database Schema

The database schema is designed to be flexible and scalable. It uses a combination of relational and document-based data models.

• Workflows, Nodes, and Edges: These are stored in a relational database using Prisma.
• Workflow Versions: These are stored in a MongoDB database as snapshots of the workflow at a specific point in time.

erDiagram
    WORKFLOWS ||--o{ NODES : contains
    WORKFLOWS ||--o{ EDGES : contains
    WORKFLOWS ||--o{ VERSIONS : has
    WORKFLOWS ||--o{ SECRETS : uses
    
    WORKFLOWS {
        uuid id PK
        string name
        string description
        timestamp created_at
        timestamp updated_at
    }
    
    NODES {
        uuid id PK
        uuid workflow_id FK
        string type
        json config
        int position_x
        int position_y
    }
    
    EDGES {
        uuid id PK
        uuid workflow_id FK
        uuid source_node_id FK
        uuid target_node_id FK
    }
    
    VERSIONS {
        uuid id PK
        uuid workflow_id FK
        int version_number
        json snapshot
        boolean published
        timestamp created_at
    }
    
    SECRETS {
        uuid id PK
        uuid workflow_id FK
        string key
        string encrypted_value
    }

API Design

The API is designed to be RESTful and easy to use. It provides endpoints for managing workflows, nodes, edges, and secrets. The API is secured using an API key, and it uses Swagger for documentation.

Conclusion

This workflow engine is a powerful and scalable solution for managing complex workflows. Its modular architecture, robust technology stack, and well-designed API make it a great choice for any application that needs to automate business processes.