Bulk create and publish benchmark flows by ravikiranvm · Pull Request #2025 · openops-cloud/openops

ravikiranvm · 2026-02-27T10:11:02Z

Additional Notes

There is another ticket to add the seeded flows to benchmark tables.

linear · 2026-02-27T10:11:05Z

OPS-3715 Import and publish benchmark workflows

Copilot

Pull request overview

This PR implements bulk creation and publishing of benchmark workflows from catalog templates. The changes add functionality to seed benchmark workflows from JSON files stored in a catalog directory, creating and publishing multiple flows atomically in a single database transaction.

Changes:

Added new benchmark-flow-bulk-create.ts module with bulk flow creation and publishing functionality
Enhanced createBenchmark to seed workflows from catalog using benchmarkConfiguration parameter
Updated seedBenchmarkWorkflowsFromCatalog to read workflow templates from files and bulk create flows
Added comprehensive test coverage for the new seeding functionality

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 11 comments.

File	Description
`packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts`	New file implementing bulk flow creation and publishing with database transaction support
`packages/server/api/src/app/benchmark/create-benchmark.service.ts`	Added `seedBenchmarkWorkflowsFromCatalog` function and updated `createBenchmark` to use benchmark configuration for workflow seeding
`packages/server/api/test/unit/benchmark/create-benchmark.service.test.ts`	Added test coverage for the new seeding functionality including empty paths scenario

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

packages/server/api/src/app/benchmark/create-benchmark.service.ts

Copilot · 2026-02-27T10:19:13Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

+    const content = await fs.readFile(filePath, 'utf-8');
+    const parsed = JSON.parse(content) as {
+      template: WorkflowTemplate['template'];
+    };


The fs.readFile and JSON.parse operations in the loop lack error handling. If a catalog file is corrupted, missing, or contains invalid JSON, the entire benchmark creation will fail without a clear error message.

Consider wrapping these operations in try-catch blocks to provide more specific error messages about which workflow file failed to load or parse. This will make debugging easier and provide better user feedback.

Suggested change

const content = await fs.readFile(filePath, 'utf-8');

const parsed = JSON.parse(content) as {

template: WorkflowTemplate['template'];

};

let content: string;

try {

content = await fs.readFile(filePath, 'utf-8');

} catch (err) {

const message =

err instanceof Error ? err.message : String(err);

throwValidationError(

`Failed to read benchmark catalog file at "${filePath}": ${message}`,

);

}

let parsed: { template: WorkflowTemplate['template'] };

try {

parsed = JSON.parse(content) as {

template: WorkflowTemplate['template'];

};

} catch (err) {

const message =

err instanceof Error ? err.message : String(err);

throwValidationError(

`Failed to parse JSON in benchmark catalog file at "${filePath}": ${message}`,

);

}

packages/server/api/test/unit/benchmark/create-benchmark.service.test.ts

packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts

packages/server/api/src/app/benchmark/create-benchmark.service.ts

packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts

Copilot · 2026-02-27T10:19:15Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

+  return results.map((r, index) => ({
+    flowId: r.id,
+    displayName: r.version.displayName,
+    isOrchestrator: index === 0,


The isOrchestrator flag is determined solely by the index position (index === 0) in the results array. This assumes that the first workflow in the paths array will always be the orchestrator.

This logic depends on the order returned by resolveWorkflowPathsForSeed. Looking at catalog-resolver.ts line 63, the order is: [orchestratorId, cleanupId, ...subWorkflowIds]. This means:

index 0 = orchestrator (correct ✓)

index 1 = cleanup (marked as isOrchestrator: false, which is correct)

index 2+ = sub-workflows (marked as isOrchestrator: false, which is correct)

However, this implicit dependency on array ordering is fragile. Consider adding a comment explaining this assumption, or better yet, explicitly mark which workflow is the orchestrator based on the workflow ID rather than position.

Suggested change

return results.map((r, index) => ({

flowId: r.id,

displayName: r.version.displayName,

isOrchestrator: index === 0,

// NOTE: The isOrchestrator flag relies on the ordering contract of

// resolveWorkflowPathsForSeed, which returns paths in the form:

// [orchestratorId, cleanupId, ...subWorkflowIds]

// This means:

// - index 0 = orchestrator

// - index 1 = cleanup

// - index 2+ = sub-workflows

// If the ordering in resolveWorkflowPathsForSeed changes, this constant

// (and the associated logic) must be updated accordingly.

const orchestratorIndex = 0;

return results.map((r, index) => ({

flowId: r.id,

displayName: r.version.displayName,

isOrchestrator: index === orchestratorIndex,

packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts

sonarqubecloud · 2026-02-27T15:06:57Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

ravikiranvm · 2026-02-27T16:11:40Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

+import { fetchConnectionsWithSupportedBlocks } from './connections-with-supported-blocks';
 import { throwValidationError } from './errors';

+function validateBenchmarkConfiguration(config: BenchmarkConfiguration): void {


We might move this validation later to the controller; keeping it here for now.

ravikiranvm · 2026-02-27T16:12:32Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

    provider,
-    workflows: [],
+    workflows,
    webhookPayload: {


Will build the payload in later PRs

MarceloRGonc · 2026-02-27T16:35:35Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

+  const parsedTemplates = paths.map(async ({ filePath }) => {
+    const content = await fs.readFile(filePath, 'utf-8');
+    const parsed = JSON.parse(content) as {
+      template: WorkflowTemplate['template'];
+    };
+    return { template: parsed.template };
+  });
+  const templates = await Promise.all(parsedTemplates);


Extract to a method:

async function loadWorkflowTemplates(provider: string, workflowIds: string[]): Promise<WorkflowTemplate[]> { const paths = resolveWorkflowPathsForSeed(provider, workflowIds); if (paths.length === 0) { return []; } const parsedTemplates = paths.map(async ({ filePath }) => { const content = await fs.readFile(filePath, 'utf-8'); const parsed = JSON.parse(content) as { template: WorkflowTemplate['template']; }; return { template: parsed.template }; }); return await Promise.all(parsedTemplates); }

MarceloRGonc · 2026-02-27T16:35:51Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

    userId,
  });

+  const paths = resolveWorkflowPathsForSeed(provider, workflowIds);


Suggested change

const paths = resolveWorkflowPathsForSeed(provider, workflowIds);

const templates = await loadWorkflowTemplates(provider, workflowIds);

MarceloRGonc · 2026-02-27T16:35:59Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts


+  const paths = resolveWorkflowPathsForSeed(provider, workflowIds);
+  const workflows = await seedBenchmarkWorkflowsFromCatalog({
+    paths,


Suggested change

paths,

templates,

MarceloRGonc · 2026-02-27T16:36:13Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

  );
 }

+export async function seedBenchmarkWorkflowsFromCatalog(params: {


Suggested change

export async function seedBenchmarkWorkflowsFromCatalog(params: {

export async function createBenchmarkWorkflows(params: {

MarceloRGonc · 2026-02-27T16:36:23Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

 }

+export async function seedBenchmarkWorkflowsFromCatalog(params: {
+  paths: ResolvedWorkflowPath[];


Suggested change

paths: ResolvedWorkflowPath[];

templates: WorkflowTemplate[];

MarceloRGonc · 2026-02-27T16:37:44Z

packages/server/api/src/app/benchmark/connections-with-supported-blocks.ts

+import { appConnectionService } from '../app-connection/app-connection-service/app-connection-service';
+import { getProviderMetadataForAllBlocks } from '../app-connection/connection-providers-resolver';
+
+export async function fetchConnectionsWithSupportedBlocks(


Suggested change

export async function fetchConnectionsWithSupportedBlocks(

export async function getConnectionsWithBlockSupport(

MarceloRGonc · 2026-02-27T16:37:59Z

packages/server/api/src/app/benchmark/create-benchmark.service.ts

+  });
+  const templates = await Promise.all(parsedTemplates);
+
+  const connections = await fetchConnectionsWithSupportedBlocks(projectId, [


Suggested change

const connections = await fetchConnectionsWithSupportedBlocks(projectId, [

const connections = await getConnectionsWithBlockSupport(projectId, [

Bulk create and publish benchmark flows

bf30146

ravikiranvm marked this pull request as ready for review February 27, 2026 10:12

Copilot AI review requested due to automatic review settings February 27, 2026 10:12

Copilot started reviewing on behalf of ravikiranvm February 27, 2026 10:13 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

Fix copilot comments about file reading and guard

7d7a8f3

MarceloRGonc reviewed Feb 27, 2026

View reviewed changes

packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts Outdated Show resolved Hide resolved

packages/server/api/src/app/benchmark/benchmark-flow-bulk-create.ts Outdated Show resolved Hide resolved

ravikiranvm added 4 commits February 27, 2026 17:03

Revert accidental changes

d7636da

Fix helper methods

1a8ae15

Remove unnecessary usage of datetime

c85d745

Extract utility method

180034e

ravikiranvm commented Feb 27, 2026

View reviewed changes

MarceloRGonc reviewed Feb 27, 2026

View reviewed changes

-    const content = await fs.readFile(filePath, 'utf-8');
-    const parsed = JSON.parse(content) as {
-      template: WorkflowTemplate['template'];
-    };
+    let content: string;
+    try {
+      content = await fs.readFile(filePath, 'utf-8');
+    } catch (err) {
+      const message =
+        err instanceof Error ? err.message : String(err);
+      throwValidationError(
+        `Failed to read benchmark catalog file at "${filePath}": ${message}`,
+      );
+    }
+    let parsed: { template: WorkflowTemplate['template'] };
+    try {
+      parsed = JSON.parse(content) as {
+        template: WorkflowTemplate['template'];
+      };
+    } catch (err) {
+      const message =
+        err instanceof Error ? err.message : String(err);
+      throwValidationError(
+        `Failed to parse JSON in benchmark catalog file at "${filePath}": ${message}`,
+      );
+    }

-  return results.map((r, index) => ({
-    flowId: r.id,
-    displayName: r.version.displayName,
-    isOrchestrator: index === 0,
+  // NOTE: The isOrchestrator flag relies on the ordering contract of
+  // resolveWorkflowPathsForSeed, which returns paths in the form:
+  //   [orchestratorId, cleanupId, ...subWorkflowIds]
+  // This means:
+  //   - index 0 = orchestrator
+  //   - index 1 = cleanup
+  //   - index 2+ = sub-workflows
+  // If the ordering in resolveWorkflowPathsForSeed changes, this constant
+  // (and the associated logic) must be updated accordingly.
+  const orchestratorIndex = 0;
+  return results.map((r, index) => ({
+    flowId: r.id,
+    displayName: r.version.displayName,
+    isOrchestrator: index === orchestratorIndex,

	const paths = resolveWorkflowPathsForSeed(provider, workflowIds);
	const templates = await loadWorkflowTemplates(provider, workflowIds);

	export async function seedBenchmarkWorkflowsFromCatalog(params: {
	export async function createBenchmarkWorkflows(params: {

	paths: ResolvedWorkflowPath[];
	templates: WorkflowTemplate[];

	export async function fetchConnectionsWithSupportedBlocks(
	export async function getConnectionsWithBlockSupport(

	const connections = await fetchConnectionsWithSupportedBlocks(projectId, [
	const connections = await getConnectionsWithBlockSupport(projectId, [

Conversation

ravikiranvm commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Additional Notes

Uh oh!

linear bot commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 27, 2026

Quality Gate passed

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ravikiranvm commented Feb 27, 2026 •

edited

Loading