Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
131 changes: 131 additions & 0 deletions js/plugins/middleware/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
# Genkit Middleware

This package provides a collection of useful middlewares for the Genkit JS SDK to enhance model execution, tool usage, and agentic workflows.

## Installation

```bash
npm install @genkit-ai/middleware
# or
pnpm add @genkit-ai/middleware
```

## Available Middlewares

### 1. FileSystem Middleware (`filesystem`)

Grants the model access to the local filesystem by injecting standard file manipulation tools (`list_files`, `read_file`, `write_file`, `search_and_replace`). All operations are safely restricted to a specified root directory.

```typescript
import { genkit } from 'genkit';
import { filesystem } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
model: 'gemini-2.5-flash',
prompt: 'Create a hello world node app in the workspace',
use: [
filesystem({ rootDirectory: './workspace' })
]
});
```

### 2. Skills Middleware (`skills`)

Automatically scans a directory for `SKILL.md` files (and their YAML frontmatter) and injects them into the system prompt. It also provides a `use_skill` tool the model can use to retrieve more specific skills on demand.

```typescript
import { genkit } from 'genkit';
import { skills } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
prompt: 'How do I run tests in this repo?',
use: [
skills({ skillPaths: ['./skills'] })
]
});
```

### 3. Tool Approval Middleware (`toolApproval`)

Restricts execution of tools to an approved list. If the model attempts to call an unapproved tool, it throws a `ToolInterruptError` allowing you to prompt the user for manual confirmation before resuming.

```typescript
import { genkit, restartTool } from 'genkit';
import { toolApproval } from '@genkit-ai/middleware';

const ai = genkit({ ... });

// 1. Initial attempt
const response = await ai.generate({
prompt: 'write a file',
tools: [writeFileTool],
use: [
toolApproval({ approved: [] }) // Empty list means call triggers interrupt
]
});

if (response.finishReason === 'interrupted') {
const interrupt = response.interrupts[0];

// 2. Ask user for approval, then recreate the tool request with approval
const approvedPart = restartTool(interrupt, { toolApproved: true });

// 3. Resume execution
const resumedResponse = await ai.generate({
messages: response.messages,
resume: { restart: [approvedPart] },
use: [
toolApproval({ approved: [] })
]
});
}
```

### 4. Retry Middleware (`retry`)

Automatically retries failed model generations on transient error codes (like `RESOURCE_EXHAUSTED`, `UNAVAILABLE`) using exponential backoff with jitter.

```typescript
import { genkit } from 'genkit';
import { retry } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
model: googleAI.model('gemini-pro-latest'),
prompt: 'Heavy reasoning task...',
use: [
retry({
maxRetries: 3,
initialDelayMs: 1000,
backoffFactor: 2
})
]
});
```

### 5. Fallback Middleware (`fallback`)

Automatically switches to a different model if the primary model fails on a specific set of error codes. Useful for falling back to a smaller/faster model when a large model exceeds quota limits.

```typescript
import { genkit } from 'genkit';
import { fallback } from '@genkit-ai/middleware';

const ai = genkit({ ... });

const response = await ai.generate({
model: googleAI.model('gemini-pro-latest'),
prompt: 'Try the pro model first...',
use: [
fallback({
models: [googleAI.model('gemini-flash-latest')], // try flash if pro fails
statuses: ['RESOURCE_EXHAUSTED']
})
]
});
```
146 changes: 146 additions & 0 deletions js/plugins/middleware/examples/coding_agent.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,146 @@
/**
* Copyright 2026 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

import { googleAI } from '@genkit-ai/google-genai';
import * as fs from 'fs';
import {
genkit,
restartTool,
type GenerateResponse,
type MessageData,
type ToolRequestPart,
} from 'genkit';
import * as path from 'path';
import * as readline from 'readline';
import { filesystem, retry, skills, toolApproval } from '../src/index.js';

const rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
});

function askQuestion(query: string): Promise<string> {
return new Promise((resolve) => rl.question(query, resolve));
}

async function main() {
const ai = genkit({
plugins: [
googleAI(),
filesystem.plugin(),
skills.plugin(),
toolApproval.plugin(),
retry.plugin(),
],
});

const currentDir = process.cwd();
const fsRoot = path.join(currentDir, 'workspace');
const skillsRoot = path.join(currentDir, 'skills');

// Ensure workspace exists
if (!fs.existsSync(fsRoot)) {
fs.mkdirSync(fsRoot, { recursive: true });
}

console.log('--- Coding Agent ---');
console.log('Type your request. To exit, type "exit".');

let messages: MessageData[] = [
{
role: 'system',
content: [
{
text:
`You are a helpful coding agent. Very terse but thoughtful and careful.\n` +
`Your working directory is in ${fsRoot}, you are not allowed to access anything outside it.\n` +
`Use skills. ALWAYS start by analyzing the current state of the workspace, ` +
`there might be something already there.`,
},
],
},
];

while (true) {
const input = await askQuestion('\n> ');
if (input.trim().toLowerCase() === 'exit') {
break;
}

try {
let interruptRestart: ToolRequestPart[] | undefined;
let response: GenerateResponse;

while (true) {
response = await ai.generate({
model: 'googleai/gemini-flash-latest',
prompt: interruptRestart ? undefined : input,
messages: messages,
resume: interruptRestart ? { restart: interruptRestart } : undefined,
use: [
toolApproval({
approved: ['read_file', 'list_files', 'use_skill'],
}),
skills({ skillPaths: [skillsRoot] }),
filesystem({ rootDirectory: fsRoot }),
],
maxTurns: 20,
});

if (response.finishReason !== 'interrupted') {
break;
}

const interrupts = response.interrupts;
if (!interrupts || interrupts.length === 0) {
console.log('Interrupted but no interrupt record found.');
break;
}

const approvedInterrupts: ToolRequestPart[] = [];
for (const interrupt of interrupts) {
console.log('\n*** Tool Approval Required ***');
console.log(`Tool: ${interrupt.toolRequest.name}`);
console.log(`Input: ${JSON.stringify(interrupt.toolRequest.input)}`);

const approval = await askQuestion('Approve? (y/N): ');
if (approval.trim().toLowerCase() === 'y') {
approvedInterrupts.push(
restartTool(interrupt, { toolApproved: true })
);
}
}

if (approvedInterrupts.length > 0) {
console.log('Resuming...');
interruptRestart = approvedInterrupts;
messages = response.messages;
} else {
console.log('Tool denied.');
break;
}
}

console.log(`\nAI Response:\n${response.text}`);
messages = response.messages;
} catch (e) {
console.error('Error during generation:', e);
}
}
rl.close();
}

main();
2 changes: 1 addition & 1 deletion js/plugins/middleware/examples/fallback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { genkit } from 'genkit';
import { fallback } from '../src/index.js';

const ai = genkit({
plugins: [googleAI()],
plugins: [googleAI(), fallback.plugin()],
});

async function main() {
Expand Down
2 changes: 1 addition & 1 deletion js/plugins/middleware/examples/retry.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ import { genkit } from 'genkit';
import { retry } from '../src/index.js'; // @genkit-ai/middleware

const ai = genkit({
plugins: [googleAI()],
plugins: [googleAI(), retry.plugin()],
});

async function main() {
Expand Down
19 changes: 16 additions & 3 deletions js/plugins/middleware/src/fallback.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,17 +40,29 @@ export const FallbackOptionsSchema = z
/**
* An array of models to try in order.
*/
models: z.array(ModelReferenceSchema),
models: z
.array(ModelReferenceSchema)
.describe('An array of models to try in order.'),
/**
* An array of `StatusName` values that should trigger a fallback.
* @default ['UNAVAILABLE', 'DEADLINE_EXCEEDED', 'RESOURCE_EXHAUSTED', 'ABORTED', 'INTERNAL', 'NOT_FOUND', 'UNIMPLEMENTED']
*/
statuses: z.array(z.string()).optional(),
statuses: z
.array(z.string())
.optional()
.describe(
'An array of StatusName values that should trigger a fallback.'
),
/**
* If true, the fallback model will not inherit the original request's configuration.
* @default false
*/
isolateConfig: z.boolean().optional(),
isolateConfig: z
.boolean()
.optional()
.describe(
"If true, the fallback model will not inherit the original request's configuration."
),
})
.passthrough();

Expand All @@ -76,6 +88,7 @@ export const fallback: GenerateMiddleware<typeof FallbackOptionsSchema> =
generateMiddleware(
{
name: 'fallback',
description: 'Fallback to a different model on specific error statuses.',
configSchema: FallbackOptionsSchema,
},
(options) => {
Expand Down
Loading
Loading