Skip to content

fix: strip quotes and uppercase EXTRACT field name in deparser #289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pyramation wants to merge 4 commits into
base: main
Choose a base branch
from
Open

fix: strip quotes and uppercase EXTRACT field name in deparser #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8726ce6
fix: strip quotes and uppercase EXTRACT field name in deparser
pyramation Apr 3, 2026
8156070
test: add EXTRACT fixture with 18 keyword variants to kitchen-sink
pyramation Apr 3, 2026
700445e
docs: add AGENTS.md and fixtures testing skill
pyramation Apr 3, 2026
343fb51
docs: add code-generation skill, expand fixtures skill and AGENTS.md …
pyramation Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
109 changes: 109 additions & 0 deletions .agents/skills/code-generation/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Code Generation & Type Inference

This skill documents the code generation pipelines in pgsql-parser: protobuf-based TypeScript generation, type inference from SQL fixtures, and keyword list generation from PostgreSQL source.

## Overview

Several packages generate TypeScript code from external sources rather than being hand-written. These generated files should **not** be edited by hand — instead, re-run the generation scripts after changing inputs.

## 1. Protobuf-Based Code Generation (`build:proto`)

Four packages generate TypeScript from the PostgreSQL protobuf definition at `__fixtures__/proto/17-latest.proto`:

| Package | Script | What it generates |
|---------|--------|-------------------|
| `@pgsql/utils` | `npm run build:proto` | AST helper functions (`src/`), wrapped helpers (`wrapped.ts`), runtime schema (`runtime-schema.ts`) |
| `@pgsql/traverse` | `npm run build:proto` | Visitor-pattern traversal utilities |
| `@pgsql/transform` | `npm run build:proto` | Multi-version AST transformer utilities |
| `pg-ast` | `npm run build:proto` | Low-level AST type helpers |

Each package has a `scripts/pg-proto-parser.ts` that configures `PgProtoParser` with package-specific options (which features to enable, output paths, type sources).

**When to re-run:** After updating `__fixtures__/proto/17-latest.proto` (e.g., when upgrading to a new PostgreSQL version).

```bash
# Re-generate for a specific package
cd packages/utils && npm run build:proto

# Or build all (build:proto runs as part of build)
pnpm run build
```

Note: `build:proto` is called automatically as part of `npm run build` in these packages, so a full `pnpm run build` from root covers everything.

### Proto-Parser Test Utils

The `pg-proto-parser` package also has its own generation script:

```bash
cd packages/proto-parser && npm run generate:test-utils
```

This generates test utility functions from a `13-latest.proto` fixture into `test-utils/utils/`.

## 2. Type Inference & Narrowed Type Generation (`pgsql-types`)

The `pgsql-types` package has a two-step pipeline that discovers actual AST usage patterns from SQL fixtures and generates narrowed TypeScript types:

### Step 1: Infer field metadata

```bash
cd packages/pgsql-types && npm run infer
```

Runs `scripts/infer-field-metadata.ts`:
- Reads all `.sql` files from `__fixtures__/kitchen-sink/` and `__fixtures__/postgres/`
- Parses each statement and walks the AST
- For every `Node`-typed field, records which concrete node tags actually appear
- Writes `src/field-metadata.json` with nullable/tag/array info per field

### Step 2: Generate narrowed types

```bash
cd packages/pgsql-types && npm run generate
```

Runs `scripts/generate-types.ts`:
- Reads `src/field-metadata.json` (must run `infer` first)
- Generates `src/types.ts` with narrowed union types instead of generic `Node`
- Example: instead of `whereClause?: Node`, generates `whereClause?: { BoolExpr: BoolExpr } | { A_Expr: A_Expr } | ...`

**When to re-run:** After adding new SQL fixtures (which may introduce new node type combinations) or after updating the runtime schema.

Note: `infer` is called automatically as part of `npm run build` in pgsql-types.

## 3. Keyword List Generation (`@pgsql/quotes`)

```bash
cd packages/quotes && npm run keywords -- /path/to/postgres/src/include/parser/kwlist.h
```

Runs `scripts/keywords.ts`:
- Reads PostgreSQL's `kwlist.h` header file (from a local PostgreSQL source checkout)
- Parses `PG_KEYWORD(...)` macros to extract keywords and their categories
- Generates `src/kwlist.ts` with typed keyword sets (RESERVED, UNRESERVED, COL_NAME, TYPE_FUNC_NAME)

**When to re-run:** When upgrading to a new PostgreSQL version that adds/removes/reclassifies keywords.

**Requires:** A local checkout of the PostgreSQL source code to provide the `kwlist.h` file. The script will prompt for the path interactively if not provided as an argument.

## 4. Version-Specific Deparser Generation (`pgsql-deparser`)

```bash
cd packages/deparser && ts-node scripts/generate-version-deparsers.ts
```

Generates `versions/{13,14,15,16}/src/index.ts` files that wire up version-specific AST transformers (e.g., `PG13ToPG17Transformer`) to the main v17 deparser. This allows deparsing ASTs from older PostgreSQL versions.

**When to re-run:** When adding support for a new PostgreSQL version or changing the transformer class names.

## Quick Reference

| Workflow | Command | Input | Output |
|----------|---------|-------|--------|
| Proto codegen (all) | `pnpm run build` | `__fixtures__/proto/17-latest.proto` | Generated TS in each package's `src/` |
| Proto codegen (one pkg) | `cd packages/<pkg> && npm run build:proto` | Same | Same |
| Type inference | `cd packages/pgsql-types && npm run infer` | `__fixtures__/kitchen-sink/**/*.sql` | `src/field-metadata.json` |
| Type generation | `cd packages/pgsql-types && npm run generate` | `src/field-metadata.json` | `src/types.ts` |
| Keyword generation | `cd packages/quotes && npm run keywords -- <kwlist.h>` | PostgreSQL `kwlist.h` | `src/kwlist.ts` |
| Version deparsers | `cd packages/deparser && ts-node scripts/generate-version-deparsers.ts` | Transformer configs | `versions/*/src/index.ts` |
204 changes: 204 additions & 0 deletions .agents/skills/testing-fixtures/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,204 @@
# pgsql-parser Fixtures & Testing System

This skill documents how the fixture-based testing pipeline works in pgsql-parser and how to add new test fixtures.

## Architecture Overview

The testing pipeline has three layers:

1. **SQL fixture files** — Raw SQL statements in `__fixtures__/kitchen-sink/`
2. **Generated fixture JSON** — `__fixtures__/generated/generated.json` (individual statements keyed by path)
3. **Generated test files** — `packages/deparser/__tests__/kitchen-sink/*.test.ts`

## Directory Structure

```
__fixtures__/
kitchen-sink/ # Source SQL fixture files
latest/postgres/ # PostgreSQL regression test extracts
misc/ # Miscellaneous test cases (issues, features)
original/ # Original upstream PostgreSQL test files
pretty/ # Pretty-print specific test cases
generated/
generated.json # Auto-generated: individual statements keyed by relative path
plpgsql/ # PL/pgSQL fixture SQL files
plpgsql-generated/
generated.json # Auto-generated: PL/pgSQL fixtures

packages/deparser/
__tests__/
kitchen-sink/ # Auto-generated test files from kitchen-sink fixtures
misc/ # Hand-written test files for specific features
pretty/ # Pretty-print tests using PrettyTest utility
scripts/
make-fixtures.ts # Splits SQL files -> generated.json
make-kitchen-sink.ts # Generates kitchen-sink test files
statement-splitter.ts # Extracts individual statements from multi-statement SQL
test-utils/
index.ts # Core test utilities (expectParseDeparse, FixtureTestUtils, TestUtils)
PrettyTest.ts # Pretty-print test helper
packages/plpgsql-deparser/
scripts/
make-fixtures.ts # Extracts PL/pgSQL statements -> plpgsql-generated/generated.json
packages/transform/
scripts/
make-kitchen-sink.ts # Generates transform kitchen-sink tests
test-ast.ts # AST round-trip validation for transform
```

## How Fixtures Work

### Step 1: Write SQL fixture file

Create a `.sql` file in `__fixtures__/kitchen-sink/`. Organize by category:
- `misc/` for bug fixes, feature-specific tests, issue reproductions
- `pretty/` for pretty-print formatting tests
- `latest/postgres/` for PostgreSQL regression test extracts
- `original/` for original upstream PostgreSQL test files

Each SQL statement in the file becomes a separate test case. Use comments for context:

```sql
-- Brief description of what's being tested
-- Ref: constructive-io/pgsql-parser#289
SELECT EXTRACT(EPOCH FROM now());
SELECT EXTRACT(YEAR FROM TIMESTAMP '2001-02-16 20:38:40');
```

### Step 2: Regenerate fixtures

From `packages/deparser/`:

```bash
# Generate generated.json from all kitchen-sink SQL files
npm run fixtures

# Generate kitchen-sink test files from generated.json
npm run fixtures:kitchen-sink

# Or do both at once:
npm run kitchen-sink
```

**What `make-fixtures.ts` does:**
- Reads all `__fixtures__/kitchen-sink/**/*.sql` files
- Splits each file into individual statements using `statement-splitter.ts`
- Validates each statement parses correctly via `libpg-query`
- Writes all statements to `__fixtures__/generated/generated.json` as `{"relative/path-N.sql": "SQL statement"}`

**What `make-kitchen-sink.ts` does:**
- Reads all `__fixtures__/kitchen-sink/**/*.sql` files
- For each file, generates a test file in `packages/deparser/__tests__/kitchen-sink/`
- Test file name: `{category}-{name}.test.ts` (slashes become hyphens)
- Each test file uses `FixtureTestUtils.runFixtureTests()` with the list of statement keys

### Step 3: Run tests

```bash
# Run all deparser tests
cd packages/deparser && npx jest

# Run only kitchen-sink tests
npx jest __tests__/kitchen-sink/

# Run a specific fixture test
npx jest __tests__/kitchen-sink/misc-extract.test.ts
```

## Test Verification

The `FixtureTestUtils.runFixtureTests()` method (via `TestUtils.expectAstMatch()`) performs this verification for each statement:

1. **Parse** the original SQL -> AST
2. **Deparse** (non-pretty) the AST -> SQL string
3. **Reparse** the deparsed SQL -> new AST
4. **Compare** the cleaned ASTs — they must match
5. **Repeat** steps 2-4 with `pretty: true`

This ensures full round-trip fidelity: `parse(sql) -> deparse(ast) -> parse(deparsed) === original AST`

The `expectParseDeparse()` utility in `test-utils/index.ts` does the same thing for hand-written tests.

## Pretty-Print Tests

Pretty-print tests use a different pattern via `PrettyTest` class in `test-utils/PrettyTest.ts`:

```typescript
import { PrettyTest } from '../../test-utils/PrettyTest';

const testCases = [
'pretty/misc-1.sql',
'pretty/misc-2.sql',
];

const prettyTest = new PrettyTest(testCases);
prettyTest.generateTests();
```

This generates two tests per case (pretty and non-pretty) and uses Jest snapshots. Pretty tests read from `generated.json` and compare output via `toMatchSnapshot()`.

## Adding a New Fixture (Quick Reference)

1. Create/edit a `.sql` file in `__fixtures__/kitchen-sink/{category}/`
2. Run `cd packages/deparser && npm run kitchen-sink`
3. Run `npx jest` to verify all tests pass
4. Commit the `.sql` file, `generated.json`, and any new generated test files in `__tests__/kitchen-sink/`

## PL/pgSQL Fixtures (`plpgsql-deparser`)

The PL/pgSQL deparser has its own fixture pipeline:

```bash
cd packages/plpgsql-deparser && npm run fixtures
```

This runs `scripts/make-fixtures.ts` which:
- Reads SQL files from `__fixtures__/plpgsql/`
- Parses with `libpg-query`, filters to `CreateFunctionStmt` (language plpgsql) and `DoStmt`
- Validates each statement parses as PL/pgSQL via `parsePlPgSQLSync()`
- Writes to `__fixtures__/plpgsql-generated/generated.json`

## Transform Kitchen-Sink (`transform`)

The transform package has its own kitchen-sink and AST test scripts:

```bash
cd packages/transform
npm run kitchen-sink # generate transform-specific kitchen-sink tests
npm run test:ast # run AST round-trip validation
```

## Package Scripts Reference

### `packages/deparser` (primary fixture pipeline)

| Script | Command | Description |
|--------|---------|-------------|
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Regenerate `generated.json` |
| `fixtures:kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Regenerate kitchen-sink test files |
| `kitchen-sink` | `npm run fixtures && npm run fixtures:kitchen-sink` | Both steps combined |
| `fixtures:ast` | `ts-node scripts/make-fixtures-ast.ts` | Generate AST JSON fixtures |
| `fixtures:sql` | `ts-node scripts/make-fixtures-sql.ts` | Generate SQL fixtures via native deparse |
| `fixtures:upstream-diff` | `ts-node scripts/make-upstream-diff.ts` | Generate diff comparing upstream (libpg-query) vs our deparser output |
| `test` | `jest` | Run all tests |
| `test:watch` | `jest --watch` | Run tests in watch mode |

### `packages/plpgsql-deparser`

| Script | Command | Description |
|--------|---------|-------------|
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Extract PL/pgSQL fixtures to `plpgsql-generated/generated.json` |

### `packages/transform`

| Script | Command | Description |
|--------|---------|-------------|
| `kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Generate transform kitchen-sink tests |
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |

### `packages/parser`

| Script | Command | Description |
|--------|---------|-------------|
| `prepare-versions` | `ts-node scripts/prepare-versions.ts` | Generate version-specific sub-packages from `config/versions.json` |
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |
Loading
Loading