Benchmark: polyglot-sql vs other Rust SQL parsers on PostgreSQL workloads

[LucaCappelletti94]

I have published an open-source benchmark comparing Rust SQL parsers on real-world PostgreSQL statements, including polyglot-sql. Sharing the results here as they may be useful to the team.

Benchmark repo: https://github.com/LucaCappelletti94/sql_ast_benchmark

What the benchmark measures

Performance: parse throughput on the Spider + Gretel PostgreSQL datasets (4,505 statements: SELECT, INSERT, UPDATE, DELETE), measured at batch sizes 1-1000.

Correctness: evaluated against the sqlparser-rs test suite using pg_query.rs (libpg_query - the actual PostgreSQL parser) as ground truth. Four metrics:

Metric	Definition	Direction
Recall	Of SQL pg_query accepts, how many does the parser also accept?	higher is better
False-positive rate	Of SQL pg_query rejects, how many does the parser wrongly accept?	lower is better
Round-trip stability	parse -> print -> re-parse -> re-print produces identical output?	higher is better
Fidelity	pg_query_canonical(parser_output) == pg_query_canonical(original) - semantically correct output, not just self-consistent?	higher is better

Performance results

polyglot-sql has a notable per-call overhead at low batch sizes but amortizes very well at scale:

Statements	sqlparser-rs	polyglot-sql	pg_query.rs
1 (SELECT)	6.6 us	29.8 us	11.0 us
10	124.9 us	120.6 us	198.5 us
100	800.6 us	715.5 us	1.44 ms
1000	10.09 ms	7.79 ms	17.67 ms

The crossover with sqlparser-rs occurs at around 10 statements. At 984 UPDATE statements, polyglot-sql is 2.5x faster (2.44 ms vs 6.23 ms). For a library only a few weeks old, this is a strong result.

Parse success rate on real-world corpus (Spider + Gretel, PostgreSQL-validated)

Statement type	Parse success rate
SELECT	100%
INSERT	100%
UPDATE	99.8%
DELETE	100%

Correctness results

Tested against the sqlparser-rs test suite on three corpora, using pg_query as PostgreSQL ground truth. Counts show absolute numbers; percentages are bolded.

PostgreSQL-specific tests (312 valid / 129 invalid)

Metric	polyglot-sql	sqlparser-rs (reference)
Recall	254/312 - 81%	310/312 - 99%
False-positive rate	79/129 - 61.2%	37/129 - 28.7%
Round-trip	247/254 - 97.2%	310/310 - 100%
Fidelity	200/254 - 78.7%	306/310 - 98.7%

Common (all-dialect) tests (323 valid / 469 invalid)

Metric	polyglot-sql	sqlparser-rs (reference)
Recall	286/323 - 89%	318/323 - 98%
False-positive rate	241/469 - 51.4%	141/469 - 30.1%
Round-trip	282/286 - 98.6%	318/318 - 100%
Fidelity	254/286 - 88.8%	318/318 - 100%

TPC-H / regression tests (21 valid / 1 invalid)

Metric	polyglot-sql	sqlparser-rs (reference)
Recall	21/21 - 100%	21/21 - 100%
False-positive rate	1/1 - 100%	1/1 - 100%
Round-trip	21/21 - 100%	21/21 - 100%
Fidelity	17/21 - 81.0%	21/21 - 100%

Key correctness observations

Strengths:

1. Excellent real-world coverage: 100% parse success on SELECT, INSERT, DELETE; 99.8% on UPDATE. The library handles production-grade PostgreSQL workloads well.

2. Strong TPC-H recall (100%): complex analytical queries with multi-join, aggregations, subqueries, and window functions all parse successfully.

3. Good round-trip stability (97-99%): the vast majority of accepted statements round-trip cleanly through parse -> print -> re-parse -> re-print.

Areas for improvement:

1. False-positive rate is the most significant correctness gap: polyglot-sql accepts 51-61% of SQL that PostgreSQL itself rejects as invalid. This is the highest false-positive rate of any parser in the benchmark. Silent acceptance of invalid queries is a problem for any use case that relies on the parser as a validation layer.

2. Fidelity gap: among statements the parser accepts, only 78-89% produce semantically equivalent output under pg_query's canonical form. The round-trip is stable (the parser is self-consistent), but the printer normalizes constructs in ways that change semantics. Several PostgreSQL constructs are emitted verbatim rather than translated: LEAST, GREATEST, DATE_TRUNC, JSON_AGG, EXTRACT, AT TIME ZONE, TIMESTAMPTZ, TSVECTOR, GRANT, REVOKE, CREATE ROLE.

3. Semantic translation bug - <=> operator: <=> is accepted and emitted unchanged in PostgreSQL-targeted output. In PostgreSQL, <=> is not a valid operator. The correct PostgreSQL equivalent of MySQL's <=> (null-safe equality) is IS NOT DISTINCT FROM.

   Minimal reproduction:

   -- Input (MySQL syntax)
   SELECT a <=> b FROM t;
   
   -- Expected PostgreSQL output
   SELECT a IS NOT DISTINCT FROM b FROM t;
   
   -- Observed: <=> passes through unchanged

Summary

polyglot-sql's performance story is solid: faster than sqlparser-rs in bulk workloads and faster than pg_query.rs at all tested sizes. The correctness picture has clear gaps. The false-positive rate (51-61%) and fidelity (78-89%) are the two metrics most in need of attention if the library is used in any PostgreSQL-validation context.

Happy to share the raw test SQL files or open a PR adding polyglot-sql to the benchmark's CI if that would be useful.

I will follow up with a separate post documenting the specific translation failures in detail (concrete SQL inputs, observed outputs, and expected outputs for each affected construct).