Further strip offending annotations from `flow://relaxed-write-schema`. Maybe strip all of them?

[jshearer]

Problem

A customer's materialization failed schema validation because SHA-256 redaction produces 71-character hashes (sha256:<64 hex>), but the connector schema's maxLength: 20 (from source varchar(20)) leaks into the read path via flow://relaxed-write-schema. The write schema overlay correctly sets maxLength: 71 with redact: {strategy: "sha256"}, and the inferred schema has maxLength: 128, but the relaxed write schema preserves the connector's maxLength: 20 unchanged, and the allOf intersection takes the most restrictive constraint:

flow/crates/doc/src/shape/intersect.rs

Lines 34 to 35 in 008278b

	let max_length = match (lhs.max_length, rhs.max_length) {
	(Some(l), Some(r)) => Some(l.min(r)),

We have triaged by applying manual workarounds (removing the relaxed-write-schema ref from readSchema) a couple of times now, but any collection combining SHA-256 redaction with a length-constrained connector field will hit this.

Context

RelaxedSchemaObj explicitly strips keywords via #[serde(skip_serializing)] and passes everything else through a catch-all:

flow/crates/models/src/schemas.rs

Lines 215 to 217 in 008278b

	// Other keywords are passed-through.
	#[serde(flatten)]
	pass_through: BTreeMap<String, serde_json::Value>,

The original design intent (Discussion #1988) was to remove "most validations (type, format, required, etc)" while preserving title, description, reduce, default, and conditional keywords, but the implementation only strips what's enumerated and passes everything else through.

The strip list has grown reactively, each addition driven by a production bug:

• Feb 2025 (#1937): type, required, format (initial implementation)
• Jun 2025 (#2209): const, enum (customer materialization failure)
• Jan 2026 (#2626): additionalProperties: false (closed-object constraint leak)

When RelaxedSchemaObj was introduced (Feb 2025), no connector emitted maxLength, so the omission had no effect. Connectors started emitting maxLength for column sizing in Oct 2025 (connectors#3327, connectors#3341), and the redaction feature landed in Sep 2025 (#2383). The bug requires both to be present on the same field, which is why it took until now to surface.

Proposed fix

Strip maxLength and minLength from the relaxed schema, following the existing skip_serializing pattern. This is safe because the relaxed schema only appears in the read path (readSchema's allOf) and does not affect the write schema, connector schema, or materialization column sizing. The inferred schema still carries its own length constraints from observed data, so the read path isn't left unconstrained.

Phil agreed with this approach: "I think I'd probably be inclined to have the relaxed schema just strip out maxLength entirely. Seems like it was perhaps an oversight that it wasn't stripped out already."

Open question

The same logic applies to every other numeric/pattern validation keyword that passes through today: minimum, maximum, exclusiveMinimum, exclusiveMaximum, minItems, maxItems, pattern, multipleOf. Should we strip all of them now to match the original "most validations removed" design intent, or just fix maxLength/minLength and wait for the next bug?

---

@jwhartley FYI

[jshearer]

Pointed Claude at the open question, and the tl;dr is we should probably strip all of them. I'm not super familiar with this mechanism yet though so would like @jgraettinger's input.

---

Summary Table

Keyword	Currently passes through?	Emitted by inference?	Risk if kept	Risk if stripped	Recommendation
minLength	YES	YES	Medium (redaction/truncation)	Low	STRIP
minimum	YES	YES	Medium (schema evolution)	Low	STRIP
maximum	YES	YES	Medium (schema evolution)	Low	STRIP
exclusiveMinimum	YES (rare)	NO	Low (rarely emitted)	Negligible	STRIP
exclusiveMaximum	YES (rare)	NO	Low (rarely emitted)	Negligible	STRIP
minItems	YES	YES	Medium (format evolution)	Low	STRIP
maxItems	YES	YES	Medium (format evolution)	Low	STRIP
pattern	YES (rare)	NO	High (redaction breaks patterns)	Negligible	STRIP
multipleOf	YES (rare)	NO	Low (precision changes)	Negligible	STRIP

Principle

The pattern from already-stripped keywords (type, required, format, const, enum, additionalProperties: false) is: strip value constraints, preserve annotations (title, description, reduce, redact, default, secret, x-*). All 9 keywords under investigation are value constraints, not annotations. The inferred schema already captures the actual observed bounds from real data, making the write schema's constraints unnecessary or harmful on the read path.

---

Claude's analysis (100% AI generated)

Consolidated Report: JSON Schema Validation Keywords in RelaxedSchemaObj

Background

The relaxed write schema (to_relaxed_schema() in crates/models/src/schemas.rs:102-105) transforms a collection's write schema into a permissive form that gets intersected with the inferred schema via allOf to produce the read schema. Currently strips: type, required, format, const, enum, additionalProperties: false. Everything else passes through a #[serde(flatten)] pass_through catch-all.

The maxLength bug demonstrated that validation keywords leaking into the read schema can cause production failures when the read path transforms data (e.g., SHA-256 redaction producing 64-char hashes that exceed the connector's maxLength).

How intersection constrains the read path

The intersection behavior for ALL numeric/length bounds is the same: take the more restrictive value. When the relaxed write schema preserves a constraint and the inferred schema has a looser (or absent) constraint, the write schema's value wins in the intersection. This means preserved validation keywords from the write schema become hard constraints on the read path.

---

1. minLength

Question	Answer
Who emits it?	Connector schemas (PostgreSQL varchar columns emit minLength). Shape inference tracks it in StringShape.min_length. schema.rs:160-161 emits it when != 0. inference.rs:161-162 parses it.
In relaxed schemas?	YES. Snapshot confirms "minLength": 16 passes through on a_string property.
Intersection behavior	lhs.min_length.max(rhs.min_length) - takes the HIGHER (more restrictive) value. Write schema's minLength: 16 wins over inferred 0.
Regression if stripped?	Low. The inferred schema captures actual observed minLength from data. No downstream consumer depends on the write schema's minLength in the read path.
Bug if kept?	YES. If a field is redacted or truncated on the read path, a minLength constraint from the write schema would reject legitimately shorter values. Same mechanism as the maxLength bug.
Recommendation	STRIP. Connector's minLength constraint is a write-time validation that should not constrain the read path.

2. minimum

Question	Answer
Who emits it?	Connector schemas (e.g., citi-rides.schema.json has minimum: 0, minimum: -90). Shape inference tracks it in NumericShape.minimum. schema.rs:172-174 emits it. inference.rs:166-173 parses it. Widen logic (widen.rs:250-278) adjusts bounds from observed data.
In relaxed schemas?	YES. Passes through pass_through.
Intersection behavior	Takes the HIGHER (more restrictive) value. Write schema's minimum: 0 beats inferred minimum: -100.
Regression if stripped?	Low-medium. The inferred schema will carry the actual observed minimum from data. If a connector intentionally constrains a field to non-negative values, the inferred schema will also reflect this if all observed data was non-negative.
Bug if kept?	Possible. If the connector schema says minimum: 0 but the inferred schema widens to include negative values (from observed data), the write schema's constraint would reject valid negative values on the read path. The write schema is enforcing a constraint at read time that the data has already evolved past.
Recommendation	STRIP. Validation-only constraint. The inferred schema provides the actual observed bounds.

3. maximum

Question	Answer
Who emits it?	Same sources as minimum. Connector schemas emit maximum for database column constraints. Shape inference tracks it in NumericShape.maximum.
In relaxed schemas?	YES. Passes through pass_through.
Intersection behavior	Takes the LOWER (more restrictive) value. Write schema's maximum: 100 beats inferred maximum: 200.
Regression if stripped?	Low. Same reasoning as minimum.
Bug if kept?	YES. If the connector schema's numeric range constraint becomes stale (connector upgrades, schema evolution), the old bound rejects valid data on the read path. Direct numeric analog of the maxLength bug.
Recommendation	STRIP.

4. exclusiveMinimum

Question	Answer
Who emits it?	JSON Schema validator fully supports it (crates/json/src/schema/keywords.rs has ExclusiveMinimumPosInt, ExclusiveMinimumNegInt, ExclusiveMinimumFloat). One example found: examples/stock-stats/schemas/L1-tick.schema.yaml uses exclusiveMinimum: 0.
In relaxed schemas?	YES if present in write schema, via pass_through.
Intersection behavior	NumericShape does NOT track exclusive bounds (only minimum/maximum). Shape-level intersection ignores them. However, at the JSON Schema validation level in allOf, the constraint IS enforced.
Regression if stripped?	Very low. Rarely emitted (1 example in entire repo).
Bug if kept?	Same mechanism as minimum. Low probability given rarity.
Recommendation	STRIP. Same class as minimum. Strip for consistency even though rarely emitted.

5. exclusiveMaximum

Question	Answer
Who emits it?	Same support infrastructure as exclusiveMinimum. No examples found in repo outside test suites.
In relaxed schemas?	YES if present, via pass_through.
Intersection behavior	Same as exclusiveMinimum - not tracked by Shape, but enforced at JSON Schema validation level.
Regression if stripped?	Negligible. No production usage found.
Bug if kept?	Same mechanism as maximum.
Recommendation	STRIP.

6. minItems

Question	Answer
Who emits it?	CDC connectors (fixture shows minItems: 3 for WAL location array, minItems: 128 for timestamp arrays). Shape inference tracks it in ArrayShape.min_items. schema.rs:115 emits it when != 0.
In relaxed schemas?	YES. Snapshot confirms "minItems": 128 and "minItems": 3 pass through.
Intersection behavior	lhs_min.max(rhs_min) - takes the HIGHER (more restrictive) value.
Regression if stripped?	Low. The inferred schema captures actual observed array lengths. The capture runtime comment at serve.rs:192 explicitly acknowledges that "tight minItems/maxItems bounds" can cause problems.
Bug if kept?	Possible. If a CDC connector's data format evolves (e.g., WAL location array goes from 3 to 4 elements), the old minItems: 3, maxItems: 3 from the write schema would reject valid 4-element arrays on the read path.
Recommendation	STRIP. The capture runtime already treats tight bounds as problematic.

7. maxItems

Question	Answer
Who emits it?	Same sources as minItems. Fixture shows maxItems: 3 and maxItems: 128.
In relaxed schemas?	YES. Confirmed in snapshot.
Intersection behavior	lhs_max.min(rhs_max) when both present - takes the LOWER (more restrictive) value.
Regression if stripped?	Same as minItems.
Bug if kept?	YES. Same scenario as minItems. Direct array analog of the maxLength bug.
Recommendation	STRIP.

8. pattern

Question	Answer
Who emits it?	JSON Schema parser supports it (crates/json/src/schema/build.rs:386-394). Flow's own system schema (flow.schema.json) uses it for collection name validation. NOT tracked by StringShape. sources/src/loader.rs:506 explicitly ignores it: Keyword::Pattern { .. } => ().
In relaxed schemas?	YES if present in write schema, via pass_through.
Intersection behavior	Not tracked by Shape. At JSON Schema allOf level, both patterns in the write schema and inferred schema would need to match. Since inference never emits pattern, only the write schema's pattern applies.
Regression if stripped?	Very low. No connector schemas found that emit pattern in write schemas.
Bug if kept?	YES. If a connector write schema says pattern: "^[0-9]+" and the field gets redacted to "sha256:abc...", the pattern would reject legitimate data.
Recommendation	STRIP. Regex constraints have the same conflict risk with data transformations as any other validation keyword.

9. multipleOf

Question	Answer
Who emits it?	JSON Schema validator supports it (3 keyword variants for different numeric types). NOT tracked by NumericShape. Appears in 39 files, almost all in official JSON Schema test suites. No connector-specific usage found in the repo.
In relaxed schemas?	YES if present, via pass_through.
Intersection behavior	Not tracked by Shape. At JSON Schema allOf level, the constraint applies as validation.
Regression if stripped?	Very low. No production usage found.
Bug if kept?	Possible but low probability. If a connector specifies multipleOf: 0.01 for decimal precision but values on the read path have different precision (e.g., after reduction/aggregation), validation would fail.
Recommendation	STRIP. Validation-only keyword with no Shape tracking. Eliminates a potential edge case.