AI merge up to PG14

.claude/skills/README.md

@@ -0,0 +1,17 @@
+# Project skills
+
+Reusable skills distilled from the PG14 → GGDB merge, fix, and
+regression-reconciliation work. Each is a Claude Code skill
+(`.claude/skills/<name>/SKILL.md`, auto-discovered); a human can also just read it.
+
+| Skill | Use it when |
+|---|---|
+| [greengage-build](greengage-build/SKILL.md) | A source change must become a running binary; avoiding the stale-binary trap |
+| [greengage-regress-tests](greengage-regress-tests/SKILL.md) | Running a regress/isolation2 test under optimizer on/off, with setup deps |
+| [greengage-answer-file-regen](greengage-answer-file-regen/SKILL.md) | A test's diff is cosmetic drift — regenerate `.out` without masking a real bug |
+| [greengage-cluster-ops](greengage-cluster-ops/SKILL.md) | Monitoring a long run, disk/health/OOM, recovering a degraded/crashed cluster |
+| [greengage-debug](greengage-debug/SKILL.md) | A crash/assert/hang/wrong-result — logs, repro, instrumentation, gdb |
+| [greengage-internals](greengage-internals/SKILL.md) | Writing/reviewing a backend fix — MPP planner/executor, merge re-graft method |
+
+These complement `CLAUDE.md` (build/test/style) and the
+major-version-merge workflow it references.

.claude/skills/greengage-answer-file-regen/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: greengage-answer-file-regen
+description: Regenerate optimizer=off (and other) regression answer files for cosmetic PG14/GGDB drift WITHOUT masking a real bug. Use when a failing test's diff is deparse/plan-shape/psql-format/error-message drift rather than a behavior change. Includes the success->error safety gate, noise stripping, and verification.
+---
+
+# Regenerating answer files safely
+
+After a PG major-version merge, many optimizer=off base `.out` files drift
+cosmetically: PG14 psql adds a `\d+` Compression column; `AT TIME ZONE`
+deparses as `(x AT TIME ZONE z)` not `timezone(z,x)`; grammar/error text changes
+(CREATE STATISTICS on expressions, `EXTRACT`); MPP plan shapes and slice numbers
+shift; upstream adds new test queries. These are answer-file regens, not bugs —
+but a real bug can hide in the same pile (e.g. a query that returned data now
+ERRORs). Never bulk-copy blindly.
+
+## The SAFETY GATE: success->error scan (the most important step)
+
+The signature of a real bug hiding in a "cosmetic" pass is a removed result row /
+`(N rows)` replaced by an added `+ERROR`. Scan for it **on the gpdiff
+`regression.diffs`, NOT the raw `git diff`**:
+
+- `regression.diffs` already honors `--start_ignore/--end_ignore` blocks and
+  atmsort row-sorting, so its success->error hits are real.
+- A raw `git diff` scan FALSE-POSITIVES: ignore-block content (the .out lines
+  aren't always individually `GP_IGNORE:`-prefixed) and row-reordering context
+  shifts a `(N rows)` next to an unrelated `+ERROR`. This burned an entire pass —
+  join/portals/subselect were flagged but were actually cosmetic (errors inside
+  `--start_ignore` blocks documenting accepted GGDB limits + `error->error`
+  stale-line drift like `pathnode.c:485 -> :275`).
+- **A clean re-run that reports `ok` is the definitive proof** there is no
+  non-ignored success->error (it would otherwise FAIL the test).
+
+Real bugs this gate caught (do NOT regen — fix them): gp_dqa multi-DQA+FILTER
+`variable not found in subplan target list`; matview `DISTRIBUTED BY` an
+aggregate `could not find hash distribution key expressions`; matview REFRESH
+CONCURRENTLY `text *= text`.
+
+## Source of truth: the CI result tarball, not a local run
+
+The CI matrix runs the SAME `expected/*.out` across FOUR jobs — {JIT, non-JIT} ×
+{ORCA `optimizer=on`, Postgres `optimizer=off`} — so a regen that greens one can
+break another. Three rules learned by breaking them:
+
+- **Regenerate from the failing job's CI result tarball** (`*_results.tar.gz` →
+  `gpdb_src/src/test/regress/results/<t>.out`), NOT a local gpdemo run — *unless*
+  the test's output is fully normalized by `explain_filter`/atmsort. A local run
+  bakes environment-specific text that CI then rejects: a fresh db name folded into
+  `current_database()` literals (e.g. `'jps2'` vs `'regression'`), `1 segment`
+  instead of the normalized `n segments`, local row-ordering. (Fully-normalized
+  tests like `explain`, where numbers map to `N`, ARE safe to regen locally.)
+  Correctness check for a big reorder diff: the sorted DATA-ROW SET must equal the
+  known-good `_optimizer.out` set (e.g. qp_misc_jiras opt=off == ORCA, 0 set diff).
+- **A shared base `<t>.out` (no `<t>_optimizer.out`) is used by ORCA too** (ORCA
+  falls back to base). Regenerating it to the Postgres-planner output BREAKS the
+  ORCA jobs. Only regen a base file if a separate `_optimizer.out` exists.
+- **`_optimizer.out` coverage is self-balancing — do NOT "backfill" it.** A test
+  without one passes under ORCA only because its *non-ignored* output already
+  matches the base (data-only, ORCA-falls-back, or the plans are in `--start_ignore`
+  blocks). Any test whose asserted output genuinely diverges under ORCA already has
+  an `_optimizer.out` or it would be red. Adding one to a passing shared-base test
+  asserts nothing new (verified for delete/insert_conflict/with). Real ORCA gaps are
+  *untested features* (find them by exercising the feature under both optimizers, cf.
+  the GROUP BY DISTINCT bug 4dd440a4e69), not missing `_optimizer.out` files.
+- **Never bake JIT-only output into a file a non-JIT job compares.** Under jit,
+  EXPLAIN adds a ` Settings: jit = ...` line (init_file-ignored) and inflated
+  `Executor memory:` (atmsort-normalized) — so they don't diff — but the wide
+  Settings row still perturbs `explain_filter`'s column width. See "What NOT" below.
+
+## Procedure
+
+1. **Prefer the CI result tarball** (above). Only when regenerating locally is
+   appropriate, run the test(s) under `optimizer=off` in a FRESH db with full setup
+   (see [greengage-regress-tests]); the polluted `regression` db gives
+   `already exists` cascades.
+2. Scan `regression.diffs` for success->error. Investigate any hit (is the new
+   ERROR inside a `--start_ignore` block? is it `error->error` with only a line
+   number change? does a `--FAIL with ERROR` comment in the `.sql` document it?).
+3. For confirmed-cosmetic tests, `cp results/<t>.out expected/<t>.out`.
+4. **Strip gpdiff-ignored noise** so the diff is real content only — `init_file`
+   drops these but the raw copy keeps them:
+   ```
+   drop lines matching  ^(HINT|NOTICE):\s+.+'DISTRIBUTED BY' clause
+                  and    Distributed by: \(
+   ```
+   (omitting this adds dozens of NOTICE lines of git noise per file).
+5. **Verify**: re-run on a fresh db → `ok` / empty `regression.diffs`. If the
+   same-db re-run fails only with `<test>_tbl already exists`, that's a re-run
+   artifact (drop the table and re-run).
+6. `docker cp` lands files root-owned on the host — `chown` before editing.
+
+## What NOT to regen
+
+- **Flaky** tests: `truncate_gp` (AO segfile-stats vary 3 rows vs 0). Regenerating
+  captures one run and flakes on the next. (`explain` LOOKS jit-flaky but is NOT —
+  the jit `Settings:` line widens the `explain_filter` output column, which atmsort
+  doesn't normalize; the real fix is to pin the jit GUCs to their boot defaults at
+  the top of explain.sql — `set jit=off; set jit_above_cost=100000; set
+  optimizer_jit_above_cost=7500;` — so EXPLAIN(SETTINGS), which reports only
+  modified-from-boot-default GUCs, drops them. Then regen both expected files.)
+- **OOM victims**: the gpdemo is too small for the full concurrent schedule;
+  qp_olap_window/qp_with_clause/etc. hit `Out of memory`/`failed to acquire
+  resources` and their results are corrupt — they pass on CI's bigger runner.
+- **Environment-specific**: gp_toolkit (live replication conninfo), gp_connections
+  (`$COORDINATOR_DATA_DIRECTORY` unset), createdb (fault injection didn't fire).
+
+## Inherent reorder noise
+
+For data-returning tests the git diff is large (thousands of lines) because
+unordered MPP results come back in segment order; gpdiff sorts them, so the test
+is clean, but git shows every moved row. Commit it as an isolated, clearly-labeled
+regen commit; the success->error gate (not the visual diff) is the correctness check.
+
+See also: [greengage-regress-tests], [greengage-internals].

.claude/skills/greengage-build/SKILL.md

@@ -0,0 +1,66 @@
+---
+name: greengage-build
+description: Build and install the GGDB backend (and ORCA) inside the docker test container, and avoid the stale-binary trap where committed source fixes silently don't take effect. Use whenever a C/C++ source change needs to become a running binary, before re-testing.
+---
+
+# Building GGDB in the container
+
+Builds run in the `gpdb_clean` docker container (not natively), as `gpadmin`,
+with `sudo`. The source tree the build compiles is `/home/gpadmin/gpdb_src`
+inside the container — **this is a plain copy, not a git checkout** — while the
+git working tree is on the host at `/home/dvoronkov/ws/gpdb`. You must copy
+edited files in before building.
+
+## Edit → build → install → run loop
+
+```bash
+# 1. copy the edited file into the container source tree
+sudo docker cp src/backend/<path>.c gpdb_clean:/home/gpadmin/gpdb_src/src/backend/<path>.c
+
+# 2. CRITICAL: touch after docker cp — docker cp preserves the host mtime, which
+#    is often OLDER than the .o, so make would NOT recompile (the "stale" trap).
+sudo docker exec -u gpadmin gpdb_clean bash -lc 'touch /home/gpadmin/gpdb_src/src/backend/<path>.c'
+
+# 3. rebuild the backend and (re)link + install
+sudo docker exec -u gpadmin gpdb_clean bash -lc \
+  'cd /home/gpadmin/gpdb_src/src/backend && make >/tmp/bld.log 2>&1 && make install >>/tmp/bld.log 2>&1 && echo BUILD_OK || (echo BUILD_FAIL; grep -iE "error:" /tmp/bld.log | head)'
+
+# 4. the running cluster only picks up the new postgres binary after a restart
+sudo docker exec -e USER=gpadmin -u gpadmin gpdb_clean bash -lc \
+  'source /usr/local/greenplum-db-devel/greenplum_path.sh; export PGPORT=7000 \
+     MASTER_DATA_DIRECTORY=/home/gpadmin/gpdb_src/gpAux/gpdemo/datadirs/qddir/demoDataDir-1; \
+   gpstop -ar'
+```
+
+`GPHOME=/usr/local/greenplum-db-devel`. `docker exec` does NOT set `$USER`;
+pass `-e USER=gpadmin` or `gpstop`/`gprecoverseg`/`gpconfig` abort with
+"USER environment variable must be set".
+
+## The stale-binary trap (read this before concluding "my fix doesn't work")
+
+The installed `/usr/local/greenplum-db-devel/bin/postgres` can be timestamp-
+stale relative to the source: a committed fix is in the `.c` but NOT in the
+running binary, so behavior looks unfixed and pass-counts are measured against
+stale code. Symptoms: a fix you can read in the source does not change runtime
+behavior. Verify with `ls -la --time-style=+%Y-%m-%d_%H:%M` on the `.o`, the
+`.c`, and `bin/postgres`; the binary must be newer than the source.
+
+## ORCA vs assert builds (two different binaries surface different bugs)
+
+- The assert campaign builds `--enable-cassert --disable-orca` (optimizer=off);
+  `set optimizer=on` then errors "ORCA is not supported by this build".
+- The ORCA jobs build `--enable-orca` (optimizer on works).
+- The **assert∩ORCA intersection** (cassert + ORCA) was historically never
+  tested and finds real crashes ("jit/regression tests with orca").
+- To switch a container to ORCA+assert: drop `--disable-orca` from the configure
+  line and rebuild. The gporca `.o` are usually already in-tree, so the rebuild
+  is ~3 min (not from scratch).
+
+## Don't `make clean`
+
+`make clean` is destructive: it nukes the ORCA build (`src/backend/gporca/`),
+forcing a long from-scratch rebuild and `-j` link races. Prefer `touch` + `make`
+for targeted rebuilds. If you must rebuild ORCA, build `gporca` before linking.
+
+See also: [greengage-cluster-ops] (restart/recovery), [greengage-debug]
+(instrumentation rebuilds), [greengage-internals] (what to change).

.claude/skills/greengage-cluster-ops/SKILL.md

@@ -0,0 +1,67 @@
+---
+name: greengage-cluster-ops
+description: Operate the gpdemo MPP cluster during long test/build cycles - read segment health correctly, monitor disk and long-running jobs, detect OOM, and recover a degraded/crashed cluster with gprecoverseg. Use when a run hangs/crashes, a segment goes down, "Cluster validation failed" appears, or disk fills.
+---
+
+# Operating & recovering the gpdemo cluster
+
+## Segment layout (gpdemo, 3 primaries + mirrors + standby)
+
+- Coordinator: content **-1**, dbid1, port **7000**, `MASTER_DATA_DIRECTORY=.../qddir/demoDataDir-1`. Standby: dbid8/7001.
+- content 0: primary dbid2/7002, mirror dbid5/7005. content 1: dbid3/7003 + dbid6/7006. content 2: dbid4/7004 + dbid7/7007.
+
+## Read health correctly
+
+```sql
+select role,preferred_role,mode,status,count(*) from gp_segment_configuration group by 1,2,3,4 order by 1,2;
+```
+Healthy looks like `m|m|s|u|4`, `p|p|s|u|3`, **`p|p|n|u|1`**. The last is the
+coordinator (content -1) — **mode `n` for content -1 is NORMAL** (its standby is
+tracked via `pg_stat_replication`, `gp_walreceiver streaming sync`), do not treat
+it as degraded. Trouble = `status=d` (down) or `role<>preferred_role` (failed over).
+
+## Disk hygiene (regress workflows leak disk)
+
+- `df -h` between runs. Cores, 13 GB datadirs, and docker images fill the disk;
+  a full disk can truncate host source files (restore from container/HEAD).
+- Set `core_pattern=|/bin/true` so assert cores don't pile up.
+- Clean `regression.diffs`, `results/`, leftover temp dbs, and `/tmp/*.log` promptly.
+
+## Monitor a long run (build or full schedule, 20-40 min)
+
+Run it detached and poll its log; don't block one command on the whole run:
+```bash
+# launch detached inside the container
+nohup env PGOPTIONS="-c optimizer=off" make installcheck-good > /tmp/run.log 2>&1 &
+# poll for completion markers (until-loop; cache-friendly 6s ticks)
+for i in $(seq 1 90); do grep -qE "tests passed|tests, .* failed|make.*Error" /tmp/run.log && break; sleep 6; done
+grep -c "\.\.\. FAILED" /tmp/run.log
+```
+
+## OOM under load (small gpdemo)
+
+The full concurrent schedule OOMs the small gpdemo: `ERROR: Out of memory` /
+`failed to acquire resources on one or more segments`, and a heavy parallel group
+can **crash a segment** (it flips to `m|p|n|d`), after which `pg_regress` prints
+`Cluster validation failed` and runs 0 tests. Mitigate with `--max-connections=3`
+(or `MAX_CONNECTIONS=3 make ...`), or run a lighter schedule prefix that excludes
+the OOM-heavy `qp_*` (those are in greenplum_schedule, not parallel_schedule).
+
+## Recover a degraded/crashed cluster
+
+```bash
+source greenplum_path.sh; export PGPORT=7000 MASTER_DATA_DIRECTORY=.../qddir/demoDataDir-1
+gprecoverseg -a                         # recover the down segment (as a mirror)
+# wait for WAL resync: poll until count(mode<>'s' and content>=0)=0
+gprecoverseg -ar                        # rebalance: restore preferred primary/mirror roles
+# if the coordinator itself is down:
+gpstart -a                              # or gpstop -ar to restart
+```
+
+Restart fragility: `gpstop -ar` on a cluster with a hung motion-IPC QE FAILS to
+shut down cleanly (sends SIGQUIT then SIGABRT), crashing it — then `gpstart -a`
+to bring it back, `gprecoverseg` to re-sync. An uninterruptible motion-IPC hang
+(e.g. an executor infinite loop) cannot be killed by `statement_timeout`; you must
+`pg_terminate_backend` the stuck pid, and it may leave a degraded segment.
+
+See also: [greengage-build] (restart after install), [greengage-debug].

.claude/skills/greengage-debug/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: greengage-debug
+description: Diagnose GGDB crashes, asserts, and wrong-results - find the crash in the segment/QD logs, reproduce reliably (base64 SQL, forced heisenbug conditions), instrument with temporary elog, and read structs live with gdb on a no-DWARF binary. Use when a query crashes a backend, hits a FailedAssertion, hangs, or returns wrong data.
+---
+
+# Debugging GGDB
+
+## Find the crash in the logs
+
+QD/segment logs are CSV at
+`gpAux/gpdemo/datadirs/qddir/demoDataDir-1/log/*.csv` (and `dbfast*/.../log/*.csv`
+for segments). Signatures:
+- `TRAP: FailedAssertion("<cond>", File: "<f>.c", Line: N)` — assert build crash.
+- `server process ... was terminated by signal 6/11` + `Failed process was running: <SQL>` — abort/segfault; grep the crashing PID's lines.
+- `PANIC`, `Out of memory`, `failed to acquire resources` (see [greengage-cluster-ops] for the OOM/crash recovery).
+Take the failing statement from the CSV, then narrow it to a minimal query.
+
+## Reproduce reliably
+
+- **base64 the SQL** to avoid nested-quote mangling: building SQL inside
+  `bash -lc '...'` eats inner single quotes (so `'x'` becomes `x` and string
+  literals break, or `$abc$` dollar-quoting expands `$` to the PID). Write the
+  `.sql` on the host, `base64 -w0`, pass via `-e B64=...`, `base64 -d` in-container.
+- Match the EXACT failing shape: column types, GROUP BY, FILTER, optimizer
+  on/off, even JIT (`set jit=on; set jit_above_cost=0`) — small differences flip
+  the code path (e.g. multi-DQA needs `FILTER` to fail, not just DISTINCT).
+
+## Force a heisenbug condition
+
+When a bug depends on transient state (e.g. a relcache descriptor that is mutated
+in place and only bites once an invalidation reloads it), inject the condition to
+make it deterministic. Example used for the matview REFRESH bug:
+```c
+CacheInvalidateRelcacheByRelid(relid);   /* temporary, in the suspect function */
+AcceptInvalidationMessages();
+```
+Rebuild, reproduce (now the unfixed code fails every time), apply the real fix,
+confirm it passes WITH the forced condition still active, then remove the
+injection. This converts "passes locally, fails in CI" into a real local repro.
+
+## Instrument with temporary elog
+
+Add `elog(LOG, "DBG ... %s", ...)` at the suspect point, rebuild
+([greengage-build]), run, and read it back from the QD CSV log
+(`grep "DBG" $QDLOG`). Print the concrete values you can't infer — e.g. the
+generated SQL string and the actual column names of a transient relation — to
+settle "what does the code really see". Remove all instrumentation before
+committing; verify `git diff` is exactly the one-line fix.
+
+## Live gdb (ptrace container, no DWARF)
+
+Commit + re-run the container with `--cap-add=SYS_PTRACE --security-opt
+seccomp=unconfined` (yama ptrace_scope=1 → attach as root). Add a `pg_sleep(30)`
+in the query to open a window, find the per-query QE pid, `gdb -p`. The release
+binary has no DWARF: read structs by **byte offset** and resolve addresses with
+`addr2line`/`objdump`. Avoids the ereport-probe rebuild loop.
+
+## Blast-radius triage
+
+A single root cause often produces many `regression.diffs` (e.g. one missing
+libpq command-queue entry broke COPY/sequences MPP-wide). Group failures by the
+common error string/PID, fix the one root cause, re-measure — don't chase each
+diff. Beware measuring against a [stale binary](greengage-build).
+
+See also: [greengage-internals] (what the fix should be), [greengage-build].