M19.1: Pin MAIL_OUTBOUND_ENABLED=true in phpunit.xml#69
Merged
Conversation
…in finding InvoiceNotificationTest and InvoiceDeliveryTest have been failing across recent CI runs because `.env.example` ships `MAIL_OUTBOUND_ENABLED=false` and the PR Tests workflow copies it verbatim into `.env`. With outbound disabled, `InvoiceDeliveryService::skipReason()` returns "Outbound mail is temporarily disabled." for every queue() call, so delivery rows land with `status='skipped'` and `DeliverInvoiceMail` never dispatches. Tests that don't check status slipped through, masking the breakage. Adding `MAIL_OUTBOUND_ENABLED=true` to phpunit.xml's `<php>` block forces a deterministic mail-outbound posture for tests, independent of whatever `.env` happens to be lying around. Same shape as the existing `MAIL_MAILER=array`, `QUEUE_CONNECTION=sync` overrides. Verified locally: 334/334 tests pass with the override in place and `.env` set to `MAIL_OUTBOUND_ENABLED=false` (CI-mirror conditions). The one test that intentionally exercises the disabled path (`InvoiceDeliveryTest::test_manual_send_is_skipped_when_outbound_mail_is_disabled`) sets the config to false inline via `config([...])` and is unaffected. Strategy doc 19.1 updates: carried-in finding section gets a Resolution note pointing to the tracking issue; the Phase 1 exit criterion for the flakiness is checked off. Fixes #68
M18.2 (Content Audit and Production) and M18.3 (Site Architecture and Publishing) are complete; applying the project's `x` prefix convention for completed strategy docs. Bundled with the M19.1 test fix PR (#69) for merge-time efficiency.
n8bar
added a commit
that referenced
this pull request
May 10, 2026
The InvoiceNotificationTest CI flakiness has been resolved (#68 / #69). Rewriting the carried-in finding section in past tense so it reads as historical context, with a one-line summary of the root cause and pointers to the issue/PR for the full story. Section heading carries `(resolved)` to make the status visible at a glance. Phase 1 actionable todo list (§1–§4) is unchanged.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MAIL_OUTBOUND_ENABLED=truetophpunit.xml's<php>block to give the test suite a deterministic mail-outbound posture independent of whatever.envexists at run time.Fixes #68.
Root cause
PR Tests has been red on every recent run (10+ in a row). Failures concentrate in
InvoiceNotificationTestandInvoiceDeliveryTest, with the same shape: assertions expectingstatus='queued'seestatus='skipped', and dispatch assertions see no job pushed.The chain:
.env.exampleshipsMAIL_OUTBOUND_ENABLED=false(a sensible cautious default for fresh installs).cp .env.example .envbefore running tests. There is no.env.testing, andphpunit.xmldoesn't pin the value, sofalsesurvives into the test environment.mail.safety.outbound_enabled === false,InvoiceDeliveryService::skipReason()returns"Outbound mail is temporarily disabled."for everyqueue()call.status='skipped'instead of'queued', andDeliverInvoiceMail::dispatch()is never called.status='queued'or "this job was pushed" fail. Tests that don't check status pass — which is why this drifted past prior reviews until the failure pattern was wide enough to notice.Locally we don't see it because dev
.envfiles haveMAIL_OUTBOUND_ENABLED=true.What changed
phpunit.xml:Same shape as the existing
MAIL_MAILER=arrayandQUEUE_CONNECTION=syncoverrides — test-deterministic config that doesn't depend on whatever.envis on disk..env.examplestays cautious for fresh installs.docs/strategies/19.1_NOTIFICATION_COVERAGE_AUDIT.md:Verification
./vendor/bin/sail artisan test→ 334/334 passing locally with.envintentionally set toMAIL_OUTBOUND_ENABLED=falseto mirror CI conditions. Override wins.InvoiceDeliveryTest::test_manual_send_is_skipped_when_outbound_mail_is_disabledalready setsconfig(['mail.safety.outbound_enabled' => false])inline, so it's insulated from the env-level flip.InvoiceNotificationTest(17/17) andInvoiceDeliveryTest(26/26) both green under the same CI-mirror conditions.Test plan
Process notes
This is the first GitHub Issue / PR pair under the M19 Issues trial documented in
docs/DOC_ROLES.mdand the M19 milestone doc. The split-of-concerns followed: issue body holds the bug report, this PR description holds the fix writeup. M20 kickoff revisits whether the trial sticks.