Live market prices: 60s fast-refresh loop between sweeps

[devin-ai-integration]

Summary

Adds a dedicated 60-second market data refresh loop that runs independently of the 15-minute full sweep cycle, so market prices update continuously between sweeps.

Problem: BTC, ETH, and all other market prices only updated every 15 minutes (when the full 32-source sweep ran). Between sweeps, prices appeared stale on the dashboard.

Fix: Two-tier refresh architecture:

• Full sweep (every 15 min): All 32 sources, full history, delta computation, LLM ideas, alerts
• Market-only refresh (every 60s): YFinance prices only, lightweight fetch, no history, patches currentData.markets in-place and pushes a market_update SSE event

Why

Users reported BTC/ETH prices looked stale. Investigation confirmed prices only updated on the 15-minute sweep cycle. There was no caching bug — the architecture just didn't have a fast path for market data.

Scope

• Small UX improvement
• Dashboard change
• Docs/config change

Changes

File	Change
apis/sources/yfinance.mjs	New collectQuick() + fetchQuoteQuick() for lightweight price-only fetch (6s timeout, no history, range=1d&interval=1d). Existing fetchQuote unchanged (range=5d&interval=1d).
server.mjs	New runMarketRefresh() on a 60s setInterval. Skips when sweep is in progress. Uses mergeCategory() to patch prices into currentData.markets while preserving symbols that failed the quick fetch. Broadcasts market_update SSE.
dashboard/public/jarvis.html	SSE handler for market_update: patches D.markets, calls renderLower() (where market panels live) + renderTopbar(). No full reinit().
.env.example	New MARKET_REFRESH_SECONDS=60 config var.

Validation

• Server starts cleanly with [Crucix] Market ticker refresh: every 60s in logs
• /api/health returns "marketRefreshSeconds": 60
• Confirmed YFinance returns fresh prices on consecutive calls (BTC varied by ~$25 across 4-minute gap)

Config and Docs

• .env.example updated — new MARKET_REFRESH_SECONDS variable
• README.md not updated (consider adding a note about the two-tier refresh)

Updates Since Initial Review

Addressed three Devin Review findings:

1. Reverted fetchQuote interval change — Initially changed fetchQuote from 5d/1d to 1d/5m, but this broke the prevClose fallback (closes[closes.length - 2] would return a 5-min candle instead of previous day's close). Reverted to 5d/1d; the fast loop uses its own fetchQuoteQuick with 1d/1d which only reads meta fields.
2. Added mergeCategory() for partial-failure resilience — Previously, if a single symbol failed the quick fetch, it was dropped from currentData.markets entirely. Now, previously-good data for failed symbols is preserved until the next successful fetch or full sweep.
3. Fixed market_update SSE render target — The handler originally called renderRight(), but D.markets is only read inside renderLower() (line 1572). Updated to call renderLower() so updated prices are actually rendered on screen.

Human Review Checklist

• Yahoo rate limiting: The fast loop makes 15 parallel requests every 60s. fetchQuoteQuick returns null on failure (graceful degradation via mergeCategory), but verify this rate is sustainable in practice.
• Silent error swallowing: fetchQuoteQuick has a bare catch { return null; } with no logging. Consider whether a debug-level log would help troubleshooting.
• mergeCategory correctness: Verify the merge logic correctly handles edge cases — e.g., when prevItems has symbols not in newItems, they should be appended (stale but visible is better than missing).
• No browser testing yet: The renderLower() fix was identified via code review. End-to-end browser testing to confirm prices visually update every 60s is recommended.

Checklist

• This PR stays within one bugfix or one feature family
• I kept unrelated changes out of the diff
• I considered security for any mixed-source content rendering
• I tested the changed path locally

Link to Devin session: https://app.devin.ai/sessions/a046d43371284e3f8175cb76185f3ced
Requested by: @jakexcosme

---

[devin-ai-integration]

Original prompt from Jake

Check our repo against this version: ltmsuma#1 - let me know how far behind we are and what we need to change in order to make our fork in COG-GTM up to date as /ltmsuma/ - come up with a plan for this

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR that start with 'DevinAI' or '@devin'.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring