Skip to content

Add on-chain reward pipeline observability#940

Open
srt0422 wants to merge 2 commits into
devfrom
reward-pipeline-observability
Open

Add on-chain reward pipeline observability#940
srt0422 wants to merge 2 commits into
devfrom
reward-pipeline-observability

Conversation

@srt0422

@srt0422 srt0422 commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add EventRewardRedirectedToEcosystem proto message emitted when topic rewards are redirected to the ecosystem bucket
  • Escalate redirect log level from Info → Warn for better alertability
  • Track redirected_topic_ids in EventTopicRewardsSet for downstream monitoring
  • Add REWARD_REDIRECTED_TO_ECOSYSTEM_EVENT metric label

Context

Supports detecting reward pipeline outages like the 30-hour mainnet incident (2026-03-25) by making reward redirections visible as chain events. These events can be consumed by off-chain monitors (e.g. prodexer-monitor) to trigger PagerDuty alerts when rewards stop flowing.

Test plan

  • Run proto codegen (make proto-gen) to generate pb.go/pulsar.go files
  • Verify existing tests pass with updated event signatures
  • Confirm new events appear in block results when rewards are redirected
  • Validate backward compatibility — new field is additive (field 3)

🤖 Generated with Claude Code


Summary by cubic

Expose reward redirections on-chain and via metrics to make reward pipeline outages visible and alertable. Adds a new event, tags redirected topics, and escalates logs so off-chain monitors can trigger alerts when rewards stop flowing.

  • New Features

    • Emit EventRewardRedirectedToEcosystem when a topic’s reward is sent to the ecosystem (includes topic_id, reward_amount, reason, block_height).
    • Add redirected_topic_ids to EventTopicRewardsSet for downstream monitoring.
    • Escalate redirect logs to Warn and add the REWARD_REDIRECTED_TO_ECOSYSTEM_EVENT metric label.
  • Migration

    • Regenerate protos (make proto-gen); fields are additive.
    • If you consume events off-chain, parse EventRewardRedirectedToEcosystem and redirected_topic_ids to alert on redirections.

Written for commit 7b69b0f. Summary will update on new commits.

srt0422 and others added 2 commits March 28, 2026 06:23
Add EventRewardRedirectedToEcosystem proto message and emitter for
visibility when topic rewards are redirected to the ecosystem bucket.
Escalate redirect log from Info to Warn. Track redirected_topic_ids
in EventTopicRewardsSet. Supports monitoring the reward pipeline to
detect outages like the 30-hour mainnet incident.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dev refactored MoveCoinsFromAlloraRewardsToEcosystem to go through
GetBankingKeeper(). Our branch captures rewardAmount before the call
for logging/events. Resolution: keep both — the local variable capture
and the new GetBankingKeeper() call path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Mar 31, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf Linter / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed⏩ skippedMar 31, 2026, 3:55 PM

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Architecture diagram
sequenceDiagram
    participant RM as Rewards Module
    participant TK as Topic Keeper
    participant BK as Banking Keeper
    participant EM as Event Manager
    participant M as Metrics
    participant L as Logger
    participant OM as Off-chain Monitor

    Note over RM, OM: On-chain Reward Distribution Flow (per Topic)

    RM->>TK: GetTopicRewardNonce(topicId)
    TK-->>RM: nonce

    alt topicRewardNonce == 0 OR error
        Note right of RM: Trigger Redirection
        RM->>BK: MoveCoinsFromAlloraRewardsToEcosystem(amount)
        RM->>M: NEW: Incr(REWARD_REDIRECTED_TO_ECOSYSTEM_EVENT)
        RM->>L: CHANGED: Log.Warn("redirecting reward", topicId)
        RM->>EM: NEW: Emit EventRewardRedirectedToEcosystem(topicId, amount, reason, height)
    else ProcessRewards logic fails
        RM->>RM: ProcessRewards(topicId)
        Note right of RM: Logic Error or State Inconsistency
        RM->>BK: MoveCoinsFromAlloraRewardsToEcosystem(amount)
        RM->>M: NEW: Incr(REWARD_REDIRECTED_TO_ECOSYSTEM_EVENT)
        RM->>L: CHANGED: Log.Error("redirecting to ecosystem", topicId, error)
        RM->>EM: NEW: Emit EventRewardRedirectedToEcosystem(topicId, amount, reason, height)
    end

    Note over RM, EM: Final Epoch Summary
    RM->>EM: CHANGED: Emit EventTopicRewardsSet(rewards, redirectedTopicIds)
    
    Note over EM, OM: Block Finalized
    EM-->>OM: Read Block Events
    
    opt NEW: Redirected topics detected in events
        OM->>OM: Trigger Alert (e.g. PagerDuty)
    end
Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant