Skip to content

Benchmark: Per-author event recall as MAB reward #11

Description

@alltheseas

Summary

Replace MAB-UCB's binary "is this author covered?" reward with a continuous signal: how many of this author's events did this relay actually deliver?

Why

The current reward function treats all covered authors equally. But a relay covering 10 authors while keeping only 7 days of history should score lower than one covering 5 authors with full retention. Per-author event recall as the reward signal would let the algorithm distinguish between relays that technically cover an author and relays that actually deliver their posts.

What to do

  1. Change the MAB reward from binary coverage to fractional event recall per author
  2. During each round's observation phase, measure events_retrieved / events_expected per (relay, author) pair
  3. Benchmark against binary-reward MAB-UCB across all 6 time windows — especially at longer windows where retention differences matter most

Effort

Higher — requires modifying the reward function and the observation phase to query event counts.

Reference

  • IMPLEMENTATION-GUIDE.md: Improvement Opportunities (higher effort)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions