Document pagination performance issues from PR #33

[justinabrahms]

Summary

Investigation of pagination slowness introduced in PR #33 reveals three performance issues, with one main culprit causing significant slowdown for certain user scenarios.

Issues Identified

1. Cache Invalidation (One-time)

• Impact: Severe slowdown on first sync after update
• Cause: URL changes from /user/following to /user/following?page=1&per_page=100 invalidated all ETag caches
• Duration: One-time issue, self-corrects after first sync

2. Extra API Calls for Exactly 100 Items ⚠️ Main Issue

• Impact: 23-50% slowdown for affected users
• Cause: Current logic can't distinguish "exactly 100 items total" from "100 items with more pages"
• Behavior: Makes unnecessary extra call to page 2 which returns 0 items
• Evidence: github/client_test.go:687 explicitly expects 2 requests for 100 items

Real-world impact: For user following 100 people where 30% have exactly 100 items in some category:

• Current: ~391 API calls
• Expected: ~301 API calls
• Extra overhead: 90 unnecessary API calls (23% increase)

3. No Multi-Page Caching

• Impact: Minor cumulative
• Cause: Pages 2+ are new URLs with no prior cache entries

Solution Proposed

Use GitHub's Link header (which explicitly indicates rel="next") instead of guessing based on result count.

Files Added

• PAGINATION_ANALYSIS.md - Complete technical analysis with test scenarios, impact tables, and comparison charts
• PAGINATION_FIX_POC.md - Full implementation guide showing exactly how to fix issue Add launchd plist for daily scheduling #2

Recommendation

Implement the Link header solution to eliminate the extra API calls. The fix is straightforward and uses GitHub's standard pagination mechanism.

Related

Follows up on #33