Skip to content

Conversation

@level09
Copy link

@level09 level09 commented Mar 7, 2025

Problem

The _slurp_changelogs in faust/faust/tables/recovery.py method in the recovery process doesn't properly handle None events that can occasionally be received from the changelog queue. When a None event is encountered, the code attempts to process it, which leads to errors since None doesn't have the expected attributes and methods.

Solution

This PR adds a simple check at the beginning of event processing to detect and skip None events. The implementation:

  1. Silently checks if the received event is None
  2. For None events, checks if there are any active partitions or tables to recover
  3. If there's nothing to recover, signals recovery end and sleeps to avoid a busy loop
  4. Otherwise, skips processing the None event and continues to the next iteration

Benefits

  • Prevents errors when None events are encountered
  • Avoids busy loops in the recovery process when there's nothing to recover
  • Maintains clean logs in production by handling edge cases silently
  • Improves overall stability of the recovery process

Testing

Tested in a production environment with Kafka where None events were occasionally being received from the changelog queue. The fix successfully prevents errors and maintains stable operation.

Adds proper handling for None events in the changelog recovery process to
prevent processing attempts on None events from the changelog queue.
The fix silently skips None events and avoids busy loops when there are no
active partitions or tables to recover.

Improves stability of the recovery process by gracefully handling edge cases
without adding unnecessary logging in production environments.
@codecov
Copy link

codecov bot commented Mar 8, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.09%. Comparing base (ff75c0b) to head (00b5b0e).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #657   +/-   ##
=======================================
  Coverage   94.09%   94.09%           
=======================================
  Files         102      102           
  Lines       11100    11100           
  Branches     1196     1196           
=======================================
  Hits        10444    10444           
  Misses        557      557           
  Partials       99       99           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Remove trailing whitespace and fix indentation to resolve CI check failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant