Leaner task workers with automatic memory cleanup#1
Open
d3banjan wants to merge 14 commits intoshubhamdipt:masterfrom
Open
Leaner task workers with automatic memory cleanup#1d3banjan wants to merge 14 commits intoshubhamdipt:masterfrom
d3banjan wants to merge 14 commits intoshubhamdipt:masterfrom
Conversation
added 3 commits
March 28, 2025 17:11
Note: the parent's DB connections are closed before forking to avoid corrupting the db connections if the child process writes to them.
…or taskworkers (cherry picked from commit 334e504)
- Add psutil to requirements.txt (used in task_worker.py for memory logging but was never declared as a dependency) - Fix TypeError in error handler when task.output is None: if an exception occurs before output is initialized (e.g. during module import), the except block crashed on `None += str`, leaving the task stuck in "In Progress" forever with no error output Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Set up runtests.py with in-memory SQLite and test_tasks.py with helper callables so subsequent commits can include tests. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Move ManagedEventLoop and process_task (renamed execute_task) into worker.py. Add signals.py with 5 lifecycle signals (stubs). Add monitor.py with orphan detection stubs. Slim task_worker.py to orchestration only. Include tests for execute_task. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add three new fields to Task model: worker_pid for tracking which process owns a task, error for storing tracebacks separately from output, and log for captured process output. Update execute_task to write errors to task.error instead of task.output. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Set worker_pid on task claim, clear after subprocess completes. Implement detect_orphaned_tasks to find PROGRESS tasks with dead PIDs and mark them FAILED. Add handle_subprocess_exit to catch non-zero exit codes. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Emit before_job, on_success, on_failure, before_loop, and after_loop signals at appropriate points in the task execution flow. Generator tasks fire before_loop/after_loop per iteration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Capture stdout, stderr, and Python logging output from the child process via os.pipe(). Parent creates pipe before fork, drains it in a background thread during p.join(), then stores the captured log in task.log. The worker.py execute_task() redirects sys.stdout, sys.stderr, and root logger to the pipe fd when log_fd is provided. Also update runtests.py to use file-based SQLite with matching TEST name so subprocess tests can access the same database. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix XSS: replace inline HTML with Django template (auto-escaping) - Fix bare except clauses with specific exception types - Replace mark_safe with format_html in admin - Add optional task allowlist (DJANGO_SIMPLE_QUEUE_ALLOWED_TASKS) - Add task execution timeout (DJANGO_SIMPLE_QUEUE_TASK_TIMEOUT) - Add conf.py settings module - Add security and timeout tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Set up MkDocs Material with mkdocstrings to generate documentation from Google-style docstrings. Adds comprehensive docstrings to all modules (models, signals, monitor, views, worker, admin), 17 documentation pages covering getting started, guides, API reference, and advanced topics, plus a GitHub Actions workflow for automatic GitHub Pages deployment. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds the following features to the django-simple-queue library --
process_taskfunction now runs in a subprocess and cleans up after itself, by terminating the subprocess. This should ideally reload code changes without restarts and keep RAM usage from accumulating over time.