Skip to content

Add FlashAttention startup fallback#56

Open
mindbomber wants to merge 1 commit into
NousResearch:mainfrom
mindbomber:codex/flash-attn-startup-hint
Open

Add FlashAttention startup fallback#56
mindbomber wants to merge 1 commit into
NousResearch:mainfrom
mindbomber:codex/flash-attn-startup-hint

Conversation

@mindbomber

Copy link
Copy Markdown

Fixes #31

Summary

  • Add --attn_implementation to functioncall.py and jsonmode.py so users can run with eager or sdpa when FlashAttention is incompatible.
  • Wrap FlashAttention model-load failures with an actionable message that explains the PyTorch/CUDA wheel mismatch and the reinstall workaround.
  • Lazily import the heavy runtime dependencies so --help works before the full GPU/model stack is installed.
  • Document the FlashAttention startup failure and fallback command in the README.

Verification

  • python -m py_compile functioncall.py jsonmode.py -> passed
  • python functioncall.py --help -> passed
  • python jsonmode.py --help -> passed
  • python -m ruff check functioncall.py jsonmode.py -> passed
  • python -m ruff format --check functioncall.py jsonmode.py -> passed
  • git diff --check -> passed
  • AANA code-change guardrail -> pass / accept, AIx 1.0, no violations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can't start function calling script on a linux machine

1 participant