Performance Bug: SentenceTransformer is re-initialized on every Milvus search request

[RajSidwadkar]

Bug Description

In server/app.py, the SentenceTransformer(EMBEDDING_MODEL) is instantiated inside the milvus_search() function. This causes the application to load the entire embedding model into memory on every single tool call, resulting in severe latency spikes (multiple seconds per search) and risking OOM crashes under concurrent WebSocket connections.

Proposed Solution

Move the SentenceTransformer initialization to the global scope so it acts as a singleton loaded once at server startup. The milvus_search function should reference this global instance.

I have tested this locally and it drastically reduces tool execution latency. I will open a PR with the fix shortly.

[eli-shiq]

Hi, I’m interested in contributing to this issue. I noticed there are already PRs linked — is there still room for contribution here, or would you recommend a related issue I could work on?

[Ayush-kathil]

Hi, I’ve submitted a PR (#186) addressing this issue by:

• Moving SentenceTransformer initialization to global scope
• Removing duplicate function definitions
• Fixing citation propagation in recursive calls

This improves performance and correctness of multi-hop queries.

Would appreciate feedback!

[RajSidwadkar]

Community Update : Backend Optimization & PR #129

Hey everyone , genuinely great to see so much momentum around the backend performance discussion. Wanted to give a proper update for everyone who's been following along.

---

What's in PR #129

Two long-standing pain points are addressed in a single, focused pull request:

1. SentenceTransformer Global Singleton - Resolved

The existing implementation was re-initializing the embedding model inside the milvus_search() function on every single call, causing multi-second I/O latency blocks. PR #129 refactors this by loading the model globally at server startup, dropping search latency to milliseconds and protecting the ASGI loop.

2. ContextWindowManager New

This is the bigger addition. The ContextWindowManager introduces a dedicated abstraction layer for token budget tracking and context truncation, directly addressing the HTTP 400/500 crashes caused by token-overflow when payloads silently exceeded API limits. The manager handles this gracefully across both APIs, with configurable overflow strategies rather than hard failures.

---

The Sliding Window - Looking for Collaborators

The current sliding window algorithm gets the job done, but there's real room to push it further. A few directions worth exploring:

• Smarter truncation heuristics : preserving semantic anchors (first turn, latest turn, high-salience spans) rather than naive oldest-first dropping
• Per-model token budget awareness : dynamic limits based on the active model rather than a static cap
• Async-safe context flushing : especially relevant for concurrent request patterns

If you've been thinking about any of these (or have a completely different angle), I'd love to collaborate. Drop ideas in the thread, or reach out directly , I'm very much open to co-authoring follow-up work.

---

Status: Waiting on maintainer review. In the meantime, feel free to review the diff, leave comments, or test against your own workloads.

Thanks for keeping this conversation going this kind of community engagement is exactly what makes the project better.

PR #129 author