feat(shared): exponential backoff retry and graceful degradation

[zong0728]

Summary

Fixes #160

milvus_search() silently swallows all exceptions and returns {"results": []},
preventing retry on transient failures and causing the LLM to hallucinate from
empty context instead of informing the user the service is unavailable.

Root Cause

Both servers wrap the entire Milvus operation in a broad try/except that
discards the error and returns a fake success:

except Exception as e:
    print(f"[ERROR] Milvus search failed: {e}")
    return {"results": []}   # exception swallowed, caller sees no failure

Because the exception never propagates, there is no opportunity to retry. execute_tool receives an empty result set and passes it to the LLM as if the search had succeeded.

Failure Scenario

User asks a Kubeflow-specific question
LLM correctly calls search_kubeflow_docs
Milvus pod is restarting (transient unavailability)
milvus_search fails → exception caught → returns {"results": []} ❌
execute_tool sends "No relevant results found." to the LLM
LLM generates a hallucinated answer — user never knows the search failed

Fix

shared/retry.py: reusable @with_retry decorator (sync + async) using AWS full-jitter exponential backoff (sleep = random.uniform(0, min(cap, base * 2^n))); exposes DEGRADED_RESULT sentinel string for LLM-visible outage messages
milvus_search: remove silent exception swallow; decorated with @with_retry(max_attempts=3, base_delay=1s, backoff_factor=2x, jitter=True); encoder loaded once at module level via _get_encoder() singleton
execute_tool: offload blocking search to asyncio.to_thread (WebSocket server) and run_in_threadpool (FastAPI server); return DEGRADED_RESULT on retry exhaustion so the LLM communicates the outage to the user

Testing

• Python syntax verified

• Simulated Milvus failure triggers 3 retry attempts with increasing delays

• DEGRADED_RESULT returned to LLM after retry exhaustion

• Single-turn happy-path behaviour unchanged

Checklist

• Commits are signed off (DCO)

• Fixes #

• Implements GSoC 2026 spec Requirement Https support and updated Readme #5 (exponential backoff + graceful degradation)

• No regressions to single-turn query paths

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign franciscojavierarceo for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment