From e18838a5242ea32506eb8373658ee1dcbc8fdc80 Mon Sep 17 00:00:00 2001
From: "codeflash-ai[bot]"
 <148906541+codeflash-ai[bot]@users.noreply.github.com>
Date: Sat, 20 Dec 2025 13:16:16 +0000
Subject: [PATCH] Optimize _PartitionerLoader._load_partitioner
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The optimization adds `@lru_cache(maxsize=128)` to the `dependency_exists` function, providing **266% speedup** by eliminating redundant dependency checks.

**Key optimization:** The original code repeatedly calls `importlib.import_module()` for the same dependency packages during partition loading. Looking at the line profiler results, `dependency_exists` was called 659 times and spent 97.9% of its time (9.33ms out of 9.53ms) in `importlib.import_module()`. The optimized version reduces this to just 1.27ms total time for dependency checks.

**Why this works:** `importlib.import_module()` is expensive because it performs filesystem operations, module compilation, and import resolution. With caching, subsequent calls for the same dependency name return immediately from memory rather than re-importing. The cache size of 128 is sufficient for typical use cases where the same few dependencies are checked repeatedly.

**Performance impact by test case:**
- **Massive gains** for scenarios with many dependencies: The test with 500 dependencies shows **7166% speedup** (1.73ms → 23.9μs)
- **Modest slowdowns** for single-call scenarios: 0-25% slower due to caching overhead
- **Best suited for:** Applications that load multiple partitioners or repeatedly validate the same dependencies

**Trade-offs:** Small memory overhead for the cache and slight performance penalty for first-time dependency checks, but these are negligible compared to the gains in repeated usage scenarios.
---
 unstructured/utils.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/unstructured/utils.py b/unstructured/utils.py
index f83f8831a2..8d6543e090 100644
--- a/unstructured/utils.py
+++ b/unstructured/utils.py
@@ -10,7 +10,7 @@
 import subprocess
 import tempfile
 import threading
-from functools import wraps
+from functools import lru_cache, wraps
 from itertools import combinations
 from typing import (
     TYPE_CHECKING,
@@ -227,6 +227,7 @@ async def wrapper_async(*args: _P.args, **kwargs: _P.kwargs):
     return decorator
 
 
+@lru_cache(maxsize=128)
 def dependency_exists(dependency: str):
     try:
         importlib.import_module(dependency)