⚡️ Speed up method _PreChunkAccumulator.will_fit by 38%

[codeflash-ai]

📄 38% (0.38x) speedup for _PreChunkAccumulator.will_fit in unstructured/chunking/base.py

⏱️ Runtime : 458 nanoseconds → 333 nanoseconds (best of 38 runs)

📝 Explanation and details

The optimization replaces an expensive string materialization operation with a direct length calculation method.

Key Change: The original code calls len(self.combine(pre_chunk)._text) which creates a full combined text string just to measure its length. The optimized version introduces _combined_text_length() that calculates the same length without building the actual string.

Why It's Faster: String concatenation and materialization in Python is expensive, especially for larger text chunks. The new method:

• Iterates through elements once to sum their text lengths
• Adds separator lengths mathematically
• Avoids allocating memory for the combined string
• Reduces garbage collection pressure

Performance Impact: The line profiler shows the critical line in can_combine dropped from 173,000ns to 100,000ns (42% improvement), contributing to the overall 37% speedup. This optimization is particularly effective for:

• Larger text chunks where string operations dominate
• Frequent combination checks during chunking workflows
• Memory-constrained environments

Behavioral Preservation: The optimization maintains identical logic and return values - it's purely an implementation efficiency gain without changing the chunking behavior or API contract.

This type of optimization is especially valuable in text processing pipelines where chunking operations may be called thousands of times on large documents.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	✅ 143 Passed
🌀 Generated Regression Tests	🔘 None Found
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	✅ 3 Passed
📊 Tests Coverage	100.0%

⚙️ Existing Unit Tests and Runtime

🔎 Concolic Coverage Tests and Runtime

Test File::Test Function	Original ⏱️	Optimized ⏱️	Speedup
codeflash_concolic_e8goshnj/tmpayrn317z/test_concolic_coverage.py::test__PreChunkAccumulator_will_fit	458ns	333ns	37.5%✅

To edit these changes git checkout codeflash/optimize-_PreChunkAccumulator.will_fit-mjdrcu8s and push.