crashes process (memory probably on modest pdf)

[meirqed]

Describe the bug

Fetched the following paper from cell: https://www.cell.com/action/showPdf?pii=S0092-8674%2819%2930277-6
It's open access.
It crashes the process, memory metrics passed 6GB before crashing. r

Code to reproduce the problem

        text = []
        with pdfplumber.open(pdf_path) as pdf:
            for page_plumb in pdf.pages:
                page_text = page_plumb.extract_text()
                if page_text:
                    text.append(page_text)
        extracted_text = "\n".join(text)

PDF file

Please attach any PDFs necessary to reproduce the problem.
https://www.cell.com/action/showPdf?pii=S0092-8674%2819%2930277-6

Expected behavior

Not crash, on a 40MB pdf
Be able to extract text from a 40MB pdf with less than 6GB of RAM

Actual behavior

Crashed, presumbly due to memory consumption

Environment

• pdfplumber version: 0.11.6
• Python version: 3.10
• OS: Linux (Ubuntu and Amazon linux)

[NebelFox]

I've encountered a similar issue. On a small PDF (1.2 MB) a script that reads the file and forces each page to be loaded and parsed, like accessing page.images, the memory usage reached 700 MB. And the usage remains on this level even after the with pdfplumber.open block exited.

To reproduce:

import pdfplumber

with pdfplumber.open(PDF_PATH) as file:
    for page in file.pages:
        print(f'Page {page.page_number}')
        page.images

input("Press any key to exit...")

I cannot share the PDF I've used, but it contains only rect, curve and char objects (according to file.objects.keys())

During the for loop the memory usage constantly increases, and it remains on the reached peak level until the script is exited.

I've examined the source code, specifically the PDF.close method:

pdfplumber/pdfplumber/pdf.py

Lines 124 to 131 in 6ebc549

	def close(self) -> None:
	self.flush_cache()
	for page in self.pages:
	page.close()
	if not self.stream_is_external:
	self.stream.close()

The memory consumption behavior remains the same if the pdfplumber.open is used without with block and without closing the file whatsoever:

import pdfplumber

file = pdfplumber.open(PDF_PATH)
for page in file.pages:
    print(f'Page {page.page_number}')
    page.images

input("Press any key to exit...")

Adding file.close() after the loop doesn't change anything as well. Hence, I suggest the close method doesn't work properly.

The flush_cache() method called first in close() seems to delete cached ._pages list. But the next for page in self.pages: page.close() seemed like it would reload the pages again. So I've modified the source code to indicate whenever the pages are loaded in pages property (just a print in case there is no ._pages property yet). With any snipped from this message, the print message appears twice -- first when opening the file, and second when closing it.

It happens because flush_cache() deletes all the Container.cached_properties attributes plus _pages for PDF class:

pdfplumber/pdfplumber/pdf.py

Line 26 in 6ebc549

cached_properties: List[str] = Container.cached_properties + ["_pages"]

pdfplumber/pdfplumber/container.py

Lines 28 to 32 in 6ebc549

	def flush_cache(self, properties: Optional[List[str]] = None) -> None:
	props = self.cached_properties if properties is None else properties
	for p in props:
	if hasattr(self, p):
	delattr(self, p)

So the PDF.close() method first deletes the _pages attribute, and then accesses it in for page in self.pages, which in turn sees no _pages property and parses the pages again. These pages are closed via Page.close(), but ._pages attribute is not deleted this time.

I've played a bit with the PDF.close() method and found that doing for page in self.pages: page.close() before self.flush_cache() reduces the memory usage from ~700 MB to ~200 MB after PDF.close() call, but still not to the starting 3 MB

A temporary solution I've found for myself is to call Page.close() manually after I process it. In my use case I iterate over the pdf pages only once, and I don't need past pages, like:

def process_pdf(path):
    with pdfplumber.open(path) as pdf:
        for page in pdf.pages:
            process_pdf_page(page)
            page.close()

This keeps the memory consumption near 40 MB during the whole execution of the code block and afterwards. Running the code multiple times still keeps the memory usage under 40 MB. Does pdfplumber cache something globally on the first invocation? Or any of its dependencies? This doesn't bring the RAM down to starting 3 MB, but still better than nothing and such approach allows to work with pdf files in somewhat memory-predictable and manageable manner.

Tried on a larger file (5.1 MB). Memory usage after closing the file is ~ 50 MB. Opening another file afterwards drops the RAM to ~ 40 MB initially before growing it as the file is read and keeping it at ~ 50 MB again.
It might be just Python actually clearing the memory but not releasing it yet for quicker further use, but I'm not sure about that.

UPD 1

Calling page.flush_cache() instead of page.close() provides a similar effect on the memory usage