write to closed file

[dmd]

I very often am getting this:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/NDATools/Download.py", line 49, in run
    func(*args, **kargs)
  File "/opt/conda/lib/python3.11/site-packages/NDATools/Download.py", line 308, in write_to_download_progress_report_file
    download_progress_report.flush()
ValueError: write to closed file

If I run the program again, it works fine - usually.

[gregmagdits]

Are you running multiple instances of the downloader at the same time when you see this error?

[dmd]

Nope.

[dmd]

But I'm also failing to reproduce it any more, so maybe something was weird about my container. I'll reopen if it recurs.

[liningpan]

I think the reason is that this thread pool is not joined download_progress_file_writer_pool

nda-tools/NDATools/Download.py

Lines 358 to 361 in e4069d4

	download_progress_file_writer_pool.add_task(write_to_download_progress_report_file, download_record)
	download_pool = ThreadPool(self.thread_num, self.thread_num*6)
	download_progress_file_writer_pool = ThreadPool(1, 1000)

You should only close the file handle after joining the download_progress_file_writer_pool. Otherwise the file handle could be invalidated.

This also means that the download report may be incomplete

[gregmagdits]

yea, I think that makes sense. Thank you for pointing that out. Eventually I think we are going to use a sqllite db instead of writing to csv files, but this should be a quick fix that we can include in the next release. Thanks again