“Missing dataset files and external dependencies in DeepRare pipeline”

[kenbougoule]

Hello,

I am currently trying to run your DeepRare pipeline as part of an academic project (L2 MIASHS).

The code runs successfully up to the data loading stage, but I am facing blocking issues that prevent the full execution of the pipeline.

Main issue

The program expects several dataset files that are not available in the repository:

dataset/xinhua_rag_0331.csv
dataset/mimic_rag.csv
dataset/rarebench_rag.csv
dataset/mygene_rag.csv
dataset/ddd_rag.csv

In addition, some external references such as:

chenxz/RareBench

are not accessible, which causes the pipeline to stop during execution.

Errors encountered
FileNotFoundError (missing dataset paths)
KeyError: '_id' in similarity matching step
ConnectTimeoutError when calling external APIs (Phenobrain service)
Some data parsing issues due to inconsistent formats
What I understand so far

The pipeline structure itself seems correct:

model loads successfully
embeddings are loaded
patient extraction starts correctly (I can see the total number of patients being processed)

However, execution stops when accessing missing datasets and external services.

Question

Could you please clarify:

Where can I obtain the missing dataset files?
Is there a simplified or demo version of the pipeline that does not rely on external APIs?
Is the repository fully reproducible as it is?

Thank you very much for your help.

[alexadamus77-ui]

Debugging broken RAG pipelines due to missing datasets and external API timeouts is easily the most frustrating part of setting up academic data ingestion layers.

Since Deep Rare relies heavily on parsing benchmarking data like Rare Bench and processing complex clinical/scholarly literature, you might want to look into ScholarAPI for your next pipeline iteration.

It serves pristine, pre-structured JSON endpoints for open-access literature, metadata, and direct full-text extractions. Integrating it can eliminate those unpredictable Connect Timeout Errors and data parsing issues caused by inconsistent document formats, giving your embedding models clean tokens to match against right out of the box.