Skip to content

add requests download backend#2

Open
RibomBalt wants to merge 1 commit intomjrutala:mainfrom
RibomBalt:main
Open

add requests download backend#2
RibomBalt wants to merge 1 commit intomjrutala:mainfrom
RibomBalt:main

Conversation

@RibomBalt
Copy link

Hello,

I'm new to SPICE, and I'm learning how to use it by playing with codes on github, and I found your repository helpful.

I noticed that you are using wget to download NAIF data. It surely can do the work, but I noticed it doing wield things, including treating Name, Last modified as downloadable files, downloading multiple version of robots.txt (and afterwards renaming it to robots.txt.1, robots.txt.2 too). And as #1 mentioned, the dependence on wget maybe problematic for Windows users. I also noticed the commented out urllib section, I'd assume you plan to at least add an alternative to wget with more native ways.

So I ended up doing with my own way, using requests to first get page html, get all <a> tags, and with all filenames that match the pattern, we try to request the server. force-update is achieved with a If-Modified-Since header, and if the server returns 304 then our file is newer than the server, so no redownload would trigger. tqdm is used to render the progress bar in console, and a null context is used when show_progress=False.

Besides I also add a requirements.txt and a unittest module (by which I can confirm the new code is working as expected). I used logging module to print debug infos, but they would be silenced unless explicitly setting the log level to INFO.

I'm currently using this version now, and I'm happy to contribute. Feel free to merge / not merge this PR, and feel free to let me know if further changes are needed.

- add requirements.txt
- add unittest
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant