Timeout for certain tables (Regionaldatenbank)

[sjockers]

Requesting certain tables from regionsalstatistik.de takes several minutes, eventually causing a timeout.

I've encountered this with the following tables: 41120-12-01-4, 41141-03-01-4, 41141-03-02-4-B – but other tables may have the same issue.

# Requesting the table 41120-12-01-4 takes several minutes and will eventually cause a timeout 
t = pystatis.Table(name="41120-12-01-4") 
t.get_data()

Result:

ReadTimeout: HTTPSConnectionPool(host='[www.regionalstatistik.de](https://file+.vscode-resource.vscode-cdn.net/Users/sjockers/Projects/swr/agrar_regionalstatistik/www.regionalstatistik.de)', port=443): Read timed out. (read timeout=300)

Requests for other tables (from regionsalstatistik.de) run through and have much shorter request times, including some from the same statistics as the problematic ones, e.g. 41141-03-02-4, 41141-03-02-4-B.

[pmayd]

I am currently looking into the issue, but I have some general troubles with the Regionalstatistik API: Currently I am not able to make any request against the API, I am always getting a Code 6 in return telling me that I am executing too many parallel requests, which is not the case...

[pmayd]

@sjockers I investigated more and it only happens for fomat=ffcsv which is very bad for us as we only support this format. And the problem is not even the background job, actually the initial request against the tablefile endpoint never finishes for the ffcsv, so the API does not even return a response and also does not start a background job like in the past. So it seems like the whole system is broken currently, probably because they want to switch from background jobs to the archive format, but I don't understand why this happens without a new API version, so in general the whole API is very unstable right now and a lot of things do not work with Regionalstatistik....the csv download works, I tried it with Postman, but I don't think it makes much sense for us to also support this format as it requires a different approach in parsing...

[sjockers]

Thanks for looking into this, Michael! I agree that it would not make sense to support the CSV-format instead, as it would make parsing much harder (and would probably cause more issues down the line).

I have contacted regionalstatistik.de (IT.NRW) about the problem, lets see what they say.

[sjockers]

Update from regionalstatistik.de: This is apparently a known issue that concerns certain tables with complex headers: These tables cannot currently be rendered as ffcsv, causing the present bug.

Upgrading to a newer version of GENESIS should fix the problem, which will happen "in the coming months". There's no concrete date for the migration, though.

I'd suggest to ignore this bug until regionalstatistik.de is migrated to 'new' Genesis.

[LeSasse]

I am experiencing the same issue. Has there been any development on this or have you gained some further insight?

[pmayd]

Hi LeSasse, I will look into the issue in the next days! I think the problem is not with the package but with the Regionaldatenbank backend, but we will try to make the timeout faster...but as I have said further above, the problem is definitely on the database side, they don't even respond with a meaningful value for the very first request (either ffcsv or starting a background job). So if Regionaldatenbank will not fix this, our options are very limited. We would need to switch to another format like csv and build another parser for this filetype, which is currently not planned. I think the best is to wait for the next update of the database, then it should work like it already does for Genesis and Zensus

[pmayd]

This is so frustrating. Wanted to check where we stand with this issue but I cannot even download the table via direct API call, I just get a 500 "Ups ein Fehler" error message...