The summarising functionality for continuous variables in `output/sc2egset_replaypack_processor/XXX/XXX_summary.json` is weird

[martinmodrak]

A very minor suggestion - the summary json correctly summarises discrete variables like matchups by counts of the individual categories. However, it does the same with continuous variables like gameTimes, where each value is unique, so the JSON has stuff like:

"gameTimes":{"1000.044643":1,"1006.339286":1,"1010.580357":1,"1023.303571":1,"1028.169643":1,"1032.946429":1,"1033.973214":1,"1041.741071":1,"1046.562500":1,"1064.196429":1,"1081.651786":1,"1084.151786":1,"1105.580357":1,"1149.732143":1,"1157.946429":1,"1159.241071":1,"1195.892857":1,"1196.428571":1,"1196.473214":1,"1200.491071":1,"1206.696429":1,"1225.982143":1,"1229.062500":1,"1255.178571":1,"1272.187500":1,"1272.589286":1,"1279.062500":1,"1282.991071":1,"1332.857143":1,"1381.116071":1,"1402.723214":1,"141.517857":1,"1422.812500":1,"1428.080357":1,"1434.017857":1,"1447.678571":1,"1453.348214":1,"1506.607143":1,"1506.651786":1,"1533.035714":1,"1554.866071":1,"1567.232143":1,"1645.089286":1,"1686.517857":1,"1689.107143":1,"1696.383929":1,"1705.267857":1,"1737.544643":1,"1810.267857":1,"191.339286":1,
....

It would probably make more sense to summarise with something like mean, sd and range (or just not summarise at all).

[Kaszanas]

Hi @martinmodrak

This kind of summary was done to have the ability to create histograms without having to touch all of the parsed replay files once more. Just mean and standard deviation would not be enough.

This kind of raw data representation helps with that.

[martinmodrak]

It is not a big deal, just noting that it is weird. If full distribution is desirable, why not just store it as an array?

Or have a data frame with the main info per game (would avoid explicitly needing to store times per matchup etc...)

[Kaszanas]

It is not a big deal, just noting that it is weird. If full distribution is desirable, why not just store it as an array?

This is an interesting question, as it was initially intended for histograms, the easiest format was having the time and associated count to immediately generate such plots. An array itself would have to be turned into a counter with bins anyway before plotting a histogram.

You are right that the summarization logic could be removed to simplify the process, as all of this can be computed from the data itself. (not without unpacking the batches though). But since it works and produces valid outputs I left it there.

Or have a data frame with the main info per game (would avoid explicitly needing to store times per matchup etc...)

Once again, interesting approach, I was inclined to solve this with the simplest means available. The main goal of such summaries was purely to get an idea about the package contents. Full analyses would have to be conducted on the unpacked data anyway.

Given that the data was extracted using Golang (not python), marshalling structs into JSON around 2020 when I first began working on this software was the easiest solution. Packaging itself was introduced because I have access to a major dataset of 2 million replays which I wanted to process in batches with built in compression (the final dataset is 3 TB) - not yet released. Hence the package summaries with their simplistic nature.

Should I turn this into an actionable item to adjust SC2InfoExtractorGo? https://github.com/Kaszanas/SC2InfoExtractorGo

[martinmodrak]

It is IMHO definitely not necessary for acceptance into JOSS. Just a suggestion and something to keep in mind if you happen to touch the code sometime in the future....