Add a split command (or option)

[f6fvy]

Hi

The LoTW service currently shows some problems when a large log is uploaded (several tens of thousands of QSOs). It is necessary to split the original log into several logs of a few hundred or thousand QSOs and upload them one by one so that all the QSOs are finally taken into account.

It would be great if your (very comprehensive) utility could perform this split operation, with a command such as

adifmt split --count=1000 log.adi

and generate log_0001.adi, log_0002.adi etc. files of 1000 QSOs each (except the last one).

Or it could be included as a new option for all existing commands (much more universal, but perhaps more difficult to code?).

73

Larry - F6FVY

[flwyd]

Thanks for the idea, Larry. There's already a feature that does something similar: the save command will treat filenames as a template and substitute field names inside curly braces for actual field values from the input, and create a separate file for each combination. For example, my POTA workflow finishes with adifmt save '{station_callsign}@{my_pota_ref}-{qso_date}.adi'. This allows converting a single ADIF file that covers multiple parks or multiple days into separate files for upload, e.g. WT0RJ@US-4564-20250704.adi plus WT0RJ@US-4567-20250704.adi for a 2-fer, since the POTA site wants each park in a separate log.

So if you have a field that has enough (but not too many) matching records, you can use this technique. If you've got a log file covering multiple years, splitting by QSO_DATE would probably be overwhelming, but maybe you could split by BAND or DXCC or STATE. But your largest file might still be too large if your operations are skewed to particular values, e.g. you mostly make 20m contacts with US stations. If you'd like to check whether a particular field would get logs of a sensible size, try adifmt count --output tsv --fields band mylog.adi.

I've thought about adding some kind of function capability in expressions, e.g. YEAR(QSO_DATE) would result in just the year value from the date. This would help create a natural split between files saved with the template adifmt save mylog_{YEAR(QSO_DATE)}.adi resulting in one file per year, which might be enough to get an "all time log" split into small enough pieces for a LotW upload.

Incidentally, if you're able to write a short script you can do the year trick (or break a log into months) by filtering by a date range. Something like this:

for (( year=2020 ; year < 2026 ; year+=1 )) do
  echo "Filtering log for $year"
  adifmt find --if "qso_date>=${year}0101" --if "qso_date<$((year + 1))0101" mylog.adi \
    | adifmt save "mylog_${year}.adi"
done

However, a dedicated command or an option to save probably makes this simpler. Perhaps adifmt save --shard-record-count=1000 mylog.adi which produces mylog_1-of-5.adi, mylog_2-of-5.adi, etc. I'll think about a structure and naming that makes sense.

(Incidentally, this feature can only work on the save command or a new command which can write multiple files; all the other commands write to standard output, which is a single stream with all the content. But I think it's only relevant at save time, otherwise you can just filter out records you don't want.)

[f6fvy]

Thanks Trevor for your quick reply and suggestions.

As you say, this can apply to logs with “not too many” QSOs per day, but there are sometimes cases of DXpedition logs where they make several thousand QSOs per day, because several stations are active in different modes and/or bands at the same time. And they face this problem of LoTW not accepting all QSOs in a single upload.

In any case, your tool is really very flexible and opens up many new perspectives when it comes to “triturating” ADI logs. Well done !

73 !

[flwyd]

I have a working implementation of this feature, but still need to add some tests. It'll be a few weeks before I get time to complete it, though.

[flwyd]

My summer had an unexpected challenge; I won't have this done until mid- or late-September.