Itak datapackages containing attachments upload fails

[skadakar]

Appears itak is using a different method of sending/fetching datapackages with attachements.

[05/Feb/2023:13:29:38 +0000] "POST /Marti/sync/upload?CreatorUid=1874F828-5CF1-4289-B6C0-D4F6ABFB0B4D&keywords=missionpackage&name=FM05-iOS.05.142741.zip HTTP/1.1" 404 207 "iTAK/602 CFNetwork/1335.0.3 Darwin/21.6.0"

Datapackages with just markers route and work as expected.

[Matthew-Beckett]

This is super wild, this looks to be caused by an issue actually in the ITAK app it's self and as it's undocumented it's likely going to prove most difficult to find out why they use this endpoint and where they got the documentation from.

This endpoint is not present anywhere, neither on FreeTakServer or in the ATAK application code.

I simply have no idea where the ITAK developers got this information from. :(

[tkuester]

@Matthew-Beckett - You've saved me a ton of time having to look that up, thank you.

Regrettably, I don't have an iTAK / iOS device to test this on, but if any users are able to help contribute to the development, it might be nice to improve iTAK support a little.

Just as a brief aside on the philosophy of taky here... There are two principles of software design that stick in my mind.

1. Be permissive in what you receive, and strict in what you send out. (Protocol design)
2. Don't break userspace. (Linux Kernel)

iTAK seems to really have gone off the rails on a few things, which I don't appreciate as a developer -- but then again, I'm not really in a place to police standards for CoT/TAK. What's more, one of the features of CoT/TAK is that the standards are very lenient, allowing for rapid development and prototyping... so it doesn't seem right to refuse to implement iTAK's data package endpoint because they made their own. On principle, iTAK is a significant contribution to the TAK ecosystem, and the end users shouldn't have to be treated as second class citizens.

I'm hesitant to add too much chaos to the codebase, but it really does seem like I should make some attempt to improve support for iTAK in this area. If anyone is willing to help contribute to development, I'd love pull requests, DM's, or issues posted here!

[Matthew-Beckett]

@Matthew-Beckett - You've saved me a ton of time having to look that up, thank you.

Regrettably, I don't have an iTAK / iOS device to test this on, but if any users are able to help contribute to the development, it might be nice to improve iTAK support a little.

Just as a brief aside on the philosophy of taky here... There are two principles of software design that stick in my mind.

1. Be permissive in what you receive, and strict in what you send out. (Protocol design)

2. Don't break userspace. (Linux Kernel)

iTAK seems to really have gone off the rails on a few things, which I don't appreciate as a developer -- but then again, I'm not really in a place to police standards for CoT/TAK. What's more, one of the features of CoT/TAK is that the standards are very lenient, allowing for rapid development and prototyping... so it doesn't seem right to refuse to implement iTAK's data package endpoint because they made their own. On principle, iTAK is a significant contribution to the TAK ecosystem, and the end users shouldn't have to be treated as second class citizens.

I'm hesitant to add too much chaos to the codebase, but it really does seem like I should make some attempt to improve support for iTAK in this area. If anyone is willing to help contribute to development, I'd love pull requests, DM's, or issues posted here!

Hey @tkuester, I've spend a bit more time on this attempting to re-create this with my iPhone and implement some semblance of support for this feature but with minimal success. It's very hard to implement this with the iOS app being as opaque as it is.

As such, I have reached out directly to the TAK Product Centre team asking for specific documentation on Datapackages in iTAK and why they are seemingly implemented differently to in ATAK.

Hopefully we get a positive reply from them, I observed in the latest TAK Meetup notes that there was quite a considerable drive within the TAK Product Centre to open-source more components and conduct more work in public.

So hopefully open-sourcing iTAK will be one of the first steps in to that endeavour. Fingers crossed!

[Matthew-Beckett]

Bit more progress on this, I have gotten a response back but the answer is not very helpful. Spending more time trying to reverse engineer this endpoint I am noticing that the data is sent as a bytes in the data and not as a file which is wildly different from Missionpacks on ATAK.

Continuing to look into this, but just a bit of info on how it's going.

-- UPDATE --

The implementation for the iOS application is sending the datapackage ZIP file as a series of bytes in the data of the POST request. I have successfully captured and output this to disk and been able to unzip it and view the datapackage contents verifying it's correct.

The issue I now have is that the arguments sent in the query parameters are very different from the ATAK ones, namely no hash is sent and instead a UID is sent from the client.

I could do with a little more input from you @tkuester on how we implement this endpoint whilst also maintaining parity with Android clients.

Is there somewhere can engage in some more real-time discussion around this such as Slack?

Here are the parameters send and what I think they correlate to:

name=Mission1 - Datapackage name
uid=e6ae7640-47d2-40fb-a428-1a34246fa0f1 - I think this is replacing hash?
keywords=missionpackage - No idea what this is, maybe some evolution of datapackages to expand info on what's in them?
CreatorUid=EEE15D20-9189-44D6-83A7-32BD032657E5 - UID of creating user
tool=public - No idea, think this is privacy?

[Matthew-Beckett]

@skadakar can you confirm if listing datapackages works for you?

[Matthew-Beckett]

Hey guys,

Another progress update on this issue, I have successfully tested what I will now be calling "Enterprise Sync" with the mainline DoD Server implementation on iOS.

All functionality for iTAK works in the approved public version of DoD's TAK Server. However, for some reason, although the built binaries are approved for public release the DoD TAK Server repo is way behind what the current version of TAK is, and I have not been able to start the code from that repo as it's a total mess, so I don't know if the Enterprise Sync code in the GitHub Repo for DoD's TAK Server is complete and working.

In my investigation I have attempted to create as close an implementation to Enterprise Sync as is possible with the publicly available information and I still cannot get iTAK to return a success response or get mission package listing working.

This is a positive because we know it can, and ultimately should work, this means we can fix this. However, until the code in the DoD's GitHub repository is updated, this will prove next to impossible.

Maybe someone intimately acquainted with Java and Spring might have a better chance than I have, but I think for now this is a dead-end.