Message framing for zcm?

[DanielArnett]

TL;DR- Given a serial array containing 10 message-types (e.g. a Lat/Lon message, a compass message, and an IMU message), how do you know which bytes correlate with which zcm-defined message types?

@jbendes I love zcm (and the Grand Challenge heritage), thank you for the fantastic documentation. My questions relate to transport on embedded systems. The documentation has made it clear that zcm can effectively serialize/deserialize. My confusion is around how zcm knows which defined message to use when deserializing. Let's say I have a file stored on a hard drive saving 10 different zcm message types in serial: how can I load those messages? Or I have a radio link transmitting/receiving 10 different zcm message types in serial: how do I know which messages to parse, or how to reject noise? I'm aware I may be asking the wrong questions for my problems, that's why I'm reaching out. I feel like there are well-established methods to solve these problems that I'm missing.

I've made home-brewed solutions, but I'm interested in learning the right way, rather than my way. As zcm is built with embedded systems in mind, I was wondering if you've ever done any kind of message framing for transmitting over serial connections.

I can give one example that solves this problem. In Mavlink for instance when the serialized message is sent over a Serial or UDP connection it frames all packets with this kind of structure:

Using Mavlink1 messages for a simple example of message framing.

Let's pretend I've integrated zcm with mavlink, and I'm using Mavlink's framing structure to send a latitude/longitude I've saved in latlon_t.zcm:

struct latlon_t {
    int32_t lat;
    int32_t lon;
}

Here's an example packet, assuming I created a mavlink message with MSG_ID 0xFE=254 with the above latlon_t:

STX	LEN	SEQ	SYS ID	COMP ID	MSG ID	PAYLOAD	CHECKSUM
0xFE	0x08	0x07	0x03	0x02	0xFE	0x 65 AF 40 18 68 B0 DD D3	0x 04 34

Here's how a device receiving this message might process it.

• STX- "This byte matches the Start-of-text: "0xFE", this must be a mavlink message."
• LEN- (assuming 8) "This payload will be 8 bytes long."
• SEQ- "Hmm, the last sequence number was 5, but this sequence number is 7, I must have missed a packet."
• SYS ID- "This message is coming from vehicle 3."
• COMP ID- "This message is coming from component 2 on vehicle 3."
• MSG ID- "Hey this message ID 0xFE matches the one I use for a zcm latlon_t message"
• PAYLOAD- "Let's unpack the payload to a latlon_t. 8 bytes total, 4 for the lat, and 4 for the lon."
• CHECKSUM- "The CRC bytes match so I'm 99.998% sure the message is as-sent."

Message framing is valuable when transmitting data over a serial link, as well as when saving/loading data to/from a serial flash memory.

• Has zcm ever been integrated with any libraries that perform serialization?
• Am I asking the wrong questions, and do you know of other ways to solve the problem of identifying messages and ensuring accuracy in serial?

Thank you for reading.

---

IssueHunt Summary

Backers (Total: $20.00)

• danielarnett ($20.00)

Submitted pull Requests

• #555 docs: describe generic serial transport framing

---

Become a backer now!

Or submit a pull request to get the deposits!

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[issuehunt-oss]

@danielarnett has funded $20.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[DanielArnett]

I didn't realize IssueHunt required a PR for funds to transfer, I just wanted to donate $20 to the project. Feel free to make a PR to some readme to complete the transfer, or let me know if there's something I can do on my side.

[jbendes]

Hey Daniel! Thanks so much for the enthusiasm! I'm glad you like the project. We love it and it has transferred trillions of events for people in all different use cases. As for embedded support, I think you're probably looking for serial:// transport that zcm provides. This transport handles serialization of zcm onto a linux serial port - handling framing, checksums, synchronization, and message type version checking. That transport relies on a file called generic_serial_transport.{h,c} in the zcm repo if you're interested in the inner workings. This file is then also used on embedded systems and you can take a look at an example of that if you fetch the third party transports submodule of zcm by doing git submodule update --init --recursive, you'll see a new folder zcm/transport/third-party that has a couple examples of implementing a serial transport on a bare metal embedded system. Let me know if you have any more questions!

[DanielArnett]

Ah, yes generic_serial_transport.c does describe the framing!

// Framing (size = 9 + chan_len + data_len)
//   0xCC
//   0x00
//   chan_len
//   data_len  (4 bytes)
//   *chan
//   *data
//   sum1(*chan, *data)
//   sum2(*chan, *data)

To continue my above example table of a lat/lon message. I'll assume the channel name is "latlon", and the data is still 4 bytes for each number.

STX	CHAN_LEN	DATA_LEN	CHAN	DATA	CHECKSUM
0x CC 00	0x06	0x08	"latlon"	0x 65 AF 40 18 68 B0 DD D3	0x 04 34

[DavideZanonArt]

Hi! I submitted a small docs PR clarifying the default serial:// generic serial framing (wire format, escaping, checksum): #555

GitHub shows one workflow run awaiting maintainer approval on that PR — if someone can approve it (and review/merge if it looks good), that would help finalize the IssueHunt-funded transfer for this issue. Thanks!