Support binary input mode for producing from stdin

[alexleigh]

Suppose we have a program that produces some kind of binary output. We would like to be able to pipe the output of this program to kafkacat so that the binary output is created as a single message to kafka. However when kafkacat reads from stdin it assumes the input is delimited, and thus may split the input into multiple messages no matter what delimiter we choose. We can work around this by saving the output of the program to a file first and have kafkacat read from the file, but this requires an extra roundtrip to a file on disk. To avoid this extra step I'd like to propose a new binary mode for reading from stdin that treats the entire input as one message. Using this mode we can pipe the output directly to kafka using create-binary-output | kafkacat -B. Hopefully the idea seems sound to you. If you decide to incorporate something like this we can change the name of the switch, its description, etc. to your liking.

[edenhill]

It is possible to do this with process substition in bash:

$ kafkacat -b .. -P ... <(my_binary_emitter_cmd)

[alexleigh]

I didn't know about this bash feature. Thanks! That's very helpful to know. However, if I run the following on OS X 10.13.3 kafkacat -P -b ... -t ... <(echo hello) I get % Failed to mmap /dev/fd/63: Invalid argument. Could this be a known issue?

[edenhill]

It probably can't map these type of pipe files, kafkacat should fall back on non-mmap reads in this case (but that's a feature to be added!)

[alexleigh]

Since kafkacat doesn't currently support reading in pipe files, do you think my approach of reading from stdin without delimiters make sense? Or would you rather I try to add the non-mmap read fallback? My C is a bit rusty, but I suspect the two approaches would be rather similar, involving read and realloc in a loop.