Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion docs/streams/stream-basics/producers.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,28 @@ Messages published by producers can be compressed during transportation in order

## Batching

If batching is enabled, then the producer accumulates and sends a batch of messages in a single request. Batching size is defined by the maximum number of messages and maximum publish latency.
If batching is enabled, then the producer accumulates and sends a batch of messages in a single request. Batching size is defined by the maximum number of messages and maximum publish latency.

## Chunking

Message chunking enables Macrometa to process large payload messages by splitting the message into chunks at the producer side and aggregating chunked messages at the consumer side.

With message chunking enabled, when the size of a message exceeds the allowed maximum payload size (5MB), the producer splits the original message into chunked messages and publishes them. The consumers automatically consume the aggregated message.

:::note

- Chunking is only available for persistent topics.
- Chunking cannot be enabled simultaneously with batching. Before enabling chunking, you must turn off batching.

:::


### Enable Message Chunking

**Prerequisite**: Turn off batching by setting the `batching_enabled` parameter to `false`.

The message chunking feature is off by default. To enable message chunking, set the `chunkingEnabled` parameter to true when creating a producer.

:::note
If the consumer fails to receive all chunks of a message within a specified period, then it expires incomplete chunks. The default value is 1 minute.
:::
1 change: 0 additions & 1 deletion docs/streams/stream-tasks/create-producers.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,4 +143,3 @@ create_producer()

</TabItem>
</Tabs>

Binary file added static/img/streams/chunking1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added static/img/streams/chunking2.png
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do these M1-C3 labels mean? Can we annotate these images more clearly?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a clue, but we will probably need to redo all images anyway. I wouldn't worry about it until we actually support chunking on the user end.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.