Send _bulk actions in batches#184
Conversation
| instructions = { _index: index_name, _id: id, retry_on_conflict: 3 } | ||
| instructions[:routing] = routing if routing | ||
| batch << { delete: instructions } | ||
| batch << [{ delete: instructions }] |
There was a problem hiding this comment.
index_document and update_document bulk actions require 2 lines each, but delete bulk actions only require 1. By grouping them together as they're added, we can paginate them correctly at HTTP request time
| expected_body = <<~DOC | ||
| {"index":{"_index":"widgets","_id":"5"}} | ||
| {"color":"green"} | ||
| {"delete":{"_index":"widgets","_id":"3","retry_on_conflict":3}} |
There was a problem hiding this comment.
Here we can see that the single-action delete bulk is correctly grouped with the 2-line index action before it.
| slice_size = options.delete(:actions_per_bulk) || ACTIONS_PER_BULK | ||
|
|
||
| current_bulk_batch.each_slice(slice_size) do |actions| | ||
| body = actions.flatten.map { |action| "#{ActiveSupport::JSON.encode(action)}\n" }.join |
There was a problem hiding this comment.
Would be nice to extract bulk post to a separate method so that slice behavior can be done optionally. Then method with slicing calls extracted method.
There was a problem hiding this comment.
I think we'll want to always slice though, no? In any case, actions_per_bulk can be specified if different slice sizes are needed
2 participants
Problem
It's deceptively easy to load hundreds of thousands of records into a single
_bulkrequest. These can result in413: Request too large exception, failing silently.Solution
Batch
_bulkrequests and only send max 1000 records at a time.