ConcurrentModificationException when an update completed out of order

[solsson]

2019-03-05T07:58:53,530 DEBUG s.y-Key Got onupdate completion for topic2-0-127 previous offset 126   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:53,629 DEBUG s.y-Key Got onupdate completion for topic2-0-128 previous offset 127   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,229 WARN  s.y-OnU Unrecognized HTTP error for topic2-0-102[a8149657-4160-4b25-966c-a9921f19a100],5000,5000,http://target-service/kafka-keyvalue/v1/updates: MessageBodyWriter not found for media type=application/octet-stream, type=class se.yolean.kafka.keyvalue.UpdateRecord, genericType=class se.yolean.kafka.keyvalue.UpdateRecord.   [main]se.yolean.kafka.keyvalue.onupdate.OnUpdateWithExternalPollTrigger
2019-03-05T07:58:54,230 WARN  s.y-Key On-update completed out of order, topic2-0-102 before 101   [main]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,232 WARN  s.y-Key On-update failed for topic2-0-102[a8149657-4160-4b25-966c-a9921f19a100]   [main]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
Exception in thread "main" java.util.ConcurrentModificationException
	at java.base/java.util.LinkedHashMap$LinkedHashIterator.remove(LinkedHashMap.java:732)
	at se.yolean.kafka.keyvalue.onupdate.OnUpdateWithExternalPollTrigger.checkCompletion(OnUpdateWithExternalPollTrigger.java:93)
	at se.yolean.kafka.keyvalue.cli.Main.pollOnupdate(Main.java:87)
	at se.yolean.kafka.keyvalue.cli.Main.main(Main.java:30)
2019-03-05T07:58:54,323 DEBUG s.y-Key Got onupdate completion for topic2-0-129 previous offset 128   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor
2019-03-05T07:58:54,332 DEBUG s.y-Key Got onupdate completion for topic2-0-130 previous offset 129   [application-id-df37f4ed-b8b6-4fb2-b7cd-d3cbe790dbfb-StreamThread-1]se.yolean.kafka.keyvalue.KeyvalueUpdateProcessor

[solsson]

I don't think this has anything to do with out of order updates. It's just that we add onupdate items from the streams thread and remove them from the main thread that polls onupdate.

I've seen documentation on streams API support for scheduled operations, which could be an alternative to polling form main, but I can't find it now. Anyhow there could be many threads in streams so concurrent updates wouldn't be ruled out anyway.

Using a ConcurrentHashMap would be possible, but I'd like iteration order to be that of inertion as with LinkedHashMap. Using synchronized is another option.

[solsson]

e7736fe introduced a punctuation that we might want to move polling to. There's some info on the threading model at https://docs.confluent.io/current/streams/architecture.html#threading-model