[flags] fix: use write lock for addListener#3132
Conversation
| val listener1 = object : FlagsStateListener { | ||
| override fun onStateChanged(newState: FlagsClientState) { | ||
| if (newState is FlagsClientState.Ready) { | ||
| if (newState == FlagsClientState.Ready) { |
There was a problem hiding this comment.
FlagClientState is a sealed class
|
|
||
| @Test | ||
| fun `M stop notifying subsequent listeners W updateState() { if listener throws }`() { | ||
| fun `M stop notifying subsequent listeners W updateState() { listener throws }`() { |
There was a problem hiding this comment.
polishing some names/comments while here
|
🎯 Code Coverage 🔗 Commit SHA: 0558c4c | Docs | Datadog PR Page | Was this helpful? Give us feedback! |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3132 +/- ##
========================================
Coverage 70.73% 70.73%
========================================
Files 893 893
Lines 33000 33000
Branches 5549 5550 +1
========================================
+ Hits 23341 23342 +1
+ Misses 8102 8097 -5
- Partials 1557 1561 +4
🚀 New features to boost your workflow:
|
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { | ||
| // Given | ||
| val stateOld = FlagsClientState.NotReady |
There was a problem hiding this comment.
nit: probably should be oldState to be consistent with naming convention used in newState.
Or maybe for the clarity we can just inline FlagsClientState.NotReady where needed, because stateOld is val anyway and cannot be changed.
| addListenerThread.start() | ||
| addListenerStarted.await() | ||
| getCurrentStateThread.start() | ||
| getCurrentStateAttempted.await() |
There was a problem hiding this comment.
getCurrentStateAttempted is not actually needed I think, the necessary wait for the getCurrentStateThread completion is already implemented by the getCurrentStateThread.join() call below.
| val addListenerSlowCallbackStarted = CountDownLatch(1) | ||
| val getCurrentStateAttempted = CountDownLatch(1) | ||
|
|
||
| val operationTimestamps = mutableListOf<Pair<String, Long>>() |
There was a problem hiding this comment.
synchronized calls can be removed if type here is CopyOnWriteArrayList
| } | ||
|
|
||
| @Test | ||
| fun `M block getCurrentState calls W addListener() { slow listener notification }`() { |
There was a problem hiding this comment.
Why do we want to block getCurrentState when addListener is called, if addListener is not changing the state?
There was a problem hiding this comment.
🤔 We may have had the right lock here in the first place...
If addListener holds the read lock, then, ostensibly, nothing can write to the state (current state or the listeners). While addListener is technically mutating the protected state by adding a listener, the underlying mechanism itself is theadsafe/synchronized via CopyOnWriteArrayList so it's fine if parallel calls to addListener attempt to mutate the set of listeners. The order of listener subscription is not important to maintain here.
We should just be able to rely on the underlying thread-safety of CopyOnWriteArrayList to avoid clobbering of the listener list, right?
There was a problem hiding this comment.
Maybe we shouldn't wrap adding listener with lock then?
There was a problem hiding this comment.
We need to block writing to the current state until the first notification and addListener are complete. We don't get that atomicity with DDCoreSubscription, only that the calls underlying calls to add the listener to the list are thread safe and synchronized so no listener will be lost when parallel calls are made.
In effect, I believe we can drop this change altogether and rely on the read lock in conjunction with the underlying lock in CopyOnWriteArrayList
There was a problem hiding this comment.
I think yes, we should. My thinking is the following:
getCurrentStatecovers onlyFlagsStateManager.currentState. Listeners is not a part ofFlagsStateManager.currentState, so they are not related to thegetCurrentStatecall.FlagsStateManager.addListenercall doesn't modifyFlagsStateManager.currentState, it only reads. So that us why we added a read lock. If something is in the process of modifyingFlagsStateManager.currentState, then there will be a wait to acquirereadlock.FlagsStateManager.addListenermutates underlying listeners collection, but anyway this is atomic and thread-safe,currentStatevalue passed down is guarded by the read lock and if there is iteration already happening inupdateState -> subscription.notifyListeners, then this new listener won't be a part of ongoing iteration (due to the usage ofCopyOnWriteArrayListinDDSubscription).
So I'm curious why should we block getCurrentState call during FlagsStateManager.addListener invocation.
The race condition could otherwise result in a listener missing a state change
Is it about FlagsStateManager.currentState property here? From the code I see, listener shouldn't miss any state change, since read lock cannot be acquired if write lock is active, and the opposite.
There was a problem hiding this comment.
Well, having read it more carefully, I agree with @nikita. The only thing that still makes me curious is the fact that we adding listener with a lock, but removing without. Yes, underlying CopyOnWriteArrayList will protect us from the ConcurrentModificationException, however if updateState and removeListener will be executed at the same time, removed listener could get an undesired update (from subscription.notifyListeners ) and could lead to undefined behavior. Not sure if this is a critical issue, but I'd prefer to maintain consistency here.
There was a problem hiding this comment.
Thanks @satween. Agreed that there's an apparent lack of synchronicity on removing a listener.
We are not just "Adding a listener", however.
We need to
- get the current state
- notify the listener
- add the listener to the list
All without thecurrentStatechanging, so we block changes with areadlock.
When we remove the listener, have only that operation to complete so there isn't a risk of missing data, only, as you noted, receiving extra data. We would need to add an extra synchronization shared between the removeListener method and the updateState method. I think you're correct here though - a long-running listener callback could cause a removed listener to be notified after the call to removeListener completes
The scale of adding/removing listeners will be very low (0, or 1 or maybe a few) so there is a very small risk here of a superfluous event.
I'll come back to this with an updated lock after a couple of other tasks
What does this PR do?
Uses a
writelock instead ofreadwhen adding a listenerMotivation
Adding a listener effectively changes the contested state we want to keep atomic {
currentState,listeners} so it must block otherwritecalls. The race condition could otherwise result in a listener missing a state changeAdditional Notes
Anything else we should know when reviewing?
Review checklist (to be filled by reviewers)