buck-http-cache cluster often descends into broken state

[jiangty-addepar]

We're trying to run the buck-http-cache as a cluster with 3 nodes (deployed via Kubernetes). However, it quickly (often within ~1 hour) reaches a bad state where requests repeatedly throw errors like this:

[29/Jun/2018:01:06:40 +0000] "PUT /artifacts/key HTTP/1.1" 500 110 "-" "okhttp/3.6.0" 1
ERROR [2018-06-29 01:06:40,490] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: 51ddcabd89d87889
! java.lang.NullPointerException: null
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl$2.call(GridCacheAtomicSequenceImpl.java:514)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl$2.call(GridCacheAtomicSequenceImpl.java:487)
! at org.apache.ignite.internal.processors.cache.GridCacheUtils$21.call(GridCacheUtils.java:1613)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.internalUpdate(GridCacheAtomicSequenceImpl.java:258)
! ... 61 common frames omitted
! Causing: org.apache.ignite.IgniteCheckedException: null
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.internalUpdate(GridCacheAtomicSequenceImpl.java:264)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.incrementAndGet(GridCacheAtomicSequenceImpl.java:178)
! ... 60 common frames omitted
! Causing: org.apache.ignite.IgniteException: null
! at org.apache.ignite.internal.util.IgniteUtils.convertException(IgniteUtils.java:946)
! at org.apache.ignite.internal.processors.datastructures.GridCacheAtomicSequenceImpl.incrementAndGet(GridCacheAtomicSequenceImpl.java:181)
! at com.uber.buckcache.datastore.impl.ignite.IgniteDataStoreProvider.putData(IgniteDataStoreProvider.java:113)
! at com.uber.buckcache.datastore.impl.ignite.IgniteDataStoreProvider.putData(IgniteDataStoreProvider.java:82)
! at com.uber.buckcache.resources.buckcache.BuckCacheResource.addArtifactToCache(BuckCacheResource.java:147)
...

We originally ran Apache Ignite in PARTITIONED mode, and we even tried out REPLICATED mode... but it's still happening. We suspect it's because either Ignite or the cache itself is kind of brittle when there's momentary network disconnections between the nodes in the buck-http-cache cluster.

Has anyone seen these problems before, or have any tips on what to do? Thanks!