Setup: -
I have a Cassandra cluster running in 3 datacenters with 3 nodes each, having Replication factor of 3-3-3 hosted on GCP.
Workload is quite write intensive with around 1.5k writes/s and 100 reads/s/node. Both read and write queries are done with LOCAL_QUORUM.
Compaction strategy used - Levelled Compaction strategy
Total Memory allocated - 32 gb
Total Heap allocated - 10 gb
Allocated off-heap - 4 gb
Issue: -
There are spikes in node Memory utilization during compactions which lasts for a short amount of time, Memory utilization is around 44% overall and the spikes go up to 85%. Is there a way to tune something and restrict the spikes to a lower value?
I understand Compaction is a natural, fundamental process, but having these large spikes makes capacity planning difficult.
I have tried tuning compaction_throughput_mb_per_sec and concurrent_compactors settings, which doesn't seem to affect the spikes at all.
