July 28th, 2014

Cassandra 2.1 introduces incremental repairs which makes repair a lot more lightweight as it does not do repair on already repaired data. Anticompaction is one of the things that makes incremental repairs possible. This blog posts aims to explain what anticompaction is and how it affects regular compaction.

Incremental repairs

The concept of incremental repairs is explained more in detail in this post, but the general idea is that we mark all SSTables that were involved in the repair with a timestamp to indicate that it was repaired, then we simply don’t include that SSTable when we do the next repair since we know that data that was once repaired will stay that way.


Since SSTables can contain any range, we need to split out the ranges that were actually repaired, this is called anticompaction. It means that one SSTable is split in two – one containing repaired data and one containing unrepaired data.

If the range in the SSTable is fully contained within the range that was repaired, we don’t actually rewrite the SSTable, instead we just change the SSTable metadata to indicate when it was repaired.

Since we now have two sets of SSTables that we can’t compact together, we need to do some adjustments to our compaction strategies.

Size tiered compaction

With size tiered compaction it is pretty simple, we split the SSTables in two sets, one with repaired and one with unrepaired. Then we try to find compaction candidates within those two sets and run the compaction on the candidates that would have the biggest benefit.

Major compaction will create two SSTables instead of one. If you use major compaction to get rid of tombstones, it should still work just as well since we can drop tombstones if the SSTables included in the compaction contain the tombstone plus all older occurrences of the key, and in this case it will, since the unrepaired set of SSTables will almost always only include newer data. This means that we do not need to check if the key is in one of those SSTables as the tombstone can’t cover any data there.

Leveled compaction

For leveled compaction we do leveling on the repaired SSTables and then size tiered compaction on the unrepaired ones. This means that once you do an incremental repair you will have to continue doing them (there are ways to clear out the repair-state to revert this, more about that later). Otherwise you will not run leveled compaction, just size tiered.

The complicated part is how to migrate to using incremental repairs since it is only after the first incremental repair run that we want to separate repaired and unrepaired SSTables, before that we want a leveling on all the SSTables.

After the incremental repair is done, we iterate over the SSTables included in the repair and run anticompaction on them one at a time. This means that after the first SSTable has been anticompacted, we will have to move all the currently leveled but unrepaired SSTables out and end up with only the first repaired and anticompacted SSTable in the leveling and possibly thousands in the unrepaired set. After that we continue and anticompact the rest of the SSTables.

We do a few things to make it better though. First, when we clear out the unrepaired SSTables from the leveling we keep the original SSTable level to make it possible to re-add the SSTable at its original position after it has been anticompacted. For example, if an SSTable is in L3 before anticompaction, it is likely that we can add it in level 3 after the anticompaction. This is especially important as we anticompact one SSTable at a time during an anticompaction session, meaning many SSTables will just temporarily go into the unrepaired set because they might have just been repaired, just not yet anticompacted.


Running the first incremental repair will affect many nodes at the same time, to avoid that there is a way to migrate one node at a time, though it requires a bit of manual labour;

  1. Disable compaction on the node (nodetool disableautocompaction)
  2. Run a classic full repair
  3. Stop the node
  4. Use the tool sstablerepairedset to mark all the SSTables that were created before you did step 1.
  5. Restart cassandra

If you run regular repairs you could note when you last ran a full repair on the node and use that time. SSTables are immutable, meaning if an SSTable has not changed since the repair started, it is still repaired. Note that you need to check when you last ran a full repair (not -pr) and you will need to do it on every node.

This tool can also be used to clear out the repaired-state on the SSTable, stop the node run the command tools/bin/sstablerepairedset –is-unrepaired <sstable> on all SSTables and restart, now all your data will be leveled again.

Anticompaction in Cassandra 2.1” was created by Marcus Eriksson