Illustration Image

ScyllaDB Table Data Loss

I'm experiencing a critical issue with my ScyllaDB setup and I'm seeking help to understand the root cause and prevent it from happening again.

Problem Description:

I have a single-node ScyllaDB instance. One of my tables, cache.artists_bridge, became completely unreachable. While other tables in the same keyspace remained accessible and responsive, even simple SELECT queries on cache.artists_bridge timed out.

Here's the output of nodetool tablestats cache.artists_bridge i took after the issue:

Total number of tables: 87
----------------
Keyspace : cache
    Read Count: 10008066
    Read Latency: 1.102521705991947E-04 ms
    Write Count: 0
    Write Latency: NAN ms
    Pending Flushes: 0
        Table: artists_bridge
        SSTable count: 8
        SSTables in each level: [8/4]
        Space used (live): 958116581
        Space used (total): 958116581
        Space used by snapshots (total): 0
        Off heap memory used (total): 13410653
        SSTable Compression Ratio: 0.5
        Number of partitions (estimate): 10368237
        Memtable cell count: 0
        Memtable data size: 0
        Memtable off heap memory used: 0
        Memtable switch count: 0
        Local read count: 10008066
        Local read latency: 0.110 ms
        Local write count: 0
        Local write latency: 0.000 ms
        Pending flushes: 0
        Percent repaired: 0.0
        Bloom filter false positives: 3519
        Bloom filter false ratio: 0.01155
        Bloom filter space used: 12961016
        Bloom filter off heap memory used: 12961016
        Index summary off heap memory used: 449637
        Compression metadata off heap memory used: 0
        Compacted partition minimum bytes: 25
        Compacted partition maximum bytes: 124
        Compacted partition mean bytes: 104
        Average live cells per slice (last five minutes): 0.0
        Maximum live cells per slice (last five minutes): 0
        Average tombstones per slice (last five minutes): 0.0
        Maximum tombstones per slice (last five minutes): 0
        Dropped Mutations: 0

Scylla version is 3.0.8

This table is read-only, i feed it once with data and after i only read it. It has 10 millions line and 2 colums

Created like that :

create table artists_bridge
(
    artist_id        text primary key,
    global_artist_id text
)

It also is not showing in scylla monitoring, which was working just yesterday (and all other tables are here).

Most intrigant issue is that the file system is empty

/mnt/data/data/cache/artists_bridge-b86483e019dc11f09e381e5fa41c82c2$ find .
.
./snapshots
./snapshots/pre-drop-1747314122021
./snapshots/pre-drop-1747314122021/schema.cql
./snapshots/pre-drop-1747314122021/manifest.json

I looked into the syslog and all and nothings appears to have arrived....

Become part of our
growing community!
Welcome to Planet Cassandra, a community for Apache Cassandra®! We're a passionate and dedicated group of users, developers, and enthusiasts who are working together to make Cassandra the best it can be. Whether you're just getting started with Cassandra or you're an experienced user, there's a place for you in our community.
A dinosaur
Planet Cassandra is a service for the Apache Cassandra® user community to share with each other. From tutorials and guides, to discussions and updates, we're here to help you get the most out of Cassandra. Connect with us and become part of our growing community today.
© 2009-2023 The Apache Software Foundation under the terms of the Apache License 2.0. Apache, the Apache feather logo, Apache Cassandra, Cassandra, and the Cassandra logo, are either registered trademarks or trademarks of The Apache Software Foundation. Sponsored by Anant Corporation and Datastax, and Developed by Anant Corporation.

Get Involved with Planet Cassandra!

We believe that the power of the Planet Cassandra community lies in the contributions of its members. Do you have content, articles, videos, or use cases you want to share with the world?