I'm experiencing a critical issue with my ScyllaDB setup and I'm seeking help to understand the root cause and prevent it from happening again.
Problem Description:
I have a single-node ScyllaDB instance. One of my tables, cache.artists_bridge
, became completely unreachable. While other tables in the same keyspace remained accessible and responsive, even simple SELECT queries on cache.artists_bridge
timed out.
Here's the output of nodetool tablestats cache.artists_bridge
i took after the issue:
Total number of tables: 87
----------------
Keyspace : cache
Read Count: 10008066
Read Latency: 1.102521705991947E-04 ms
Write Count: 0
Write Latency: NAN ms
Pending Flushes: 0
Table: artists_bridge
SSTable count: 8
SSTables in each level: [8/4]
Space used (live): 958116581
Space used (total): 958116581
Space used by snapshots (total): 0
Off heap memory used (total): 13410653
SSTable Compression Ratio: 0.5
Number of partitions (estimate): 10368237
Memtable cell count: 0
Memtable data size: 0
Memtable off heap memory used: 0
Memtable switch count: 0
Local read count: 10008066
Local read latency: 0.110 ms
Local write count: 0
Local write latency: 0.000 ms
Pending flushes: 0
Percent repaired: 0.0
Bloom filter false positives: 3519
Bloom filter false ratio: 0.01155
Bloom filter space used: 12961016
Bloom filter off heap memory used: 12961016
Index summary off heap memory used: 449637
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 25
Compacted partition maximum bytes: 124
Compacted partition mean bytes: 104
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0
Dropped Mutations: 0
Scylla version is 3.0.8
This table is read-only, i feed it once with data and after i only read it. It has 10 millions line and 2 colums
Created like that :
create table artists_bridge
(
artist_id text primary key,
global_artist_id text
)
It also is not showing in scylla monitoring, which was working just yesterday (and all other tables are here).
Most intrigant issue is that the file system is empty
/mnt/data/data/cache/artists_bridge-b86483e019dc11f09e381e5fa41c82c2$ find .
.
./snapshots
./snapshots/pre-drop-1747314122021
./snapshots/pre-drop-1747314122021/schema.cql
./snapshots/pre-drop-1747314122021/manifest.json
I looked into the syslog and all and nothings appears to have arrived....