Spotify drives personalization with Apache Cassandra

Spotify utilizes Apache Cassandra in its infrastructure to serve data behind a personalization algorithm, which recommends songs to its users. Spotify’s data infrastructure is built on open-source Apache software platforms like Kafka, Storm, and Cassandra. These systems work together to collect and analyze data in real-time and store it in user profiles as personalized suggestions.

Cassandra is used by Spotify for storage. It consists of over 100 Cassandra clusters, each containing a nested storage system. User profile attributes and metadata about playlists and artists are stored in these clusters.

The personalization algorithm at Spotify is responsible for creating playlists like Discover Weekly. It uses three data analysis models: Collaborative Filtering, Natural Language Processing (NLP), and Audio Analysis.

Collaborative Filtering analyzes data stored in matrices to determine a user’s taste in music and a song’s characteristics. Spotify recommends songs by comparing a user’s vector to other users’ vectors to find similar songs. NLP models analyze text on the internet about songs to determine cultural similarity. Sentiment analysis APIs are used to create vectors from the collected text.

Audio models analyze the raw audio tracks themselves by using convolutional neural networks to compute representative statistics of the song, such as the time signature, key, and tempo.
Spotify’s Home Screen is another example of personalization. It is organized like a bookcase, with each shelf displaying personalized content based on a user’s recent activity, favorite playlists, new releases, and custom mixtapes.

Spotify uses machine learning for both analysis (displaying content based on previous activity) and exploration (displaying suggested content to initiate user engagement and modifying it based on user interaction). Once this process is complete, the data is written to Cassandra and served via its (micro) service layer.

Overall, Apache Cassandra provides a great platform from which to serve recommendation data, and Spotify is confident in its ability to scale to meet the needs of their ever-growing customer base.