Illustration Image

12/2/2023

Reading time:2

Build a Website Scraper with Astra DB + Python Examples | DataStax

logo

This resource is based on an article originally published here.

Learn how to scrape websites with Astra DB, Python, Selenium, Requests HTML, Celery, & FastAPI.

180 minutes • Advanced

Updated October 19, 2021


180 minutes, Advanced, Start Building

Learn how to scrape websites with Astra DB, Python, Selenium, Requests HTML, Celery, & FastAPI by following along with CodingEntrepreneurs' video, located here.

image

Quick Start

  • Signup for DataStax Astra, or login to your already existing account.

  • Create an Astra DB Database if you don't already have one.

  • Create an Astra DB Keyspace called sag_python_scraper in your database.

  • Generate an Application Token with the role of Database Administrator for the Organization that your Astra DB is in.

  • Get your secure connect bundle from the connect page of your database and save it to your project folder. Rename it to bundle.zip

  • Setup your system: Below is a preflight checklist to ensure you system is fully setup to work with this course. All guides and setup can be found in the setup directory of this repo.

    • Install Selenium & Chromedriver - setup guide

    • Install Redis - setup guide

    • Create a virtual environment & install dependencies

    • Setup an account with DataStax

    • Create your first AstraDB and get API credentials

    • Use cassandra-driver to verify your connection to AstraDB

How this works

Follow along in this video tutorial: https://youtu.be/NyDT3KkscSk.

This series is broken up into 4 parts:

  • Scraping How to scrape and parse data from nearly any website with Selenium & Requests HTML.
  • Data models how to store and validate data with cassandra-driver, pydantic, and AstraDB.
  • Worker & Scheduling how to schedule periodic tasks (ie scraping) integrated with Redis & AstraDB
  • Presentation How to combine the above steps in as robust web application service

Here's what each tool is used for:

  • Python 3.9 download - programming the logic.
  • AstraDB sign up - highly perfomant and scalable database service by DataStax. AstraDB is a Cassandra NoSQL Database. Cassandra is used by Netflix, Discord, Apple, and many others to handle astonding amounts of data.
  • Selenium docs - an automated web browsing experience that allows:
    • Run all web-browser actions through code
    • Loads JavaScript heavy websites
    • Can perform standard user interaction like clicks, form submits, logins, etc.
  • Requests HTML docs - we're going to use this to parse an HTML document extracted from Selenium
  • Celery docs - Celery providers worker processes that will allow us to schedule when we need to scrape websites. We'll be using redis as our task queue.
  • FastAPI docs - as a web application framework to Display and monitor web scraping results from anywhere

Related Articles

logo
cluster
troubleshooting
datastax

Explore Further

web.scraping

scraping

datastax

Become part of our
growing community!
Welcome to Planet Cassandra, a community for Apache Cassandra®! We're a passionate and dedicated group of users, developers, and enthusiasts who are working together to make Cassandra the best it can be. Whether you're just getting started with Cassandra or you're an experienced user, there's a place for you in our community.
A dinosaur
Planet Cassandra is a service for the Apache Cassandra® user community to share with each other. From tutorials and guides, to discussions and updates, we're here to help you get the most out of Cassandra. Connect with us and become part of our growing community today.
© 2009-2023 The Apache Software Foundation under the terms of the Apache License 2.0. Apache, the Apache feather logo, Apache Cassandra, Cassandra, and the Cassandra logo, are either registered trademarks or trademarks of The Apache Software Foundation.

Get Involved with Planet Cassandra!

We believe that the power of the Planet Cassandra community lies in the contributions of its members. Do you have content, articles, videos, or use cases you want to share with the world?