NoSQL Big Data with Databricks

Context

ITS course project by the Big Data NoSQL Gang group, developed in the former Databricks Community Edition. The recovered archive contains Bronze, Silver and Gold source notebooks, documentation and a visualization configuration; it contains neither CSV files nor execution outputs. “NoSQL” is the project name, not evidence that a NoSQL database was used.

Problem

Prepare Inside Airbnb data for Amsterdam and Rio de Janeiro to analyze listings and reviews while keeping ingestion, normalization and analytical queries separate.

Goal

Organize a Bronze/Silver/Gold pipeline and write queries for comparing the two cities. The group goal does not establish that the pipeline completed an end-to-end run.

My responsibilities

I was team leader and worked mainly on coordination and data ingestion. The exported Bronze notebook bears my name and documents that stage. Silver transformations, Gold queries and visualizations are described as group work: filenames alone do not prove authorship of individual cells.

Documented implementation

The Bronze notebook reads four detailed listings and reviews CSVs from a Databricks volume, adds city and ingestion timestamp, and defines four Delta tables. The Silver notebook creates silver_listings with a UNION ALL of Amsterdam and Rio, selected columns, and type and price conversions. The documentation mentions silver_reviews, but its defining code is absent from the archive. The Gold notebook defines SQL views for counts and distributions; the IPYNB retains two chart configurations without exported values.

Technical decisions and tradeoffs

• CSV parsing

  Bronze reads columns as strings and configures multiline input, quotes, escaping and PERMISSIVE mode for complex text fields. The documentation explains the move from summary to detailed datasets to retain ratings; the original data files are not in the archive.

• Layer separation

  Silver preserves records with UNION ALL and defers analytical filtering to Gold. The documentation states that listings were not deduplicated; no exported check establishes ID uniqueness.

• Price comparability

  The documentation distinguishes euros for Amsterdam from Brazilian reais for Rio and proposes city-specific comparisons. Some exported Gold queries do not apply the intended city filter, so the safeguard is not established for every visualization.

Checks and verification

Review is limited to the exported source. Datasets, run logs and outputs are missing; silver_reviews code is absent and some Gold queries need correction.

Documented evidence and results

• Source artifacts

  An ingestion notebook defining four Bronze tables, a notebook defining silver_listings, and a Gold notebook containing analytical views and queries are available.

• Visualizations

  An IPYNB export preserves settings for two charts of counts by city and accommodation type. It contains neither chart data nor a snapshot of the final dashboard.

Limits and improvements

Counts, performance and full-pipeline completion cannot be verified without datasets, run logs and outputs. The silver_reviews code is missing; some Gold queries depend on that table or contain errors and inconsistencies. The Databricks volume and catalog must be recreated for a new run. Publishing group artifacts requires a rights review and coauthor consent.

Technologies

• Databricks Community Edition (original environment)
• PySpark
• Spark SQL
• Delta Lake
• Inside Airbnb (stated data source)