Open-Source Data Observability with Elementary — From Zero to Hero (Part 1)

A step-by-step hands-on guide I wish I had when I was a beginner

Data observability and its importance have often been discussed and written about as a crucial aspect of modern data and analytics engineering. Many tools are available on the market with various features and prices. In this 2 part article, we will focus on the open-source version of Elementary, one of these data observability platforms, tailored for and designed to work seamlessly with dbt. We will start by setting up from zero and aiming to understand how it works and what is possible in different data scenarios by the end of part 2. Before we start, I also would like to disclose that I have no affiliation with Elementary, and all opinions expressed are my own.

In part 1, we will set up the Elementary and check how to read the Elementary’s daily report. If you are comfortable with this part already and interested in checking different types of data tests and which one suits bests for which scenario, you can directly jump into part 2 here:

• Opensource Data Observability with Elementary — From Zero to Hero (Part 2)

I have been using Elementary for quite some time and my experiences as a data engineer are positive, as to how my team conceives the results. Our team uses Elementary for automated daily monitoring with a self-hosted elementary dashboard. Elementary also has a very convenient cloud platform as a paid product, but the open-source version is far more than enough for us. If you want to explore the differences and what features are missing in open source, elementary compares both products here. Let us start by setting up the open-source version first.

• How to Install Elementary

Installing Elementary is as easy as installing any other package in your dbt project. Simply add the following to your packages.yml file. If you don't have one yet, you can create a packages.yml file at the same level as your dbt_project.yml file. A package is essentially another dbt project, consisting of additional SQL and Jinja code that can be incorporated into your dbt project.

packages:
  - package: elementary-data/elementary
    version: 0.15.2
    ## you can also have different minor versions as:
      ##  version: [">=0.14.0", "<0.15.0"]
    ## Docs: https://docs.elementary-data.com