Data Engineering — Useful Sites

Airbyte

2026-09-18T00:00:00Z

Open-source data integration (ELT) with a huge connector catalogue; self-hostable. Unlike closed services such as Fivetran it is open source, and you can write a missing connector yourself.

https://airbyte.com

Apache Flink

2026-09-18T00:00:00Z

A real-time stream-processing engine that handles events as they arrive, with low latency. Unlike Spark’s batch roots it treats streaming as the first-class model.

https://flink.apache.org/

Apache Iceberg

2026-09-18T00:00:00Z

An open table format for large analytical tables, giving schema evolution, time travel and atomic writes. Unlike a raw Parquet directory it manages the table as a consistent, versioned unit.

https://iceberg.apache.org/

Apache Spark

2026-09-18T00:00:00Z

An engine that processes big data in a distributed, in-memory way — batch and streaming. Unlike single-machine libraries it splits terabytes across a cluster and processes them in parallel.

https://spark.apache.org/

Awesome Data Engineering

2026-09-18T00:00:00Z

A list gathering every layer of data engineering — ingestion, storage, processing, orchestration — with tools. Unlike single-tool lists it maps the whole stack.

https://github.com/igorbarinov/awesome-data-engineering

Awesome Hadoop

2026-09-18T00:00:00Z

A resource list for Hadoop, the distributed storage and processing framework. Unlike lists for in-memory engines such as Spark, it focuses on the disk-based HDFS and MapReduce ecosystem.

https://github.com/youngwookim/awesome-hadoop

Awesome HPC

2026-09-18T00:00:00Z

A list of tools, libraries and resources for high-performance computing (HPC). Unlike general DevOps lists, it focuses on cluster management, job schedulers such as Slurm and parallel programming with MPI.

https://github.com/dstdev/awesome-hpc

Awesome Spark

2026-09-18T00:00:00Z

A list of packages, tutorials and examples for the Apache Spark ecosystem. Unlike the official docs it gathers community libraries and real-world examples.

https://github.com/awesome-spark/awesome-spark

ClickHouse

2026-09-18T00:00:00Z

A columnar OLAP database for analytics, extremely fast at aggregations at scale. Unlike row-based databases such as PostgreSQL it finishes aggregation queries over billions of rows in seconds; it is weak at single-record updates.

https://clickhouse.com

Dagster

2026-09-18T00:00:00Z

An orchestrator centred on data assets, tracking each table as an “asset”. Unlike Airflow it focuses on the data produced rather than the task, with data quality built in.

https://dagster.io/

Data Engineering Podcast

2026-09-18T00:00:00Z

Long-form interviews digging deep into data-engineering tools and architectures. Unlike tool docs it tells why and how a tool was actually chosen, from the people who use it.

https://www.dataengineeringpodcast.com

DataTalks.Club

2026-09-18T00:00:00Z

A free community offering hands-on data-engineering and ML zoomcamps. Unlike paid bootcamps it is entirely free and community-run — the camps run in cohorts on set dates.

https://datatalks.club

dbt

2026-09-18T00:00:00Z

A tool managing data-warehouse transformations with SQL + version control + tests. Unlike hand-written SQL scripts it brings a dependency graph, docs and tests like a software project.

https://www.getdbt.com/

Delta Lake

2026-09-18T00:00:00Z

An open storage layer bringing ACID transactions and time travel to data lakes. Unlike plain Parquet files it adds transactional safety and version history, so the lake behaves reliably like a table.

https://delta.io

DuckDB

2026-09-18T00:00:00Z

An in-process analytical database — no server, it queries files directly. Unlike server-based warehouses it is “OLAP at SQLite speed” on one machine, over Parquet/CSV.

https://duckdb.org/

Great Expectations

2026-09-18T00:00:00Z

A data-quality framework that validates and documents data with declarative expectations. Unlike hand-written validation scripts it turns rules into data documentation and embeds tests inside the pipeline.

https://greatexpectations.io

Hamilton

2026-09-18T00:00:00Z

A lightweight Python library that builds data transformations as a graph from functions. Unlike heavy orchestrators it gives a dependency graph in a single file with no infrastructure.

https://github.com/dagworks-inc/hamilton

Meltano

2026-09-18T00:00:00Z

An open-source DataOps ELT platform, Singer-based and CLI/config-driven. Unlike Airbyte’s UI-first approach everything is defined in text config that lives in version control — you manage the pipeline like code.

https://meltano.com

Spark Internals

2026-09-18T00:00:00Z

A deep document on how Apache Spark works inside — scheduling, shuffle, memory. Unlike a usage guide it opens up the engine, for performance tuning.

https://github.com/JerryLead/SparkInternals

Trino

2026-09-18T00:00:00Z

A distributed SQL query engine that queries data where it lives, across many sources. Unlike classic databases it does not move the data to itself — you can query S3, PostgreSQL and Kafka together in one query.

https://trino.io

Data Engineering Zoomcamp

2026-08-17T20:53:20Z

A free, project-based course teaching data engineering end to end (Docker, dbt, Spark, Kafka, cloud). Unlike scattered tutorials it has you build a real pipeline from start to finish.

https://github.com/DataTalksClub/data-engineering-zoomcamp

Microsoft · Data Engineer Path

2026-08-17T20:53:20Z

A Microsoft learning path teaching cloud data engineering module by module. Unlike product docs it is framed as an ordered, exam-measured skill path.

https://learn.microsoft.com/en-us/training/career-paths/data-engineer

Databricks

2026-07-26T15:42:56Z

A cloud platform built on Spark that merges the data lake and warehouse (“lakehouse”). Unlike running a separate warehouse and lake it unites them under one governance and notebook interface.

https://www.databricks.com/

Apache Kafka

2026-04-20T13:01:27Z

Retains messages after consumption, with consumers carrying their own offsets so they can replay history. The real split from classic queues (RabbitMQ) is that durability and replay.

https://kafka.apache.org/