Data Engineering

Open-source data integration (ELT) with a huge connector catalogue; self-hostable. Unlike closed services such as Fivetran it is open source, and you can write a missing connector yourself.

A real-time stream-processing engine that handles events as they arrive, with low latency. Unlike Spark’s batch roots it treats streaming as the first-class model.

An open table format for large analytical tables, giving schema evolution, time travel and atomic writes. Unlike a raw Parquet directory it manages the table as a consistent, versioned unit.

Retains messages after consumption, with consumers carrying their own offsets so they can replay history. The real split from classic queues (RabbitMQ) is that durability and replay.

An engine that processes big data in a distributed, in-memory way — batch and streaming. Unlike single-machine libraries it splits terabytes across a cluster and processes them in parallel.

A list gathering every layer of data engineering — ingestion, storage, processing, orchestration — with tools. Unlike single-tool lists it maps the whole stack.

A resource list for Hadoop, the distributed storage and processing framework. Unlike lists for in-memory engines such as Spark, it focuses on the disk-based HDFS and MapReduce ecosystem.

A list of tools, libraries and resources for high-performance computing (HPC). Unlike general DevOps lists, it focuses on cluster management, job schedulers such as Slurm and parallel programming with MPI.

A list of packages, tutorials and examples for the Apache Spark ecosystem. Unlike the official docs it gathers community libraries and real-world examples.

A columnar OLAP database for analytics, extremely fast at aggregations at scale. Unlike row-based databases such as PostgreSQL it finishes aggregation queries over billions of rows in seconds; it is weak at single-record updates.

An orchestrator centred on data assets, tracking each table as an “asset”. Unlike Airflow it focuses on the data produced rather than the task, with data quality built in.

Long-form interviews digging deep into data-engineering tools and architectures. Unlike tool docs it tells why and how a tool was actually chosen, from the people who use it.

A free, project-based course teaching data engineering end to end (Docker, dbt, Spark, Kafka, cloud). Unlike scattered tutorials it has you build a real pipeline from start to finish.

A cloud platform built on Spark that merges the data lake and warehouse (“lakehouse”). Unlike running a separate warehouse and lake it unites them under one governance and notebook interface.

A free community offering hands-on data-engineering and ML zoomcamps. Unlike paid bootcamps it is entirely free and community-run — the camps run in cohorts on set dates.

A tool managing data-warehouse transformations with SQL + version control + tests. Unlike hand-written SQL scripts it brings a dependency graph, docs and tests like a software project.

An open storage layer bringing ACID transactions and time travel to data lakes. Unlike plain Parquet files it adds transactional safety and version history, so the lake behaves reliably like a table.

An in-process analytical database — no server, it queries files directly. Unlike server-based warehouses it is “OLAP at SQLite speed” on one machine, over Parquet/CSV.

A data-quality framework that validates and documents data with declarative expectations. Unlike hand-written validation scripts it turns rules into data documentation and embeds tests inside the pipeline.

A lightweight Python library that builds data transformations as a graph from functions. Unlike heavy orchestrators it gives a dependency graph in a single file with no infrastructure.

An open-source DataOps ELT platform, Singer-based and CLI/config-driven. Unlike Airbyte’s UI-first approach everything is defined in text config that lives in version control — you manage the pipeline like code.

A Microsoft learning path teaching cloud data engineering module by module. Unlike product docs it is framed as an ordered, exam-measured skill path.

A deep document on how Apache Spark works inside — scheduling, shuffle, memory. Unlike a usage guide it opens up the engine, for performance tuning.

A distributed SQL query engine that queries data where it lives, across many sources. Unlike classic databases it does not move the data to itself — you can query S3, PostgreSQL and Kafka together in one query.