Articles by Tag #dataengineering
Browse our collection of articles on various topics related to IT technologies. Dive in and explore something new!
The Ultimate Linux Command Cheat Sheet for Data Engineers and Analysts
Introduction As a data engineer or analyst, your day-to-day responsibilities likely...
When Small Parquet Files Become a Big Problem (and How I Ended Up Writing a Compactor in PyArrow)
It all began with a fairly normal data pipeline. Events were coming in through Kafka, landing in AWS...
🌍 Automating Africa’s Energy Data Collection Using Python, Playwright(+Why Playwright ?), and MongoDB (2000–2024)
⚡ Introduction In today’s data-driven world, access to reliable and structured energy data...
Why Parquet Is Everywhere - And What Makes It Actually Fast?
Hey folks 👋, As I kept building more data pipelines, I noticed one file format showing up...
RIP Amazon Data Firehose Change Data Capture
When a couple of weeks before re:Invent 2024, AWS announced in a blog post titled Replicate changes...
Writes, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™
Writes, 3 ways: Postgres, Apache Kafka® and Apache Iceberg™ As a part of my new job at...
Building a 75,000-Product Image Feature Dataset for the Amazon ML Challenge 2025
Hey everyone! Hope you’re all doing great today. So I’ve got something pretty exciting to share with...
From smog to streams: how data engineering helps us breathe easier.
Building a Real-Time Air Quality Data Pipeline for Mombasa & Nairobi The...
Data Quality at Scale: Why Your Pipeline Needs More Than Green Checkmarks
Originally published on Medium:...
Simulating An Event-Driven Python Shopping App with Kafka on AWS For Real-Time Processing.
Business Use Application/Relevance: Applications like banking applications, streaming(like...
How I Built a MongoDB Archiving System for Crawled Data
How I Built a MongoDB Archiving System for Crawled Data The Problem: Data Chaos...
Interoperating Open Table Formats on AWS Using Apache XTable (Delta Iceberg)
Original Japanese article: Apache XTableを使ったAWS上でのOpen Table Format相互運用（Delta→Iceberg） ...
Designing Data-Intensive Applications — Chapter 1: Reliable, Scalable, and Maintainable Applications
This post is part of a series summarizing key ideas from Designing Data-Intensive Applications by...
Auto-Detecting CSV Schemas for Lightning-Fast ClickHouse Ingestion with Parquet
As a data engineer, one of the most repetitive tasks I face is ingesting data from CSV files. The...
Real-Time Crypto Data Pipeline
Introduction Ever wondered how trading platforms display live crypto prices? In this...
🧠 ClickHouse LEFT JOINs: Why join_use_nulls Matters
🧠 Understanding join_use_nulls in ClickHouse ClickHouse is famous for being blazing fast —...
Another Data Nerd Guide to re:Invent 2025
Well, it's that time of year again. In less than two months we'll be in amazing and weird Las...
Real-Time Streaming Platform with Pulsar, Flink & ClickHouse
Real-Time Streaming Platform: Building Enterprise-Grade Data Infrastructure with Pulsar,...
How I Streamed Live Binance L2 Order Book Data on AWS for ~$10/Month
A fully automated Binance Level 2 order book streaming system on AWS Free Tier - 260K+ snapshots/day for ~$15/month with 6-person team access
Fixing Type Hints for Callable Objects with Custom Signatures in Dagster
So... it's been an interesting week. After my last contribution to Scikit-learn (which was honestly...
Building a Real-Time Data Platform with Kubernetes (Kind) - A Complete Local Setup Guide
Ever wondered how to build a production-grade real-time data pipeline that can handle millions of...
The Offline Data Engineer: Building Resilient API Pipelines that Work on an Airplane
Development loops for API integrations are usually painful. We’ve all been there: You are building a...
Building an Enterprise Patching Dashboard with AWS - A Complete Guide
Learn how to build a centralized patching and inventory management solution using AWS...
Real-Time Crypto Data Pipeline
Real-Time Crypto Data Pipeline: From Binance API to Cassandra with CDC and...
Polyglot Data Engineering: Python + Go in the Same Pipeline
Hey Devs 👋, If you're exploring modern data engineering stacks or curious about mixing languages in...
End-to-End Data Workflow: Kestra, Redshift, and dbt Integration
Imagine that at the end of every month, you are required to download data from a particular source,...
Apache Doris 4.0: One Engine for Analytics, Full-Text Search, and Vector Search
We're excited to announce the official release of Apache Doris 4.0: a major milestone release that...
Realtime Data Streaming Platform: Building a Unified Monitoring Stack
When you're running a real-time streaming platform processing 1 million messages per second, you...
A Deep Dive into Apache Spark Architecture
Introduction The digital world constantly generates enormous volumes of data — from social media...