Articles by Tag #dataengineering
Browse our collection of articles on various topics related to IT technologies. Dive in and explore something new!
The Ultimate Linux Command Cheat Sheet for Data Engineers and Analysts
Introduction As a data engineer or analyst, your day-to-day responsibilities likely...
Streaming SQL in Stateful DataFlows
Streaming SQL Functionality SQL Streaming Queries and Stream Processing Operations is...
All About Parquet Part 08 - Reading and Writing Parquet Files in Python
Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse...
Comprehensive LuxDevHQ Data Engineering Course Guide
This comprehensive course spans 4 months (16 weeks) and equips learners with expertise in Python,...
Open Source Data Engineering Landscape 2025
Alireza Sadeghi Reposted from...
When Small Parquet Files Become a Big Problem (and How I Ended Up Writing a Compactor in PyArrow)
It all began with a fairly normal data pipeline. Events were coming in through Kafka, landing in AWS...
Data Orchestration Tool Analysis: Airflow, Dagster, Flyte
Introduction Data orchestration tools are key for managing data pipelines in modern...
All About Parquet Part 05 - Compression Techniques in Parquet
Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse...
All About Parquet Part 10 - Performance Tuning and Best Practices with Parquet
Free Copy of Apache Iceberg the Definitive Guide Free Apache Iceberg Crash Course Iceberg Lakehouse...
🎥 Speed Up PostgreSQL Queries by 7x with Smarter Indexing
PostgreSQL is powerful, but scanning huge datasets can slow down queries. What if you could skip...
Designing robust and scalable relational databases: A series of best practices.
Throughout my experience as a data engineer and software developer, I've had the pleasure (and...
What Apache Iceberg REST Catalog is and isn't
Free Copy of Apache Iceberg: The Definitive Guide Free Apache Iceberg Crash Course I've recently...
How to Design a Relational Database Schema in 2025
Designing a good relational database schema is one of the most important steps when building any...
Engenharia de Dados com Scala: masterizando o processamento de dados em tempo real com Apache Flink e Google Pub/Sub
Note: this article is also available in english 🌎 O Apache Flink é um framework de processamento...
Understanding Apache Iceberg Delete Files
Free Copy of Apache Iceberg: The Definitive Guide Free Apache Iceberg Crash Course Apache Iceberg...
Ultimate guide to creating a pipeline(Apache Airflow)
Hello there data enthusiasts. Today's guide walks you through building a complete data pipeline using...
Serverless PDF Processing with AWS Lambda and Textract
Serverless computing has transformed the way we build applications by eliminating the need to manage...
Introdução à Engenharia de Dados
Antes de começar com o roadmap, acho importante discutir o que realmente é Engenharia de...
Create, Read, Update, Delete (CRUD) | MongoDB Tutorial 2025
Table of Contents Introduction to MongoDB Installation & Database Creation CRUD...
The Developer’s Guide to Real-Time Data Platforms!
The data landscape has expanded immensely, making it crucial for organizations to leverage data...
SQL Query Optimization for Data Engineers
Moving data efficiently can make the difference between a smooth system and a frustratingly slow one....
Jupyter Notebooks in Docker
Why use Docker for Jupyter Notebooks? Docker provides an efficient and reproducible...
The Apache Iceberg™ Small File Problem
If you've been following Apache Iceberg™ at all, you've no doubt heard whispers about "the small file...
SQLite Database GUI Tool | Design and Visualize with DbSchema
If you are looking for a way to design and manage your SQLite database, DbSchema helps you create...
Building an Automated Weather Data Pipeline with Apache Kafka and Cassandra
This article creates an end-to-end weather data pipeline that collects weather data from multiple...
Understanding Apache Iceberg's metadata.json file
Free Copy of Apache Iceberg: The Definitive Guide Free Apache Iceberg Crash Course ...
Understanding RAID Levels: A Comprehensive Guide to RAID 0, 1, 5, 6, 10, and Beyond
In today’s fast-paced digital landscape, data storage is crucial for safeguarding critical...
Get the Records after and before the Searched One — From SQL to SPL #18
Problem description & analysis: The ProductionLine_Number in a certain table of the...
Data Engineering in 2024: Innovations and Trends Shaping the Future
** As 2024 unfolds, data engineering is becoming more integral to organizational success than ever...
Data Engineering with Scala: Mastering Real-Time Data Processing with Apache Flink and Google Pub/Sub
Note: this article is also available in brazilian portuguese 🌎 Apache Flink is a distributed data...