🌩️ Cloud Databases and Data Management 📊
#cloud#database#tutorial#docker
Madhurima Rawat

Madhurima Rawat @madhurima_rawat

About: 🚀 Passionate about tech & growth | 💻 Data Scientist & Frontend Dev | C, C++, Python, R | 📊 InfluxDB, Grafana | 🤖 ML Enthusiast | 🐙 GitHub & Open Source | 📚 Learning & Sharing

Location:
Bhilai, Chhattisgarh, India
Joined:
Apr 11, 2024

🌩️ Cloud Databases and Data Management 📊

Publish Date: May 16
10 0

⚖️ Hello Balancers!

From cloud load to life load, you can balance it all! 🐘📦

📚 All code, docs, and resources are available in my GitHub repository:

GitHub logo madhurimarawat / Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Cloud-Computing

This repository focuses on cloud computing and demonstrates how to set up virtual machines, S3, and other services using LocalStack. It provides a comprehensive guide to simulating AWS services locally for development and testing purposes.

Repo Size GitHub Stars GitHub Forks GitHub Issues Closed Issues Open Pull Requests Closed Pull Requests GitHub Discussions GitHub Contributors Top Language License Last Commit Repository Age Workflow Status GitHub Watchers

Tools and Technologies ⚙️💻

1. AWS CLI

AWS Command Line Interface (CLI) is a powerful tool that allows users to interact with AWS services directly from the terminal. It simplifies managing cloud resources by providing commands for a wide range of AWS services, enabling tasks such as provisioning, managing, and automating workflows with ease.

2. LocalStack

LocalStack is a fully functional, local testing environment for AWS services. It enables developers to simulate AWS services on their local machines, facilitating the development and testing of cloud-based applications without needing access to an actual AWS account.

3. Docker

Docker is a containerization platform that allows developers to build, share, and run applications in isolated environments called…


View on GitHub


In the last post,

madhurima_rawat

⚖️ Cloud Load Balancing and Auto Scaling 📈

Madhurima Rawat ・ May 13

#cloud #tutorial #cloudcomputing #aws
we explored Cloud Load Balancing and Auto Scaling, the dynamic duo behind resilient cloud infrastructure.

Now, before we float too high in the clouds, let’s bring things down to earth and local.

In this post, we’ll set up a PostgreSQL database using Docker on your local machine.
It’s quick, clean, and the perfect way to prototype or develop cloud-ready apps without relying on live cloud resources.

No installations. No clutter. Just a few Docker commands and you're good to go. 🐳🧪

Let’s containerize our data journey from the ground up! 📦🔌

Cloud Databases and Data Management

Overview

This experiment covers setting up and managing cloud-based relational databases using PostgreSQL, Docker, and LocalStack. It provides hands-on experience with database management in a simulated cloud environment, replicating real-world cloud database operations using containerized solutions.

What are Databases?

Databases store, organize, and manage data efficiently. They are crucial in applications ranging from websites to large-scale enterprise systems.

Types of Databases

Databases are broadly categorized into:

  • Relational Databases (SQL-based) – Structured data stored in tables (e.g., PostgreSQL, MySQL).
  • NoSQL Databases – Flexible schema for handling unstructured data (e.g., MongoDB, Cassandra).
  • Cloud Databases – Managed services with scalable storage (e.g., AWS RDS, Google Cloud Firestore).

PostgreSQL Features

PostgreSQL is a powerful, open-source relational database with features like:

  • ACID Compliance – Ensures data integrity.
  • Extensibility – Supports custom functions and data types.
  • Scalability – Handles large volumes of data.

Real-World Application & Case Study

Use Case: Financial Data Management

A leading bank implemented PostgreSQL on the cloud to handle transaction processing, fraud detection, and real-time analytics. With Dockerized deployments, they achieved high availability, ensuring 99.9% uptime and enhanced security.

This experiment provides insights into deploying such systems using Docker and LocalStack to simulate real-world cloud database management.

🖼️ About the Cover Image:
It illustrates the journey of data in a cloud environment. On the left, a PostgreSQL database (🐘🗄️) stores structured information securely. That data flows into a brain (🧠), symbolizing the tools and technologies used for managing, querying, and analyzing the data, like Docker, pgAdmin, and SQL scripts. From there, the insights are transformed into visual representations, a pie chart (📊) and bar graph (📈) displayed on a blackboard (🧾), indicating how data is communicated, presented, and used for decision-making in cloud-native setups.

Database Operations with Postgres

1. Creating an RDS Instance Using LocalStack

Command: 

aws rds create-db-instance --db-instance-identifier mydb \
  --db-instance-class db.t3.micro \
  --engine mysql \
  --master-username admin \
  --master-user-password password \
  --allocated-storage 20 \
  --endpoint-url=http://localhost:4566
Enter fullscreen mode Exit fullscreen mode

Error Output: 

Could not connect to the endpoint URL: "http://localhost:4566/"
Enter fullscreen mode Exit fullscreen mode 
An error occurred (InternalFailure) when calling the CreateDBInstance operation:
API for service 'rds' not yet implemented or pro feature - please check
https://docs.localstack.cloud/references/coverage/ for further information
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • The command attempts to create an RDS instance in LocalStack.
  • --endpoint-url=http://localhost:4566 → Uses LocalStack instead of AWS.
  • Errors indicate:
    • LocalStack is either not running or misconfigured.
    • RDS API might not be fully implemented in the free version of LocalStack.

Output Breakdown:

  • Could not connect to the endpoint URL → LocalStack might not be running or accessible.
  • InternalFailure error → The RDS API might require LocalStack Pro for full functionality.

  • Possible Fixes:

    • Ensure LocalStack is running: 
    docker run --rm -d --name localstack_main -p 4566:4566 localstack/localstack

2. Starting a PostgreSQL Container

Command: 

docker start my-postgres
Enter fullscreen mode Exit fullscreen mode 
docker start postgres
Enter fullscreen mode Exit fullscreen mode

Error Output: 

Error response from daemon: No such container: my-postgres
Error: failed to start containers: my-postgres
Enter fullscreen mode Exit fullscreen mode 
Error response from daemon: No such container: postgres
Error: failed to start containers: postgres
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • The containers do not exist under the specified names.
  • Verify running containers with: 
  docker ps -a
Enter fullscreen mode Exit fullscreen mode
  • If needed, create a new container: 
  docker run --name my-postgres -e POSTGRES_USER=admin \
    -e POSTGRES_PASSWORD=password -e POSTGRES_DB=mydb \
    -p 5432:5432 -d postgres:15
Enter fullscreen mode Exit fullscreen mode

Output Breakdown:

  • No such container: my-postgres → The container was never created or was removed.
  • failed to start containers → The container name does not match any existing instances.
  • Possible Fixes:
    • Check existing containers: docker ps -a
    • Create and start a new PostgreSQL container using docker run (above).

3. Listing Available Docker Images

Command: 

docker images
Enter fullscreen mode Exit fullscreen mode

Output: 

| REPOSITORY            | TAG    | IMAGE ID     | CREATED      | SIZE   |
| --------------------- | ------ | ------------ | ------------ | ------ |
| my-flask-app          | latest | f5feae0ac7a4 | 6 hours ago  | 139MB  |
| flask-app             | latest | ae4054c49614 | 7 hours ago  | 139MB  |
| hackvortex-backend    | latest | 14e63c26d40b | 21 hours ago | 1.05GB |
| postgres              | 15     | e45d3f5ec589 | 7 days ago   | 430MB  |
| localstack/localstack | latest | b686f3948f42 | 6 weeks ago  | 1.18GB |
| python                | 3.9    | 9f98746e2033 | 3 months ago | 999MB  |
| nginx                 | latest | b52e0b094bc0 | 4 weeks ago  | 192MB  |
Enter fullscreen mode Exit fullscreen mode

Explanation:

  • Displays available images in the local Docker environment.
  • PostgreSQL (postgres:15) is available.
  • LocalStack (localstack/localstack) is present but needs verification (docker ps -a).

Output Breakdown:

  • postgres:15 is listed → The image exists but the container may not be running.
  • localstack/localstack exists → LocalStack is installed but may need to be started.

  • Possible Fixes:

    • Start PostgreSQL if not running: 
    docker run --name my-postgres -e POSTGRES_USER=admin \
  -e POSTGRES_PASSWORD=password -e POSTGRES_DB=mydb \
  -p 5432:5432 -d postgres:15
    • Ensure LocalStack is running: 
    docker start localstack_main

4. Starting a PostgreSQL Container

Command: 

C:\Users\rawat>docker start postgres
Enter fullscreen mode Exit fullscreen mode

Error Output: 

Error response from daemon: No such container: postgres
Error: failed to start containers: postgres
Enter fullscreen mode Exit fullscreen mode

5. Listing All Containers

Command: 

C:\Users\rawat>docker ps -a
Enter fullscreen mode Exit fullscreen mode

Output:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a10c5a71f625 localstack/localstack "docker-entrypoint.sh" 2 minutes ago Up 2 minutes (healthy) 127.0.0.1:4510-4560->4510-4560/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack-main
7f0fa023ac4f 3a669f02efff "python app.py" 7 hours ago Exited (255) 5 minutes ago 8080/tcp, 0.0.0.0:5002->5000/tcp backend2
9ff472da8892 3a669f02efff "python app.py" 7 hours ago Exited (255) 5 minutes ago 8080/tcp, 0.0.0.0:5001->5000/tcp backend1

6. Running a PostgreSQL Container

Command: 

C:\Users\rawat>docker run --name my-postgres -e
POSTGRES_USER=admin -e POSTGRES_PASSWORD=password -e
POSTGRES_DB=mydb -p 5432:5432 -d postgres:15
Enter fullscreen mode Exit fullscreen mode

Error Output: 

docker: Error response from daemon: driver failed
programming external connectivity on endpoint my-postgres
(feae7f0fb87909bde1853a7ddefa49bb518f11250e54304f75109
68f7a88cca1): Bind for 0.0.0.0:5432 failed: port is already allocated.
Enter fullscreen mode Exit fullscreen mode

7. Resolving Port Conflict and Running PostgreSQL on a Different Port

Command: 

C:\Users\rawat>docker run --name my-new-postgres -e
POSTGRES_USER=admin -e POSTGRES_PASSWORD=password -e
POSTGRES_DB=mydb -p 5433:5432 -d postgres:15
Enter fullscreen mode Exit fullscreen mode

Output: 

b2efdca3c6f0af6cf4154fce236f0b66b5efba0f4f9e14972c94b3e0a5afa9de
Enter fullscreen mode Exit fullscreen mode

8. Verifying Running Containers

Command: 

C:\Users\rawat>docker ps
Enter fullscreen mode Exit fullscreen mode

Output:

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2efdca3c6f0 postgres:15 "docker-entrypoint.s…" 42 seconds ago Up 41 seconds 0.0.0.0:5433->5432/tcp my-new-postgres
a10c5a71f625 localstack/localstack "docker-entrypoint.sh" 3 minutes ago Up 3 minutes (healthy) 127.0.0.1:4510-4560->4510-4560/tcp, 127.0.0.1:4566->4566/tcp, 5678/tcp localstack-main

9. Connecting to PostgreSQL and Performing SQL Operations

Command: 

C:\Users\rawat>docker exec -it my-
new-postgres psql -U admin -d mydb
Enter fullscreen mode Exit fullscreen mode

Output: 

psql (15.12 (Debian 15.12-1.pgdg120+1))
Type "help" for help.
Enter fullscreen mode Exit fullscreen mode

Creating a Table and Inserting Data: 

CREATE TABLE students (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100) NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL
);
INSERT INTO students (name, email) VALUES
('Alice Johnson', 'alice@example.com'),
('Bob Smith', 'bob@example.com'),
('Charlie Brown', 'charlie@example.com');
Enter fullscreen mode Exit fullscreen mode

Output: 

CREATE TABLE
INSERT 0 3
Enter fullscreen mode Exit fullscreen mode

10. Performing SQL Queries

Selecting Data: 

SELECT * FROM students;
Enter fullscreen mode Exit fullscreen mode

Output:

id Name Email
1 Alice Johnson alice@example.com
2 Bob Smith bob@example.com
3 Charlie Brown charlie@example.com

Updating Data: 

UPDATE students SET email = 'bob.smith@example.com'
WHERE name = 'Bob Smith';
Enter fullscreen mode Exit fullscreen mode

Output: 

UPDATE 1
Enter fullscreen mode Exit fullscreen mode

Deleting Data: 

DELETE FROM students WHERE name = 'Charlie Brown';
Enter fullscreen mode Exit fullscreen mode

Output: 

DELETE 1
Enter fullscreen mode Exit fullscreen mode

Selecting Data with a Condition: 

SELECT * FROM students WHERE name LIKE 'A%';
Enter fullscreen mode Exit fullscreen mode

Output:

id Name Email
1 Alice Johnson alice@example.com

Exiting PostgreSQL: 

\q
Enter fullscreen mode Exit fullscreen mode

📄 Want to see how it all worked step by step? Check it out here:

🔗 Experiment 7 Output (PDF)

🧠 Curious about how each command runs and responds? See the detailed input-output flow here:

🖥️ PostgreSQL + Docker Setup Flow (PDF)

🎉 And that’s a wrap on Cloud Databases and Data Management!

💡 We went hands-on with setting up PostgreSQL using Docker, a clean, fast, and containerized way to build and test database-backed apps locally before going cloud-native.

💬 Tried other tools like MySQL, MongoDB, or hosted services like Firebase? I’d love to hear what worked best for you and why!

📚 Know a helpful resource or walkthrough?. I’d love to include helpful links in the article for others to explore!

🔥 Coming up next:
We're diving into Cloud Security: Identity and Access Management (IAM), the guardrails of modern cloud architecture. 🔐☁️
Let’s talk users, roles, policies, and how to keep your cloud castle secure!

Comments 0 total

    Add comment