Ever wondered how real-world security teams monitor and analyze logs across systems? Let’s build a Mini SIEM using open-source tools: Elasticsearch, Logstash, Kibana (ELK), along with Filebeat and Winlogbeat for log forwarding.
What's the ELK Stack?
The ELK Stack is a powerful open-source platform for managing and analyzing large-scale logs in real time.
| Tool | Purpose |
|---|---|
| Elasticsearch | Stores and indexes log data for fast search and analytics |
| Logstash | Ingests, parse, and transform logs before sending to Elasticsearch |
| Kibana | Visualizes and queries log data using interactive dashboards |
Architecture of Mini SIEM
We simulate a real-world SOC(Security Operations Center) environment using multiple VMs:
- VM1(Ubuntu Server): Runs Elasticsearch, Logstash, and Kibana
- VM2(Ubuntu Server): Sends logs via File beat (CSV and Apache Logs)
- VM3(Windows 10): Sends Windows Event Logs via Winlogbeat
- Host Machine: Access Kibana Dashboards
Component Setup(Quick Links)
| Component | Setup Guide |
|Elasticsearch |01-install-elasticsearch.md https://susheelthapa.github.io/mini-siem-elk/setup/01-install-elasticsearch.html |
|Kibana |02-install-kibana.md |
| Logstash | 03-install-logstash.md|
| File beat | 04-integrate-filebeat.md |
|Winlogbeat|05-integrate-winlogbeat.md |
Integrations Highlights
- Filebeat → ELK(CSV + Apache Logs)
- Filebeat monitors CSV/Apache logs on VM2.
- Sends data to Logstash for enrichment(Grok).
- Logstash forwards it to Elasticsearch.
- Kibana visualize the log pattern, anomalies, or insights.
- Winlogbeat → ELK Stack(Windows Event Logs)
- Collects security, application, and system logs.
- Useful for tracking login attempts, system changes, and malware activity.
Key Features
- Full ELK Stack setup (locally and across VMs)
- Real-time log ingestion using Filebeat/Winlogbeat
- Custom parsing & filtering with Logstash
- Apache, Windows, and CSV logs visualized
- Port forwarding to enable access from host to VM
- Basic dashboards and alert-ready architecture
Common Pitfalls & How we solved them
| Problem | Solution |
|---|---|
| Missing enrollment token | Re-read initial Elasticsearch log and store credentials securely. |
| Port access issues | Used VirtualBox NAT Port Forwarding(9200, 5601) |
| Logstash connection errors | Verified SSL settings and path to http_ca_cert
|
| Filebeat/Winlogbeat misconfig | Ensured correct log path, IPs and disabled unnecessary outputs |
| Index not visible in Kibana | Created new Data Views in Stack Management |
What's Next?
This is just the beginning. Here's how you can take this Mini SIEM further:
- Add alerting rules using ElastAlert or Kibana Watcher
- Integrate Metricbeat for system metrics.
- Ingest Firewall/Syslog/NIDS logs for deeper security analysis
- Combine with Wazuh for host intrusion detection.
Final Thoughts
If you’re an aspiring SOC analyst, threat hunter, or DevSecOps engineer, this project gives you a working foundation of a SIEM system without spending money on enterprise tools. It's also great for homelabbers and cybersecurity students!
If this helped you, feel free to star the GitHub Repository or share it with your team. Let's make security engineering fun and accessible! 🔐⚙️🚀