This is a submission for the Redis AI Challenge: Real-Time AI Innovators.
What I Built
This project began as a simple RSS feed visualizer on a map. Over time, it evolved into a real-time political news intelligence platform powered by Redis, Telegram ingestion, NLP, and vector search.
The system is split into three main parts:
- Telegram Scraper – collects news data, extracts named entities, and generates vector embeddings.
- API – handles semantic search, clustering, timeline analysis, and pushes real-time updates via WebSockets.
- UI – built with React + Vite + Leaflet to display the interactive map, article data, and activity timeline.
The (Very Rough) Dev Journey
Initial Proof of Concept
- Scrape data from RSS feeds.
- Normalize data for each source.
- Run scraper on a schedule.
- Use NLP (spaCy) for NER and geo-tagging (for mapping).
- Generate article embeddings with
all-MiniLM-L6-v2for search and clustering. - Build a basic FastAPI backend for data retrieval.
- Use React + Leaflet.js for the map UI.
- Visualize activity with free country polygons and hover states.
- Sidebar displays articles per country interaction.
- 24-hour cumulative article count shown as activity.
- Store data in postgresql.
Second Phase
- Switched to real-time ingestion via Telegram using Telethon (no more scheduling).
- Normalized data once.
- Upgraded NER using
spacy/en_core_web_trf. - Retained embedding generation for clustering.
- Started storing processed data in Redis.
- Updated API to use Redis as the data source.
Third Phase
- API now provides article counts grouped by hour to power a timeline.
- Added NLP-based semantic search.
- Search initially ran in-memory using the same embedding model (response time ~4–5 seconds).
Fourth Phase
- Switched Redis storage to use
HSET,ZADD, and byte arrays for vector search. - Created Redisearch index for fast KNN similarity lookup.
- Improved search response time to ~0.5 seconds.
- Switched activity metric from cumulative to time-weighted scoring.
- Implemented similar article detection via embeddings.
- Cleaned up and improved the UI.
- Added map legend for better UX.
Fifth Phase
- Final UI cleanup and layout polish.
- Integrated Redis Pub/Sub for real-time notifications via WebSockets.
- UI now receives live article alerts.
- Refactored scraper and API codebases for maintainability.
Demo
Source Code
https://github.com/Shaikhhas/RedisDemo_PoC
Redis Usage
- FT.SEARCH + KNN for vector-based semantic search and clustering.
- ZADD and sorted sets for timeline activity.
- Pub/Sub for pushing real-time notifications to the frontend.
Personal Wishlist / Future Ideas
- Merge notification data into the main dataset
- Precompute per-hour article counts for performance
- Implement lazy loading or paging
- Add dark mode
- Add "Follow" functionality for developing stories
- Expose a scraper ingestion API for other sources
This was a solo project built over about a week, based on a prior prototype.