When I first started diving into real-world data extraction, I wanted a project that would not only sharpen my web scraping skills but also deliver something useful, something with practical value to people looking for insights in a specific market. That’s when I turned my attention to Kenya’s real estate market.
Kenya’s housing sector, especially rental properties, is booming. With platforms like Property24 Kenya listing hundreds of new properties every week, there’s a wealth of data waiting to be tapped into. This article is a walk-through of how I built a Python-based web scraper that collects detailed listings of rental houses in Kenya and stores them in a structured CSV file for further analysis.
Why Scrape Property Listings?
Real estate listings provide rich, structured data that’s perfect for analysis. Whether you’re comparing pricing trends, examining location-based inventory or monitoring how property descriptions evolve over time, having access to a clean dataset makes all the difference.
Instead of manually copying property details from individual web pages (which would take forever), I wrote a script that does all the heavy lifting automatically.
Tools & Libraries Used
For this project, I stuck to Python and some of its most reliable libraries for web scraping and data handling:
• requests: To fetch web page contents.
• BeautifulSoup: For parsing the HTML.
• pandas: To structure and clean the data and finally save it as a CSV.
What the Script Does
Loops Through Pages
It starts from the first page of rental listings and increments through the pagination by updating the current_page variable. It checks if listings are found, if not, the loop breaks and scraping stops.
Parses and Extracts Key Information
Each rental listing is parsed to extract:
• Title
• Location
• Bedrooms
• Bathrooms
• House Size
• Description
• Price
All this information is stored as a dictionary and added to a list.
Cleans and Saves the Data
Finally, the list of dictionaries is converted into a pandas DataFrame. Empty rows (where all values are None) are dropped, and the dataset is saved to a CSV file named kenyarentalsfile1.csv.
Real-Life Use Cases
So, why is this kind of project useful?
• Market Analysis: Realtors and housing investors can use the dataset to understand trends.
• Academic Research: Urban planners and students can analyze spatial housing availability.
• Price Forecasting Models: Data scientists can train regression models on historical rental price trends.
• Business Intelligence Dashboards: You can build a Power BI or Tableau dashboard from the CSV file.
Challenges Faced
No project is without hurdles. Here are a few I encountered:
- Inconsistent HTML Structure Some listings didn’t include all the expected fields like bedrooms or bathrooms. I solved this by wrapping each field in a conditional check.
- Pagination Handling Initially, the script kept running endlessly because I didn’t break the loop after reaching an empty page. Adding a check for an empty all_rentals list fixed this.
- Keyboard Interrupts / Timeouts Sometimes, the requests.get() call takes too long or hangs. This can be mitigated using timeouts and try-except blocks (a possible future enhancement).
Future Enhancements
• Add timeout & retry logic for better reliability
• Integrate logging to replace print statements
• Export to database for scalability
• Add filters like city-specific scraping or price ranges
• Build a dashboard using the cleaned CSV file
Final Thoughts
Building this scraper was a rewarding experience. It not only gave me real-world exposure to web scraping techniques but also taught me how to handle edge cases, optimize data collection, and store clean, structured data for analysis.
If you’re learning Python, this is a great beginner-to-intermediate level project to work on. It touches on many important concepts like iteration, HTML parsing, data cleaning, and file I/O. Plus, it’s highly practical. You will end up with a dataset that can be analyzed and visualized.
Check Out the Project
GitHub Repository
Feel free to improve the code. Contributions are welcome!
Good Job!