Introduction

Imagine a world where anyone-doctors, researchers, or even patients-can upload a medical image or report and instantly receive a detailed, AI-powered analysis. That's the vision behind my Medical Diagnostic Tool, a project that brings together deep learning, natural language processing, and a user-friendly web interface to make advanced diagnostics accessible to everyone.

In this post, I'll walk you through how I built this tool, the technology behind it, and how you can use or extend it for your own needs. Whether you're a developer, a healthcare professional, or just curious about AI in medicine, you'll find practical insights, code snippets, and visual diagrams to make everything clear-even if you're a complete beginner.

Why Build a Medical Diagnostic Tool?

Healthcare is full of data-X-rays, CT scans, lab reports, and more. But interpreting this data quickly and accurately is a huge challenge, especially in busy clinics or underserved areas. AI can help bridge this gap by:

• Speeding up diagnosis
• Reducing human error
• Providing a "second opinion" for clinicians
• Empowering patients with understandable results

My goal was to create a tool that's not just powerful, but also easy to use and open for anyone to improve.

System Overview: How Does It All Work?

Here's a high-level look at the system:

• Frontend: Clean, mobile-first React app for uploading files and viewing results.

• Backend: FastAPI server that routes requests to the right analysis pipeline.
• Imaging Pipeline: Deep learning models for X-rays, CT, MRI, and ultrasound.
• Text Pipeline: NLP models for extracting info from medical reports and lab results.
• Results: Structured findings, confidence scores, and visual explanations.

Step-by-Step: From Upload to Diagnosis

1. Uploading Your Data

Users can upload images (like X-rays or MRIs) or PDF reports. The interface is designed to be as simple as possible-just drag, drop, or snap a photo with your phone.

Mobile-first design ensures it works great on any device.

2. Smart Preprocessing

The backend automatically detects the type of file and its medical modality (e.g., X-ray vs. MRI). Images are resized, normalized, and prepared for the AI models. PDFs are scanned for text using OCR.

3. AI Analysis

• Imaging: The tool uses pre-trained deep learning models (DenseNet, EfficientNet, ResNet) fine-tuned on medical datasets. It can spot abnormalities, highlight them with heatmaps, and estimate confidence.
• Text: For reports, the system uses advanced NLP (like BioClinicalBERT) to extract diagnoses, lab values, and recommendations. It even flags abnormal lab results based on reference ranges.

4. Results & Visualization

Results are shown in a clear, structured format:

• Primary Findings: Most likely diagnoses, with confidence scores.
• Secondary Findings: Other possible issues or notes.
• Differential Diagnosis: Ranked list of possible conditions.
• Heatmaps: Visual overlays on images to show areas of concern.
• Lab Value Flags: Highlights abnormal results in red.

All results are explained in plain language, so anyone can understand them.

Under the Hood: Key Technologies

Backend (FastAPI + Python)

• FastAPI for blazing-fast, async APIs.
• Pydantic for data validation.
• Torch, torchvision, timm for deep learning.
• Transformers for NLP.
• PIL, OpenCV, PyMuPDF for image and PDF processing.

Example: Loading an X-ray Model

import timm
import torch

model = timm.create_model('densenet121', pretrained=True)
model.classifier = torch.nn.Linear(model.classifier.in_features, 14)  # 14 chest conditions
model.load_state_dict(torch.load('models/weights/chexpert_densenet121.pt'))
model.eval()

Imaging Pipeline

• Handles X-ray, CT, MRI, and ultrasound.
• Uses transfer learning and custom fine-tuning.
• Generates heatmaps for explainability.
• Estimates uncertainty with Monte Carlo dropout.

Example: Generating a Heatmap

def generate_heatmap(image, class_idx):
    # Get model output
    output = model(image)

    # Setup for gradient computation
    model.zero_grad()
    output[0, class_idx].backward(retain_graph=True)

    # Get gradients flowing back to last convolutional layer
    gradients = self.get_activation_gradients()
    activations = self.get_activations().detach()

    # Pool gradients across color channels
    weights = torch.mean(gradients, dim=[0, 2, 3])

    # Create heatmap by weighted combination of activation maps
    cam = torch.zeros(activations.shape[1:], dtype=torch.float32)
    for i, w in enumerate(weights):
        cam += w * activations[0, i, :, :]

    # Apply ReLU and normalize
    cam = np.maximum(cam.cpu(), 0)
    cam = cam / np.max(cam)

    # Resize and overlay on original image
    return self.overlay_heatmap(cam, original_image)

Text Pipeline

• Extracts text from PDFs with OCR.
• Named Entity Recognition (NER) for diagnoses, labs, and more.
• Classifies report types (radiology, pathology, labs).
• Flags abnormal lab values.

Example: Lab Value Extraction

reference_ranges = {
    "hemoglobin": {"unit": "g/dL", "low": 12.0, "high": 16.0},
    "wbc": {"unit": "K/µL", "low": 4.5, "high": 11.0},
    "platelets": {"unit": "K/µL", "low": 150, "high": 450},
    "glucose": {"unit": "mg/dL", "low": 70, "high": 100},
    "creatinine": {"unit": "mg/dL", "low": 0.6, "high": 1.2}
}

def is_abnormal(lab_name, value):
    if lab_name.lower() in reference_ranges:
        range_info = reference_ranges[lab_name.lower()]
        return value < range_info["low"] or value > range_info["high"]
    return False

Frontend (React + Tailwind CSS)

• Drag-and-drop file upload
• Progress bars and loading states
• Responsive, touch-friendly UI
• Results displayed with clear headings, tables, and color-coded highlights

Real-World Use Cases

For Clinicians

• Get a quick "second opinion" on tricky cases
• Screen large patient populations efficiently
• Standardize interpretation across different providers
• Prioritize urgent cases through automated triage

For Researchers

• Analyze large imaging datasets quickly
• Create annotated training materials
• Benchmark against expert diagnoses
• Generate standardized case presentations

For Patients

• Better understand your medical images and reports
• Get context for interpreting test results
• Access educational content related to your condition
• Have documentation for seeking second opinions

How to Get Started

1.Install

cd medical-diagnostic-tool
pip install -r requirements.txt
npm install

2. Run the Backend

cd medical_service
uvicorn main:app --reload

3. Run the Frontend

cd ..
npm start

4. Try the API

You can also use the REST API directly:

// Example JavaScript integration
async function analyzeMedicalImage(imageFile) {
  const formData = new FormData();
  formData.append('file', imageFile);
  formData.append('file_type', 'image');
  formData.append('analysis_type', 'comprehensive');

  const response = await fetch('https://your-api-endpoint/analyze', {
    method: 'POST',
    body: formData
  });

  return await response.json();
}

Customization & Extensibility

• Add new models: Drop your own PyTorch weights in the models/weights folder.
• Change reference ranges: Edit the reference_ranges dictionary in the text pipeline.
• Integrate with hospital systems: The API is ready for integration with EMR/EHR platforms.
• Translate the UI: The frontend supports multiple languages.

What's Next?

Model Improvements

• Integration of Vision Transformer architectures
• Support for 3D volumes (CT/MRI series)
• Fine-tuning on specialized datasets

Better Explainability

• More advanced visualization techniques
• Natural language explanations of findings
• Links to relevant clinical guidelines

Integration Capabilities

• DICOM compatibility for medical imaging
• FHIR-compliant data exchange
• EMR/EHR system plugins

Final Thoughts

AI won't replace doctors—but it can make their work faster, more accurate, and more accessible. My hope is that this tool inspires others to build, share, and improve open medical AI for everyone.

With the right approach, AI can democratize access to medical diagnostics, helping clinicians make better decisions and patients understand their health better.

Disclaimer: This tool is for research and educational purposes only. It should not be used as the sole basis for clinical decisions without professional oversight.

If you found this helpful, please share or leave a comment!