MedicalAI — Light Weight

Try It

Runs offline · research prototype

There's no live web demo here, on purpose. The point of this model is to run offline on modest hardware — but more importantly, it's an early research prototype, not a validated classifier. Putting a medical-looking "diagnosis" button on a portfolio page would overstate what it can honestly do. Below is why it exists and how it's built.

A learning and research artifact, not a medical device, and not intended for real clinical use.

Why I Made It

This started as a challenge on a Discord server, framed as a research project: build an AI from scratch that could give diagnosis suggestions to people living in rural areas. The brief came with hard constraints, not nice-to-haves — it had to be light-weight, run with no internet connection, and work on the lowest-spec hardware possible. Those are exactly the conditions where a cloud model is useless: no reliable connectivity, no powerful machine, and no nearby specialist.

It's also my first project working with vision-language models. Medical triage is inherently multimodal — a scan means little without the patient's symptoms, and symptoms mean more when paired with an image — so a VLM approach fit the problem, and gave me a real reason to learn how to combine an image encoder and a text encoder into a single decision.

What It Does

You give it a chest X-ray and a short description of the patient's symptoms. The model encodes both, fuses them, and predicts a likely diagnosis from the conditions it was trained on. Crucially, it also knows when to stay quiet: if its confidence is below a threshold, it flags the case as inconclusive and recommends follow-up rather than guessing. In a rural, no-specialist setting, an honest "this needs a human" is as important as a confident answer.

How It's Built

The core is a fusion model. Chest X-ray images are encoded with CLIP's vision encoder; the symptom text is encoded with Bio_ClinicalBERT, a BERT model pre-trained on clinical notes. Both pretrained encoders are frozen, and a small trainable classifier head sits on top of their concatenated embeddings — so training is fast and light, which is the whole point. It learns on the Indiana University Chest X-ray Collection, pairing each scan's indication (symptoms) with its impression (diagnosis). At inference the softmax confidence is checked against a threshold to decide between a prediction and an "inconclusive / needs follow-up" flag.

Architecture

  MedicalAI — Light Weight — multimodal fusion
  ═════════════════════════════════════════════════

   chest X-ray                         symptom text
   ┌──────────┐                        ┌──────────────────┐
   │  image   │                        │ "persistent cough│
   │  .jpg    │                        │  and fever..."   │
   └────┬─────┘                        └────────┬─────────┘
        │                                       │
        ▼                                       ▼
   ┌──────────────┐                    ┌────────────────────┐
   │ CLIP vision  │  (frozen)          │ Bio_ClinicalBERT   │  (frozen)
   │ encoder      │                    │ text encoder       │
   └──────┬───────┘                    └─────────┬──────────┘
          │ image vector                         │ symptom vector
          └───────────────┬──────────────────────┘
                          ▼
                 ┌──────────────────┐
                 │  concatenate     │
                 │  [image ‖ text]  │
                 └────────┬─────────┘
                          ▼
                 ┌──────────────────┐
                 │ classifier head  │  ← the only trained part
                 │ (small MLP)      │
                 └────────┬─────────┘
                          ▼
                 softmax over conditions
                          │
             ┌────────────┴─────────────┐
             ▼                          ▼
   confidence ≥ threshold      confidence < threshold
      → diagnosis suggestion      → "inconclusive,
                                      needs follow-up"

  No network. Frozen encoders + tiny head = low footprint.

Freezing the pretrained encoders and training only a small head keeps compute and memory low enough to run offline on modest hardware — the rural-care constraint driving the whole design.

Dev Notes

Problem Solved

Diagnostic help where there is no connectivity, no specialist, and no powerful machine — a fully offline model small enough to run on low-end hardware.

New Tech Learned

My first vision-language model: combining a frozen image encoder (CLIP) and a frozen clinical-text encoder (Bio_ClinicalBERT) into one classifier, and using a confidence threshold to abstain instead of over-committing.

What's Next

More conditions and data, better calibration of the confidence threshold, and shrinking the runtime footprint further. It remains a research/learning project — not a medical device.