Inverse AI
Research · Sight

Did the model look at the retina?

Toward Fair Explainable AI in Medical Imaging: Mask-Aware Benchmarking of Grad-CAM and SHAP for Diabetic Retinopathy. IEEE, 2026.

Saeed Thabit and colleagues · United Arab Emirates University, College of Computing and Artificial Intelligence, with the INDUCE lab.

The question

A fundus photograph is graded on a five-step scale, from no retinopathy to proliferative disease. A model can return that grade in a moment. The clinician’s question is narrower: which part of the eye carried the decision, and did the highlighted region actually matter?

The study answers with two maps of the same prediction. Grad-CAM shows the region the network used. SHAP assigns each pixel a signed push, for the grade or against it. Both maps are then restricted to the retinal disc, so credit that spills onto the dark corners isn’t counted.

  1. Fundus image
  2. Pre-processing
  3. Trained model
  4. Grade
  5. Grad-CAM and SHAP
  6. Retinal-disc mask
  7. Report
Results
82.2%
accuracy on a 550-image test set from APTOS 2019
0.896
quadratic weighted kappa
0.290 → 0.206
Grad-CAM mean border ratio, once the score is kept inside the retinal disc

The two explanation methods are stronger on different grades.

A moderate diabetic retinopathy photograph through resize, contrast enhancement, background subtraction, and a circular crop.
One photograph as the model sees it. The circular crop is the region the explanation is later scored on.
Four fundus photographs beside Grad-CAM heatmaps. Three predictions match the true grade. One moderate case is predicted severe.
Grad-CAM on four test cases. A green frame is a grade that matched; a red frame is one that didn’t. The moderate case graded severe is the miss.
The same four cases with SHAP maps. Colour shows pixels that support or oppose each grade.
SHAP on the same cases. Red supports the grade named above the map; blue argues against it. The maps are faint because credit is spread across the retina, which is part of what the comparison measures.
Limits

What this study does not show.

APTOS grades the whole image and does not outline lesions. A heatmap is not yet proof that the model found a microaneurysm, a haemorrhage, or an exudate. That is the limit of this study, and the next measurement the work needs.

Not a screening deviceThis study explains a grade. It is not a medical device, and this site does not accept images.

Have lesion-annotated data or a clinical question?

Lesion-level validation is the next step for this work. We’d like to hear from clinicians and dataset holders.

Collaborate All research