Next generation medical image interpretation with MedGemma 1.5 and medical speech to text with MedASR

Improved performance for medical imaging use cases

MedGemma was designed from the ground up as a multimodal model, reflecting the multimodal nature of medicine. MedGemma 1 included support for interpreting two-dimensional medical images, including chest X-rays, dermatology images, fundus images and histopathology patches.

With MedGemma 1.5, we are expanding support for high-dimensional medical imaging, starting with three-dimensional volume representations of CT imaging and MRI, as well as whole-slide histopathology imaging. Developers can create applications in which multiple slices (for CT or MRI) or multiple patches (for histopathology) are provided as input along with a prompt that describes the task.

On internal benchmarks, the baseline absolute accuracy of MedGemma 1.5 improved by 3% over MedGemma 1 (61% vs. 58%) on classification of disease-related CT findings and by 14% (65% vs. 51%) on classification of disease-related MRI findings, averaged over findings. Additionally, on an internal diverse benchmark of histopathology slides and associated findings, the fidelity of MedGemma 1.5’s predictions, based on ROUGE-L score on cases with exactly one histopathology slide, improved by 0.47 over MedGemma 1 (0.49 vs. 0.02), matching the 0.498 score achieved by the task-specific PolyPath model.

This new high-dimensional support is the natural evolution of CT foundation, our previous API-based tool for generation of CT embeddings. To our knowledge, MedGemma 1.5 is the first public release of an open multimodal large language model that can interpret high-dimensional medical data while also retaining the ability to interpret general 2D data and text. Although these capabilities are in their early stages and remain imperfect, developers will achieve improved results by fine-tuning MedGemma models on their own data, and we hope to continually improve MedGemma models over time. We’ve released tutorial notebooks that illustrate how to use this high dimensional image capability for CT (Hugging Face, Model Garden) and histopathology (Hugging Face, Model Garden).