NeuroVFM: Advancing Medical Imaging Through Health System Learning

Researchers at the University of Michigan have developed NeuroVFM, a generalist medical foundation model designed to interpret neuroimaging data directly from routine clinical operations. By training on over 560,000 uncurated CT and MRI scans, the model demonstrates the ability to learn diagnostic features without the need for manual labeling, marking a shift toward more scalable and efficient AI in radiology.

Understanding Health System Learning

Traditional machine learning in medicine often relies on carefully curated datasets tailored to specific diseases, such as isolating brain tumors or Alzheimer’s markers. In contrast, “health system learning” leverages the vast, uncurated data already residing within hospital picture archiving and communication systems (PACS).

According to the research team, this approach allows for the development of generalist medical foundation models that emerge from the environment of routine care rather than from internet-scale pretraining. By utilizing the UM-NeuroImages dataset, the team processed 566,915 neuroimaging studies, including both CT and MRI modalities, to train the NeuroVFM encoder. This method bypasses the bottleneck of requiring paired radiology reports for every image during the initial learning phase, allowing the model to grasp the shared anatomy of the head, brain, and neck through self-supervision.

Training Methodology and Vol-JEPA

The core of NeuroVFM is the use of a Joint-Embedding Predictive Architecture (JEPA), specifically adapted for volumetric neuroimaging, referred to as Vol-JEPA. Instead of attempting to reconstruct every voxel, the model predicts representations of masked 3D target patches based on visible context patches.

This objective encourages the model to learn semantic features efficiently. The team utilized FlashAttention-2 to handle variable-length sequences of image data, ensuring the model remains robust to different patient orientations and scan types. During the training process, the model was exposed to various windowing presets for CT scans—specifically brain, subdural, and bone—to ensure it could interpret the different contrasts required for clinical diagnosis.

Evaluating Diagnostic Grounding

Evaluating how an AI model “grounds” its diagnosis—or identifies the specific region of an image that supports a clinical finding—is a significant challenge in medical AI. Because most neurological pathologies occupy a small portion of a full scan, standard study-level labels provide only weak supervision.

To address this, the researchers implemented an attention-based multiple instance learning (AB-MIL) framework with a “classify-then-aggregate” design. This allows the model to produce interpretable, label-specific attention maps. By reversing the standard aggregation order, the developers were able to disentangle where the model looks from how that region contributes to a specific diagnosis, providing a scalable way to evaluate the model’s clinical reasoning without requiring pixel-level annotations.

Clinical Utility and Future Directions

While these results are promising, the research team emphasizes that NeuroVFM is currently a research tool. The model remains susceptible to biases inherent in its training data and architecture, and it is intended to support, rather than replace, clinical decision-making.

Key Takeaways

• Data Scale: NeuroVFM was trained on 566,915 unique neuroimaging studies, significantly increasing the breadth of data compared to traditional disease-specific models.
• Self-Supervision: By using the Vol-JEPA objective, the model learns anatomical and pathological features without needing manual labels for every scan.