MIT researchers reviewed nearly 4,000 studies from PubMed Central and found code sharing variable and inadequate.
“The choices we make affect the results, and a methods section in a paper can only capture so much detail,” said Tom Pollard, a research scientist at MIT’s Laboratory for Computational Physiology and Computer Science and Artificial Intelligence Laboratory.
The findings, published in Nature Medicine, are informing TRIPOD-Code, a reporting guideline under development to define what researchers should share when they release analytical code.
Clinical prediction models estimate patient outcomes such as disease risk and directly inform care decisions, making scrutiny of their construction essential. Yet the review revealed wide variation in sharing rates across journals — ranging from zero to more than 70 percent — and showed that written code-sharing policies alone did not guarantee compliance.
The team identified studies citing the TRIPOD or TRIPOD+AI checklists, which guide reporting of prediction model development and validation. Full texts were drawn from PubMed Central’s open-access collection, yielding almost 4,000 papers.
“The challenge was that the information was highly inconsistent. First, we had to identify whether or not a paper shared code,” said Thomas Sounack, the study’s first author. He added that authors “discuss and link to code from a paper in many different ways.”
An LLM screened papers for code-availability statements and extracted repository links. Software then retrieved the linked repositories for assessment against criteria including documentation completeness, software requirements, and licensing terms.
“The real benefit of LLMs is their ability to process unstructured information such as text. These models helped us work with information that was scattered across papers and software repositories, rather than neatly organized in a table,” Pollard said. “That allowed us to examine how code was being shared in much more detail across a large collection of studies.”
Two researchers independently checked 500 papers and a separate sample of repositories, resolving disagreements before comparing their assessments with the automated output. Agreement was stronger for concrete characteristics — such as the presence of a license file — than for judgments about documentation adequacy. The authors emphasize overall patterns rather than exact counts because classification errors remain possible.

Where code was shared, repository contents varied widely. “Some repositories were extremely well-organized and -documented, while others gave almost no guidance at all,” Sounack said. Good code sharing, he explained, means “giving enough information to understand what the code does and be able to run it to reproduce the results reported in the paper.”
The review also surfaced effective practices, such as providing synthetic datasets to enable code reuse when original patient records cannot be made public.
TRIPOD-Code builds on the existing TRIPOD and TRIPOD+AI checklists. Through an expert review process, the team is developing guidance on which elements of shared analytical code should be considered essential.
“We want to give researchers practical guidance on how to share code, and give journals clearer expectations to communicate to authors,” Pollard said. “The review gives us evidence to inform those discussions.”
The open-access findings are available in Nature Medicine, and the TRIPOD-Code protocol is published at PubMed Central.
Related reading