MASAI Gets Even Better at ECR 2024

One of the biggest radiology stories of 2023 was the release of impressive interim results from the MASAI study, a large-scale trial of AI for breast screening in Sweden. At ECR 2024, MASAI researchers put an emphatic cap on the conference by presenting final data indicating that AI could have an even bigger impact on mammography screening than we thought. 

If you remember, MASAI’s interim results were published in August in Lancet Oncology and showed that ScreenPoint Medical’s Transpara AI algorithm was able to reduce radiologist workload by 44% when used as part of the kind of double-reading screening program that’s common in Europe.

  • Another MASAI finding was that AI-aided screening had a 20% higher cancer detection rate than conventional double-reading with human radiologists, but the difference was not statistically significant. 

That’s all changed with the final MASAI results, presented at ECR on March 2 by senior author Kristina Lång, MD, of Lund University.

  • Lång presented data from 106k participants who were randomized to either screening with Transpara V. 1.7 or conventional double reading without AI.

Transpara triaged mammograms by giving them a risk score of 1-10, and only those classified as high risk received double reading; lower-risk mammograms got a single human reader. In the final analysis, AI-aided screening … 

  • Had a 28% higher cancer detection rate per 1k women (6.4 vs. 5.0), a difference that was statistically significant (p=0.002)
  • Detected more cancers 10-20 mm (122 vs. 79)
  • Detected more cancers of non-specific histologic type (204 vs. 155)
  • Detected 20 more non-luminal A invasive cancers and 12 more DCIS grade 3 lesions

The Takeaway

When combined with the Lancet Oncology data, the new MASAI results indicate that AI could enable breast radiologists to have their cake and eat it too: a lower workload with higher cancer detection rates. 

Real-World AI Experiences

Clinical studies showing that AI helps radiologists interpret medical images are great, but how well does AI work in the real world – and what do radiologists think about it? These questions are addressed in a new study in Applied Ergonomics that takes a deep dive into the real-world implementation of a commercially available AI algorithm at a German hospital. 

A slew of clinical studies supporting AI were published in 2023, from the MASAI study on AI for breast screening to smaller studies on applications like opportunistic screening or predicting who should get lung cancer screening

  • But even an AI algorithm with the best clinical evidence behind it could fall flat if it’s difficult to use and doesn’t integrate well with existing radiology workflow.

To gain insight into this issue, the new study tracked University Hospital Bonn’s implementation of Quantib’s Prostate software for interpreting and documenting prostate MRI scans (Quantib was acquired by RadNet in January 2022). 

  • Researchers described the solution as providing partial automation of prostate MRI workflow, such as helping segment the prostate, generating heat maps of areas of interest, and automatically producing patient reports based on lesions it identifies. 

Prostate was installed at the hospital in the spring of 2022, with nine radiology residents and three attending physicians interviewed before and after implementation, finding…

  • All but one radiologist had a positive attitude toward AI before implementation and one was undecided 
  • After implementation, seven said their attitudes were unchanged, one was disappointed, and one saw their opinion shift positively
  • Use of the AI was inconsistent, with radiologists adopting different workflows and some using it all the time with others only using it occasionally
  • Major concerns cited included workflow delays due to AI use, additional steps required such as sending images to a server, and unstable performance

The findings prompted the researchers to conclude that AI is likely to be implemented and used in the real world differently than in clinical trials. Radiologists should be included in AI algorithm development to provide insights into workflow where the tools will be used.

The Takeaway

The new study is unique in that – rather than focusing on AI algorithm performance – it concentrated on the experiences of radiologists using the software and how they changed following implementation. Such studies can be illuminating as AI developers seek broader clinical use of their tools. 

AI Models Go Head-to-Head in Project AIR Study

One of the biggest challenges in assessing the performance of different AI algorithms is the varying conditions under which AI research studies are conducted. A new study from the Netherlands published this week in Radiology aims to correct that by testing a variety of AI algorithms head-to-head under similar conditions. 

There are over 200 AI algorithms on the European market (and even more in the US), many of which address the same clinical condition. 

  • Therefore, hospitals looking to acquire AI can find it difficult to assess the diagnostic performance of different models. 

The Project AIR initiative was launched to fill the gap in accurate assessment of AI algorithms by creating a Consumer Reports-style testing environment that’s consistent and transparent.

  • Project AIR researchers have assembled a validated database of medical images for different clinical applications, against which multiple AI algorithms can be tested; to ensure generalizability, images have come from different institutions and were acquired on equipment from different vendors. 

In the first test of the Project AIR concept, a team led by Kicky van Leeuwen of Radboud University Medical Centre in the Netherlands invited AI developers to participate, with nine products from eight vendors validated from June 2022 to January 2023: two models for bone age prediction and seven algorithms for lung nodule assessment (one vendor participated in both tests). Results included:

  • For bone age analysis, both of the tested algorithms (Visiana and Vuno) showed “excellent correlation” with the reference standard, with an r correlation coefficient of 0.987-0.989 (1 = perfect agreement)
  • For lung nodule analysis, there was a wider spread in AUC between the algorithms and human readers, with humans posting a mean AUC of 0.81
  • Researchers found superior performance for Annalise.ai (0.90), Lunit (0.93), Milvue (0.86), and Oxipit (0.88)

What’s next on Project AIR’s testing agenda? Van Leeuwen told The Imaging Wire that the next study will involve fracture detection. Meanwhile, interested parties can follow along on leaderboards for both bone age and lung nodule use cases. 

The Takeaway

Head-to-head studies like the one conducted by Project AIR may make many AI developers squirm (several that were invited declined to participate), but they are a necessary step toward building clinician confidence in the performance of AI algorithms that needs to take place to support the widespread adoption of AI. 

Lunit’s Deal for Volpara and AI Consolidation

Is the long-awaited consolidation of the healthcare AI sector gaining steam? In a deal valued at close to $200M, South Korean AI developer Lunit announced a bid to acquire Volpara Health, a developer of software for calculating breast density and cancer risk. 

At first glance, the alliance seems to be a match made in heaven. Lunit is a well-regarded AI developer that has seen impressive results in clinical trials of its Insight family of algorithms for indications ranging from mammography to chest imaging. 

  • Most recently, Lunit received FDA clearance for its Insight DBT software, marking its entry into the US breast screening market, and it also raised $150M in a public stock offering. 

Volpara has a long pedigree as a developer of breast imaging software, although it has shied away from image analysis applications to instead focus on breast center operations and risk assessment, in particular by calculating breast density. 

  • Thus, combining Lunit’s concentration in image analysis with Volpara’s focus on operations and risk assessment enables the combined company to offer a wider breadth of products to breast centers.

Lunit will also be able to take advantage of the marketing and sales structure that Volpara has built in the US mammography sector (97% of Volpara’s sales come from the US, where it has an installed base of 2k sites). Volpara expects 2024 sales of $30M and is cash-flow positive.

The question is whether the acquisition is a sign of things to come in the AI market. 

  • As commercial AI sales have been slow to develop, AI firms have largely funded their operations through venture capital firms – which are notoriously impatient in their quest for returns.

In fact, observers at the recent RSNA 2023 meeting noted that there were very few new start-up entrants into the AI space, and many AI vendors had smaller booths. 

  • And previous research has documented a slowdown in VC funding for AI developers that is prompting start-up firms to seek partners to provide more comprehensive offerings while also focusing on developing a road to profitability. 

The Takeaway

It’s not clear yet whether the Lunit/Volpara deal is a one-off combination or the start of a renewed consolidation trend in healthcare AI. Regardless of what happens, this alliance unites two of the stronger players in the field and has exciting potential for the years to come. 

AI’s Incremental Revolution

So AI dominated the discussion at last week’s RSNA 2023 meeting. But does that mean it’s finally on the path to widespread clinical use? 

Maybe not so much. For a technology that’s supposed to have a revolutionary impact on medicine, AI is taking a frustratingly long time to arrive. 

Indeed, there was plenty of skepticism about AI in the halls of McCormick Place last week. (For two interesting looks at AI at RSNA 2023, also see Hugh Harvey, MD’s list of takeaways in a post on X/Twitter and Herman Oosterwijk’s post on LinkedIn.) 

But as one executive we talked to pointed out, AI’s advance to routine clinical use in radiology is likely to be more incremental than all at once. 

  • And from that perspective, last week’s RSNA meeting was undoubtedly positive for AI. Scientific sessions were full of talks on practical clinical applications of AI, from breast AI to CT lung screening

Researchers also discussed the use of AI apart from image interpretation, with generative AI and large language models taking on tasks from answering patient questions about their reports to helping radiologists with dictation.

It’s fine to be a skeptic (especially when it comes to things you hear at RSNA), but for perspective look at many of the past arguments casting doubt on AI: 

  • AI algorithms don’t have FDA clearance (the FDA authorized 171 algorithms in just the past year)
  • You can’t get paid for using AI clinically (16 algorithms have CPT codes, with more on the way) 
  • There isn’t enough clinical evidence backing the use of AI (tell that to the authors of MASAI, PERFORMS, and a number of other recent studies with positive findings)
  • The AI market is overcrowded with companies and ripe for consolidation (what exciting new growth market isn’t?)

The Takeaway

Sure, it’s taking longer than expected for AI to take hold in radiology. But last week’s conference showed that AI’s incremental revolution is not only advancing but expanding in ways no one expected when IBM Watson was unveiled to an RSNA audience a mere 6-7 years ago. One can only imagine what the field will look like at RSNA 2030.

Looking for more coverage of RSNA 2023? Be sure to check out our videos from the technical exhibit floor, which you can find on our new Shows page.

Reimbursement Drives AI Adoption

It’s no secret that insurance reimbursement drives adoption of new medical technology. But a new analysis in NEJM AI shows exactly how reimbursement is affecting the diffusion into clinical practice of perhaps the newest medical technology – artificial intelligence. 

Researchers analyzed a database of over 11B CPT claims from January 2018 to June 2023 to find out how often reimbursement claims are being submitted for the use of the over 500 AI devices that had been approved by the FDA at the time the paper was finalized. 

  • The authors chose to focus on CPT claims rather than claims under the NTAP program for new technologies because CPT codes are used by both public and private payors in inpatient and outpatient settings, while NTAP only applies to Medicare inpatient payments. 

They found 16 medical AI procedures billable under CPT codes; of these, 15 codes were created since 2021 and the median age of a CPT code was about 374 days, indicating the novelty of medical AI.

  • Also, only four of the 16 had more than 1k claims submitted, leading the authors to state “overall utilization of medical AI products is still limited and focused on a few leading procedures,” such as coronary artery disease and diabetic retinopathy.

The top 10 AI products and number of CPT claims submitted are as follows:

  1. HeartFlow Analysis for coronary artery disease (67,306)
  2. LumineticsCore for diabetic retinopathy (15,097)
  3. Cleerly for coronary atherosclerosis (4,459)
  4. Perspectum LiverMultiScan for liver MRI (2,428)
  5. Perspectum CoverScan for multiorgan MRI (591)
  6. Koios DS for breast ultrasound (552)
  7. Anumana for ECG cardiac dysfunction (435)
  8. CADScor for cardiac acoustic waveform recording (331)
  9. Perspectum MRCP for quantitative MR cholangiopancreatography (237)
  10. CompuFlo for epidural infusion (67)

While radiology may rule in terms of the sheer number of FDA-approved AI products (79% in a recent analysis), the list shows that cardiology is king when it comes to paying the bills. 

The Takeaway

Amid the breathless hype around medical AI, the NEJM AI study comes as a bit of a wake-up call, showing how the cold reality of healthcare economics can limit technology diffusion – a finding also indicated in other studies of economic barriers to AI

On the positive side, it shows that a rosy future lies ahead for those AI algorithms – like HeartFlow Analysis – that can make the leap.

FDA Data Show AI Approval Boom

In the previous issue of The Imaging Wire, we discovered how venture capital investment in AI developers is fueling rapid growth in new AI applications for radiologists (despite a slowdown this year). 

This trend was underscored late last week with new data from the FDA showing strong growth in the number of regulatory authorizations of AI and machine learning-enabled devices in calendar 2023 compared to the year before. The findings show:

  • A resurgence of AI/ML authorizations this year, with over 30% growth compared to 14% in 2022 and 15% in 2021 – The last time authorizations grew this fast was in 2020 (+39%)
  • The FDA authorized 171 AI/ML-enabled devices in the past year. Of the total, 155 had final decision dates between August 1, 2022 to July 30, 2023, while 16 were reclassifications from prior periods 
  • Devices intended for radiology made up 79% of the total (122/155), an impressive number but down slightly compared to 87% in 2022 
  • Other medical specialities include cardiology (9%), neurology (5%), and gastroenterology/urology (4%)

One interesting wrinkle in the report was the fact that despite all the buzz around large language models for generative AI, the FDA has yet to authorize a device that uses generative AI or that is powered by LLMs. 

The Takeaway

The FDA’s new report confirms that radiology AI shows no sign of slowing down, despite a drop in AI investment this year. 

The data also offer perspective on a JACR report last week predicting that by 2035 radiology could be seeing 350 new AI/ML product approvals for the year. Product approvals would only have to grow at about a 10% annual rate to hit that number – a figure that seems perfectly achievable given the new FDA report.

What’s Fueling AI’s Growth

It’s no secret that the rapid growth of AI in radiology is being fueled by venture capital firms eager to see a payoff for early investments in startup AI developers. But are there signs that VCs’ appetite for radiology AI is starting to wane?

Maybe. And maybe not. While one new analysis shows that AI investments slowed in 2023 compared to the year before, another predicts that over the long term, VC investing will spur a boom in AI development that is likely to transform radiology. 

First up is an update by Signify Research to its ongoing analysis of VC funding. The new numbers show that through Q3 2023, the number of medical imaging AI deals has fallen compared to Q3 2022 (24 vs. 40). 

  • Total funding has also fallen for the second straight year, to $501M year-to-date in 2023. That compares to $771M through the third quarter of 2022, and $1.1B through the corresponding quarter of 2021. 

On the other hand, the average deal size has grown to an all-time high of $20.9M, compared to 2022 ($15.4M) and 2021 ($18M). 

  • And one company – Rapid AI – joined the exclusive club of just 14 AI vendors that have raised over $100M with a $75M Series C round in July 2023. 

In a look forward at AI’s future, a new analysis in JACR by researchers from the ACR Data Science Institute (DSI) directly ties VC funding to healthcare AI software development, predicting that every $1B in funding translates into 11 new product approvals, with a six-year lag between funding and approval. 

  • And the authors forecast long-term growth: In 2022 there were 69 FDA-approved products, but by 2035, funding is expected to reach $31B for the year, resulting in the release of a staggering 350 new AI products that year.

Further, the ACR DSI authors see a virtuous cycle developing, as increasing AI adoption spurs more investment that creates more products available to help radiologists with their workloads. 

The Takeaway

The numbers from Signify and ACR DSI don’t match up exactly, but together they paint a picture of a market segment that continues to enjoy massive VC investment. While the precise numbers may fluctuate year to year, investor interest in medical imaging AI will fuel innovation that promises to transform how radiology is practiced in years to come.

How Vendors Sell AI

Better patient care is the main selling point used by AI vendors when marketing neuroimaging algorithms, followed closely by time savings. Farther down the list of benefits are lower costs and increased revenue for providers. 

So says a new analysis in JACR that takes a close look at how FDA-cleared neuroimaging AI algorithms are marketed by vendors. It also includes several warning signs for both AI developers and clinicians.

AI is the most exciting technology to arrive in healthcare in decades, but questions percolate on whether AI developers are overhyping the technology. In the new analysis, researchers focused on marketing claims made for 59 AI neuroimaging algorithms cleared by the FDA from 2008 to 2022. Researchers analyzed FDA summaries and vendor websites, finding:

  • For 69% of algorithms, vendors highlighted an improvement in quality of patient care, while time savings for clinicians were touted for 44%. Only 16% of algorithms were promoted as lowering costs, while just 11% were positioned as increasing revenue
  • 50% of cleared neuroimaging algorithms were related to detection or quantification of stroke; of these, 41% were for intracranial hemorrhage, 31% for stroke brain perfusion, and 24% for detection of large vessel occlusion 
  • 41% of the algorithms were intended for use with non-contrast CT scans, 36% with MRI, 15% with CT perfusion, 14% with CT angiography, and the rest with MR perfusion and PET
  • 90% of the algorithms studied were cleared in the last five years, and 42% since last year

The researchers further noted two caveats in AI marketing: 

  • There is a lack of publicly available data to support vendor claims about the value of their algorithms. Better transparency is needed to create trust and clinician engagement.
  • The single-use-case nature of many AI algorithms raises questions about their economic viability. Many different algorithms would have to be implemented at a facility to ensure “a reasonable breadth of triage” for critical findings, and the financial burden of such integration is unclear.

The Takeaway

The new study offers intriguing insights into how AI algorithms are marketed by vendors, and how these efforts could be perceived by clinicians. The researchers note that financial pressure on AI developers may cause them to make “unintentional exaggerated claims” to recoup the cost of development; it is incumbent upon vendors to scrutinize their marketing activities to avoid overhyping AI technology.

Understanding AI’s Physician Influence

We spend a lot of time exploring the technical aspects of imaging AI performance, but little is known about how physicians are actually influenced by the AI findings they receive. A new Scientific Reports study addresses that knowledge gap, perhaps more directly than any other research to date. 

The researchers provided 233 radiologists (experts) and internal and emergency medicine physicians (non-experts) with eight chest X-ray cases each. The CXR cases featured correct diagnostic advice, but were manipulated to show different advice sources (generated by AI vs. by expert rads) and different levels of advice explanations (only advice vs. advice w/ visual annotated explanations). Here’s what they found…

  • Explanations Improve Accuracy – When the diagnostic advice included annotated explanations, both the IM/EM physicians and radiologists’ accuracy improved (+5.66% & +3.41%).
  • Non-Rads with Explainable Advice Rival Rads – Although the IM/EM physicians performed far worse than rads when given advice without explanations, they were “on par with” radiologists when their advice included explainable annotations (see Fig 3).
  • Explanations Help Radiologists with Tough Cases – Radiologists gained “limited benefit” from advice explanations with most of the X-ray cases, but the explanations significantly improved their performance with the single most difficult case.
  • Presumed AI Use Improves Accuracy – When advice was labeled as AI-generated (vs. rad-generated), accuracy improved for both the IM/EM physicians and radiologists (+4.22% & +3.15%).
  • Presumed AI Use Improves Expert Confidence – When advice was labeled as AI-generated (vs. rad-generated), radiologists were more confident in their diagnosis.

The Takeaway
This study provides solid evidence supporting the use of visual explanations, and bolsters the increasingly popular theory that AI can have the greatest impact on non-experts. It also revealed that physicians trust AI more than some might have expected, to the point where physicians who believed they were using AI made more accurate diagnoses than they would have if they were told the same advice came from a human expert.

However, more than anything else, this study seems to highlight the underappreciated impact of product design on AI’s clinical performance.

Get every issue of The Imaging Wire, delivered right to your inbox.

You might also like..

Select All

You're signed up!

It's great to have you as a reader. Check your inbox for a welcome email.

-- The Imaging Wire team

You're all set!