Tag: AI

Radiation and Cancer Risk

Posted on by Brian Casey

New research on the cancer risk of low-dose ionizing radiation could have disturbing implications for those who are exposed to radiation on the job – including medical professionals. In a new study in BMJ, researchers found that nuclear workers exposed to occupational levels of radiation had a cancer mortality risk that was higher than previously estimated.

The link between low-dose radiation and cancer has long been controversial. Most studies on the radiation-cancer connection are based on Japanese atomic bomb survivors, many of whom were exposed to far higher levels of radiation than most people receive over their lifetimes – even those who work with ionizing radiation. 

The question is whether that data can be extrapolated to people exposed to much lower levels of radiation, such as nuclear workers, medical professionals, or even patients. To that end, researchers in the International Nuclear Workers Study (INWORKS) have been tracking low-dose radiation exposure and its connection to mortality in nearly 310k people in France, the UK, and the US who worked in the nuclear industry from 1944 to 2016.

INWORKS researchers previously published studies showing low-dose radiation exposure to be carcinogenic, but the new findings in BMJ offer an even stronger link. For the study, researchers tracked radiation exposure based on dosimetry badges worn by the workers and then rates of cancer mortality, and calculated rates of death from solid cancer based on their exposure levels, finding: 

  • Mortality risk was higher for solid cancers, at 52% per 1 Gy of exposure
  • Individuals who received the occupational radiation limit of 20 mSv per year would have a 5.2% increased solid cancer mortality rate over five years
  • There was a linear association between low-dose radiation exposure and cancer mortality, meaning that cancer mortality risk was also found at lower levels of exposure 
  • The dose-response association seen the study was even higher than in studies of atomic bomb survivors (52% vs. 32%)

The Takeaway

Even though the INWORKS study was conducted on nuclear workers rather than medical professionals, the findings could have implications for those who might be exposed to medical radiation, such as interventional radiologists and radiologic technologists. The study will undoubtedly be examined by radiation protection organizations and government regulators; the question is whether it leads to any changes in rules on occupational radiation exposure.

How Vendors Sell AI

Posted on by Brian Casey

Better patient care is the main selling point used by AI vendors when marketing neuroimaging algorithms, followed closely by time savings. Farther down the list of benefits are lower costs and increased revenue for providers. 

So says a new analysis in JACR that takes a close look at how FDA-cleared neuroimaging AI algorithms are marketed by vendors. It also includes several warning signs for both AI developers and clinicians.

AI is the most exciting technology to arrive in healthcare in decades, but questions percolate on whether AI developers are overhyping the technology. In the new analysis, researchers focused on marketing claims made for 59 AI neuroimaging algorithms cleared by the FDA from 2008 to 2022. Researchers analyzed FDA summaries and vendor websites, finding:

  • For 69% of algorithms, vendors highlighted an improvement in quality of patient care, while time savings for clinicians were touted for 44%. Only 16% of algorithms were promoted as lowering costs, while just 11% were positioned as increasing revenue
  • 50% of cleared neuroimaging algorithms were related to detection or quantification of stroke; of these, 41% were for intracranial hemorrhage, 31% for stroke brain perfusion, and 24% for detection of large vessel occlusion 
  • 41% of the algorithms were intended for use with non-contrast CT scans, 36% with MRI, 15% with CT perfusion, 14% with CT angiography, and the rest with MR perfusion and PET
  • 90% of the algorithms studied were cleared in the last five years, and 42% since last year

The researchers further noted two caveats in AI marketing: 

  • There is a lack of publicly available data to support vendor claims about the value of their algorithms. Better transparency is needed to create trust and clinician engagement.
  • The single-use-case nature of many AI algorithms raises questions about their economic viability. Many different algorithms would have to be implemented at a facility to ensure “a reasonable breadth of triage” for critical findings, and the financial burden of such integration is unclear.

The Takeaway

The new study offers intriguing insights into how AI algorithms are marketed by vendors, and how these efforts could be perceived by clinicians. The researchers note that financial pressure on AI developers may cause them to make “unintentional exaggerated claims” to recoup the cost of development; it is incumbent upon vendors to scrutinize their marketing activities to avoid overhyping AI technology.

Grading AI Report Quality

Posted on by Brian Casey

One of the most exciting new use cases for medical AI is in generating radiology reports. But how can you tell whether the quality of a report generated by an AI algorithm is comparable to that of a radiologist?

In a new study in Patterns, researchers propose a technical framework for automatically grading the output of AI-generated radiology reports, with the ultimate goal of producing AI-generated reports that are indistinguishable from those of radiologists. 

Most radiology AI applications so far have focused on developing algorithms to identify individual pathologies on imaging exams. 

  • While this is useful, helping radiologists streamline the production of their main output – the radiology report – could have a far greater impact on their productivity and efficiency. 

But existing tools for measuring the quality of AI-generated narrative reports are limited and don’t match up well with radiologists’ evaluations. 

  • To improve that situation, the researchers applied several existing automated metrics for analyzing report quality and compared them to the scores of radiologists, seeking to better understand AI’s weaknesses. 

Not surprisingly, the automated metrics fell short in several ways, including false prediction of findings, omitting findings, and incorrectly locating and predicting the severity of findings. 

  • These shortcomings point out the need for better scoring systems for gauging AI performance. 

The researchers therefore proposed a new metric for grading AI-generated report quality, called RadGraph F1, and a new methodology, RadCliQ, to predict how well an AI report would measure up to radiologist scrutiny. 

  • RadGraph F1 and RadCliQ could be used in future research on AI-generated radiology reports, and to that end the researchers have made the code for both metrics available as open source.

Ultimately, the researchers see the construction of generalist medical AI models that could perform multiple complex tasks, such as conversing with radiologists and physicians about medical images. 

  • Another use case could be applications that are able to explain imaging findings to patients in everyday language. 

The Takeaway

It’s a complex and detailed paper, but the new study is important because it outlines the metrics that can be used to teach machines how to generate better radiology reports. Given the imperative to improve radiologist productivity in the face of rising imaging volume and workforce shortages, this could be one more step on the quest for the Holy Grail of AI in radiology.

Mammography AI’s Leap Forward

Posted on by Brian Casey

A new study out of Sweden offers a resounding vote of confidence in the use of AI for analyzing screening mammograms. Published in The Lancet Oncology, researchers found that AI cut radiologist workload almost by half without affecting cancer detection or recall rates.

AI has been promoted as the technology that could save radiology from rising imaging volumes, growing burnout, and pressure to perform at a higher level with fewer resources. But many radiology professionals remember similar promises made in the 1990s around computer-aided detection (CAD), which failed to live up to the hype.

Breast screening presents a particular challenge in Europe, where clinical guidelines call for all screening exams to be double-read by two radiologists – leading to better sensitivity but also imposing a higher workload. AI could help by working as a triage tool, enabling radiologists to only double-read those cases most likely to have cancer.

In the MASAI study, researchers are assessing AI for breast screening in 100k women in a population-based screening program in Sweden, with mammograms being analyzed by ScreenPoint’s Transpara version 1.7.0 software. In an in-progress analysis, researchers looked at results for 80k mammography-eligible women ages 40-80. 

The Transpara software applies a 10-point score to mammograms; in MASAI those scored 1-9 are read by a single radiologist, while those scored 10 are read by two breast radiologists. This technique was compared to double-reading, finding that:

  • AI reduced the mammography reading workload by almost 37k screening mammograms, or 44%
  • AI had a higher cancer detection rate per 1k screened participants (6.1 vs. 5.1) although the difference was not statistically significant (P=0.052)
  • Recall rates were comparable (2.2% vs. 2.0%)

The results demonstrate the safety of using AI as a triage tool, and the MASAI researchers plan to continue the study until it reaches 100k participants so they can measure the impact of AI on detection of interval cancers – cancers that appear between screening rounds.

The Takeaway

It’s hard to overestimate the MASAI study’s significance. The findings strongly support what AI proponents have been saying all along – that AI can save radiologists time while maintaining diagnostic performance. The question is the extent to which the MASAI results will apply outside of the double-reading environment, or to other clinical use cases.

Does ‘Automation Neglect’ Limit AI’s Impact?

Posted on by Brian Casey

Radiologists ignored AI suggestions in a new study because of “automation neglect,” a phenomenon in which humans are less likely to trust algorithmic recommendations. The findings raise questions about whether AI really should be used as a collaborative tool by radiologists. 

How radiologists use AI predictions has become a growing area of research as AI moves into the clinical realm. Most use cases see radiologists employing AI in a collaborative role as a decision-making aid when reviewing cases. 

But is that really the best way to use AI? In a paper published by the National Bureau of Economic Research, researchers from Harvard Medical School and MIT explored the effectiveness of radiologist performance when assisted by AI, in particular its impact on diagnostic quality.

They ran an experiment in which they manipulated radiologist access to predictions from the CheXpert AI algorithm for 324 chest X-ray cases, and then analyzed the results. They also assessed radiologist performance with and without clinical context. The 180 radiologists participating in the study were recruited from US teleradiology firms, as well as from a health network in Vietnam. 

It was expected that AI would boost radiologist performance, but instead accuracy remained unchanged:

  • AI predictions were more accurate than two-thirds of the radiologists
  • Yet, AI assistance failed to improve the radiologists’ diagnostic accuracy, as readers underweighted AI findings by 30% compared to their own assessments
  • Radiologists took 4% longer to interpret cases when either AI or clinical context were added
  • Adding clinical context to cases had a bigger impact on radiologist performance than adding AI interpretations

The findings show automation neglect can be a “major barrier” to human-AI collaboration. Interestingly, the new article seems to run counter to a previous study finding that radiologists who received incorrect AI results were more likely to follow the algorithm’s suggestions – against their own judgment. 

The Takeaway

The authors themselves admit the new findings are “puzzling,” but they do have intriguing ramifications. In particular, the researchers suggest that there may be limitations to the collaborative model in which humans and AI work together to analyze cases. Instead, it may be more effective to assign AI exclusively to certain studies, while radiologists work without AI assistance on other cases.

Can You Believe the AI Hype?

Posted on by Brian Casey

Can you believe the hype when it comes to marketing claims made for AI software? Not always. A new review in JAMA Network Open suggests that marketing materials for one-fifth of FDA-cleared AI applications don’t agree with the language in their regulatory submissions. 

Interest in AI for healthcare has exploded, creating regulatory challenges for the FDA due to the technology’s novelty. This has left many AI developers guessing how they should comply with FDA rules, both before and after products get regulatory clearance.

This creates the possibility for discrepancies between products the FDA has cleared and how AI firms promote them. To investigate further, researchers from NYU Langone Health analyzed content from 510(k) clearance summaries and accompanying marketing materials for 119 AI- and machine learning (ML)-enabled devices cleared from November 2021 to March 2022. Their findings included:

  • Overall, AI/ML marketing language was consistent with 510(k) summaries for 80.67% of devices
  • Language was considered “discrepant” for 12.61% and “contentious” for 6.72% 
  • Most of the AI/ML devices surveyed (63.03%) were developed for radiology use; these had a slightly higher rate of consistency (82.67%) than the entire study sample

The authors provided several examples illustrating when AI/ML firms went astray. In one case labeled as “discrepant,” a developer touted the “cutting-edge AI and advanced robotics” in its software for measuring and displaying cerebral blood flow with ultrasound. But the product’s 510(k) summary never discussed AI capabilities, and the algorithm isn’t included on the FDA’s list of AI/ML-enabled devices.

In another case labeled as “contentious,” marketing materials for an ECG mapping software application mention that it includes computation modeling and is a smart device, but require users to request a pamphlet from the developer for more information.

The Takeaway 

So, can you believe the AI hype? This study shows that most of the time you can, with a consistency rate of 80.67% – not bad for a field as new as AI (a fact acknowledged in an invited commentary on the paper). But the study’s authors suggest that “any level of discrepancy is important to note for consumer safety.” And for a technology that already has trust issues, it’s probably best that developers not push the envelope when it comes to marketing.

H1 Radiology Recap

Posted on by Brian Casey

That’s a wrap for the first half of 2023. Below are the top stories in radiology for the past 6 months, as well as some tips on what to look for in the second half of the year.

  • Radiology Bounces Back – After several crushing years in the wake of the COVID-19 pandemic, the first half brought welcome news to radiology on several fronts. The 2023 Match wrapped up with diagnostic radiology on top as the most popular medical specialty for medical students over the past 3 years. Radiology was one of the highest-compensated specialties in surveys from Medscape and Doximity, and even vendors got into the act, reporting higher revenue and earnings as supply chain delays cleared up. Will the momentum continue in the second half? 
  • Burnout Looms Large – Even as salaries grow, healthcare is grappling with increased physician burnout. Realization is growing that burnout is a systemic problem – tied to rising healthcare volumes – that defies self-care solutions. Congressional legislation would boost residency slots 5% a year for 7 years, but is even this enough? Alternatively, could IT tools like AI help offload medicine’s more mundane tasks and alleviate workloads? Both questions will be debated in the back half of 2023. 
  • In-Person Shows Are Back – The pandemic took a wrecking ball to the trade show calendar, but things began to return to normal in the first half of 2023. Both ECR and HIMSS held meetings that saw respectable attendance, following up on a successful RSNA 2022. By the time SIIM 2023 rolled around in early June, the pandemic was a distant memory as radiology focused on the value of being together. 

The Takeaway

As the second half of 2023 begins, all eyes will be on ChatGPT and whether a technology that’s mostly a curious novelty now can evolve into a useful clinical tool in the future. 

Better Together at SIIM

Posted on by Brian Casey

Humans have a deep-seated need for interpersonal contact, and understanding that need should guide not only how we structure our work relationships in the post-COVID era, but also our development and deployment of new technologies like AI in radiology. 

That’s according to James Whitfill, MD, who gave Thursday’s opening address at SIIM 2023. Whitfill’s talk – which was followed by a raucous audience participation exercise – was a ringing demonstration that in-person meetings like SIIM still have relevance despite the proliferation of Zoom calls and remote work. 

Whitfill, chief transformation officer at HonorHealth in Arizona and an internist at the University of Arizona, was chair of the SIIM board in 2020 when the society made the difficult decision to move its annual meeting to be fully online during the pandemic.

The experience led Whitfill to ponder whether technology designed to help us work and collaborate virtually was an adequate substitute for in-person interaction. Unfortunately, the research suggests otherwise: 

  • Numerous studies have demonstrated the negative effect that the isolation of the COVID pandemic has had on adolescent mental health and academic performance 
  • Loneliness can also have a negative effect on physical well-being, with a recent U.S. Surgeon General’s report finding that prolonged isolation is the health equivalent of smoking 15 cigarettes a day
  • Peer-reviewed studies have shown that people working in in-person collaborative environments are about 10% more productive and creative than those working virtually. 

Whitfill’s talk was especially on-point given recent research indicating that workers across different industries who used AI were more lonely than those who didn’t, a phenomenon that shouldn’t be ignored by those planning radiology’s AI-based future. 

That said, virtual technologies can still play a role in making access to information more equitable. Whitfill noted that some 160 people were following the SIIM proceedings entirely online, and they otherwise would not have been able to benefit from the meeting’s content.

To drive the point home, Whitfill then had audience members participate in a team-based Rochambeau competition that sent peals of laughter ringing through Austin Convention Center.  

The Takeaway
Whitfill’s point was underscored repeatedly by SIIM 2023 attendees, who reiterated the value of interpersonal connections and networking at the conference. It’s ironic that a meeting devoted at least in part to intelligence that’s artificial has made us better appreciate relationships that are real.

Mayo’s AI Model

Posted on by Brian Casey

SAN DIEGO – What’s behind the slow clinical adoption of artificial intelligence? That question permeated the discussion at this week’s AIMed Global Summit, an up-and-coming conference dedicated to AI in healthcare.

Running June 4-7, this week’s meeting saw hundreds of healthcare professionals gather in San Diego. Radiology figured prominently as the medical specialty with a lion’s share of the over 500 FDA-cleared AI algorithms available for clinical use.

But being available for use and actually being used are two different things. A common refrain at AIMed 2023 was slow clinical uptake of AI, a problem widely attributed to difficulties in deploying and implementing the technology. One speaker noted that less than 5% of practices are using AI today.

One way to spur AI adoption is the platform approach, in which AI apps are vetted by a single entity for inclusion in a marketplace from which clinicians can pick and choose what they want. 

The platform approach is gaining steam in radiology, but Mayo Clinic is rolling the platform concept out across its entire healthcare enterprise. First launched in 2019, Mayo Clinic Platform aims to help clinicians enjoy the benefits of AI without the implementation headache, according to Halim Abbas, senior director of AI at Mayo, who discussed Mayo’s progress on the platform at AIMed. 

The Mayo Clinic Platform has several main features:

  • Each medical specialty maintains its own internal AI R&D team with access to its own AI applications 
  • At the same time, Mayo operates a centralized AI operation that provides tools and services accessible across departments, such as data de-identification and harmonization, augmented data curation, and validation benchmarks
  • Clinical data is made available outside the -ologies, but the data is anonymized and secured, an approach Mayo calls “data behind glass”

Mayo Clinic Platform gives different -ologies some ownership of AI, but centralizes key functions and services to improve AI efficiency and smooth implementation. 

The Takeaway 

Mayo Clinic Platform offers an intriguing model for AI deployment. By removing AI’s implementation pain points, Mayo hopes to ramp up clinical utilization, and Mayo has the organizational heft and technical expertise to make it work (see below for news on Mayo’s new generative AI deal with Google Cloud). 

But can Mayo’s AI model be duplicated at smaller health systems and community providers that don’t have its IT resources? Maybe we’ll find out at AIMed 2024.

When AI Goes Wrong

Posted on by Brian Casey

What impact do incorrect AI results have on radiologist performance? That question was the focus of a new study in European Radiology in which radiologists who received incorrect AI results were more likely to make wrong decisions on patient follow-up – even though they would have been correct without AI’s help.

The accuracy of AI has become a major concern as deep learning models like ChatGPT become more powerful and come closer to routine use. There’s even a term – the “hallucination effect” – for when AI models veer off script to produce text that sounds plausible but in fact is incorrect.

While AI hallucinations may not be an issue in healthcare – yet – there is still concern about the impact that AI algorithms are having on clinicians, both in terms of diagnostic performance and workflow. 

To see what happens when AI goes wrong, researchers from Brown University sent 90 chest radiographs with “sham” AI results to six radiologists, with 50% of the studies positive for lung cancer. They employed different strategies for AI use, ranging from keeping the AI recommendations in the patient’s record to deleting them after the interpretation was made. Findings included:

  • When AI falsely called a true-pathology case “normal,” radiologists’ false-negative rates rose compared to when they didn’t use AI (20.7-33.0% depending on AI use strategy vs. 2.7%)
  • AI calling a negative case “abnormal” boosted radiologists’ false-positive rates compared to without AI (80.5-86.0% vs. 51.4%)
  • Not surprisingly, when AI calls were correct, radiologists were more accurate with AI than without, with increases in both true-positive rates (94.7-97.8% vs. 88.3%) and true-negative rates (89.7-90.7% vs. 77.3%)

Fortunately, the researchers offered suggestions on how to mitigate the impact of incorrect AI. Radiologists had fewer false negatives when AI provided a box around the region of suspicion, a phenomenon the researchers said could be related to AI helping radiologists focus. 

Also, radiologists’ false positives were higher when AI results were retained in the patient record versus when they were deleted. Researchers said this was evidence that radiologists were less likely to disagree with AI if there was a record of the disagreement occurring. 

The Takeaway 
As AI becomes more widespread clinically, studies like this will become increasingly important in shaping how the technology is used in the real world, and add to previous research on AI’s impact. Awareness that AI is imperfect – and strategies that take that awareness into account – will become key to any AI implementation.

Get every issue of The Imaging Wire, delivered right to your inbox.

You might also like..

Select All

You're signed up!

It's great to have you as a reader. Check your inbox for a welcome email.

-- The Imaging Wire team

You're all set!