Silence as a Feature: Why Cinnamon’s Healthcare AI Doesn't Answer Every Question

Depending on whom you ask, AI is either transforming healthcare or dangerously overhyped. Strangely, it might just be both. Today, a growing number of people credit popular large language models (LLMs) for helping them solve health problems that medical professionals couldn’t. Others point out that AI systems still give out questionable medical advice, even in relatively straightforward situations.

Why does a system capable of solving complex medical problems struggle to answer basic questions? LLMs often excel at semantic matching and synthesizing information, but can make simple reasoning errors that humans rarely make. To use LLMs well, especially in high-stakes environments like healthcare, organizations have to understand where LLMs perform reliably, where they don’t, and how to design processes that capitalize on their strengths while protecting against their weaknesses.

Where AI excels, and where it doesn’t

One known LLM limitation is clinical reasoning under uncertainty. A recent study found that LLMs struggle with differential diagnosis, which requires working through a range of possible diagnoses and ordering tests to rule conditions out. Most of the models tested failed to identify conditions that could be ruled in or out at rates exceeding 80%. Physicians were more willing to admit uncertainty, seek additional information, and reason through multiple possibilities, while LLMs sometimes jumped to conclusions prematurely.

However, the same study found that once all the relevant information was made available, LLMs performed well on final diagnoses. The finding suggests that while humans are more adept at navigating ambiguity and deciding what questions to ask next, LLMs can be highly effective when given a complete set of facts and a well-defined problem to solve.

A second study noted a similar finding. Leading LLMs correctly identified relevant medical conditions in nearly 95% of standardized test scenarios when given complete case information. Yet when members of the public recruited online interacted with those same models to answer the same questions, performance plummeted. People often didn’t share the right information with the model, didn't interpret responses correctly, and didn't know when to trust the model’s outputs. As a result, they identified the correct conditions in fewer than 35% of cases.

Taken together, these studies show that LLM performance is heavily dependent on how information is gathered and framed, and how human-AI workflows are designed. Those factors can determine whether AI becomes a valuable clinical tool or a source of costly errors.

Cinnamon's four design principles

At Cinnamon, we are deliberate about using AI only where it can add meaningful value without creating frustrating or unreliable experiences. To that end, we follow four key design principles:

1. Use AI only where it offers a clear advantage

Not all tasks require AI, and many can be accomplished faster, more reliably, and at lower cost through traditional automation and deterministic workflows. As a result, Cinnamon is selective about where and how we use AI. At this time, we deliberately limit its use to a single workflow: answering clinical questions during prior authorizations.

To craft these answers, an LLM within our AI system reviews and analyzes real-time patient data. In practice, this often involves synthesizing information from patient records and drug labeling that a clinician may not have immediately top of mind while completing a prior authorization. For example, the system may identify FDA-labeled contraindications, relevant drug interactions, and other safety considerations while distinguishing them from related concepts such as limitations of use. It then presents a drafted answer along with the supporting evidence to the prescribing team for review. This shifts clinicians away from time-consuming record review and toward the higher-value task of evaluating evidence and making decisions.

This is also exactly the type of workflow LLMs are known to perform well. By the time a prior authorization is submitted, clinicians have already performed the tricky clinical reasoning steps and selected a treatment. At that point, our AI model steps in to draft answers to well-defined, fact-based questions with clear answers based on the patient’s medical history and drug labeling.

2. Keep clinical judgement at the center

At Cinnamon, we believe decisions that affect patient health and safety should remain under human control. Our AI model is built to support clinicians by surfacing relevant information: Whenever applicable, AI-generated answers are accompanied by relevant verbatim excerpts from the patient record. Users are trained to verify the AI's reasoning against those excerpts and can review the underlying documentation directly rather than relying on the model's interpretation alone. Only a human can approve and submit the final response.

The system is also designed to work only within its limits. Today, it drafts answers for about 85% of prior authorization questions. When a question requires unknown information, relates to future events, concerns provider certifications, or otherwise falls outside the system's scope, the model employs categorical refusals: Rather than generating a speculative response, it simply provides no answer and surfaces available, relevant documentation for the prescribing team to review.

3. Minimize human-AI communication failures

Since LLM performance depends on how people interact with it, Cinnamon pulls relevant data directly from payer and clinical systems and presents it to the AI in a structured format rather than requiring users to determine what information to provide or how to phrase a question. This helps ensure the model has access to the information it needs while reducing the “game of telephone” errors that can occur when information is manually relayed between multiple systems and users. Research has shown these human-AI communication errors can reduce model performance far more than the capabilities of the model itself.

This approach also improves security and reliability. By controlling how information enters and moves through the workflow, Cinnamon significantly reduces opportunities for prompt injection, data exfiltration, and other forms of misuse.

4. Design for and measure real-world adoption

LLMs are probabilistic systems, meaning they will always have some level of error. Cinnamon is designed with that reality in mind. By building the workflow around a human in the loop, we add a crucial layer of redundancy to safeguard against AI errors while creating a positive feedback loop to continuously improve outcomes.

At Cinnamon, we measure performance in real-world workflows and continuously study how the system performs in practice. Before launching with a new partner, we evaluate local data formats and identify non-standard EHR implementations so that information is interpreted correctly and results are reliable. Today, users accept nine out of ten answers the system provides without making any edits. Those reviews provide human-in-the-loop verification so we can continue to monitor and improve performance over time.

What responsible healthcare AI looks like

Already, the conversation around AI in healthcare is shifting away from building ever more powerful models. Increasingly, experts argue that successful healthcare AI implementations depend not just on model performance but on specific, high-impact use cases, interoperable ecosystems, and transparent workflows with outputs that can be clearly verified and audited.

That’s the philosophy behind Cinnamon’s approach. We focus our use of AI on the exact workflows in which it can excel, integrating the technology directly into provider workflows and making sure every response is supported by evidence.

See for yourself how Cinnamon supports prescriber teams while keeping clinical judgement at the center of healthcare. Request a demo today.

Photo by National Cancer Institute on Unsplash