When AI Suggests Eating Rat-Nibbled Cheese (Really)

Three guardrails to implement AI chatbots that won't embarrass you in front of your students

(Warning - cheese puns ahead)

‍

Last year, New York City rolled out an official business chatbot designed to help small business owners navigate the city’s maze of health codes, labor laws, and licensing requirements.

The idea was to give quick, trustworthy answers (anyone who’s had to get speedy, trustworthy answers from their local government KNOWS the challenges there). It certainly was speedy, but at the expense of, uh, accuracy:

• Telling restaurants it’s okay to take a cut of employee tips (It isn’t).
• Claiming it’s legal to fire an employee who refuses to cut their dreadlocks (It’s not).
• Advising that it’s fine to serve cheese nibbled by rodents — as long as customers are informed. (Yes, really).

It’s an absurd image: nobody is lining up to eat rat-nibbled cheese, no matter how you slice it. But the humor masks a sharper problem: this wasn’t some random internet chatbot. It was an official city tool, giving official-sounding answers that were flat-out wrong. Even in a lower-stakes context, a bad AI answer can erode trust, create liability, and put people at real risk. It’s enough for anyone to say, that’s not gouda. (That’s the last cheese pun, we promise)

And weirdly, this is just the latest in a string of AI confabulations about cheese.

‍

When AI Misfires in Healthcare

When AI gets it wrong in health care, the consequences are far more serious. Consider the National Eating Disorder Association’s chatbot, “Tessa.”

In an effort to modernize their helpline, NEDA replaced human counselors with an AI bot meant to offer support to patients dealing with related health concerns. Almost immediately, users reported that Tessa was recommending exactly the kinds of harmful behaviors those users were trying to recover from: calorie restriction, body measurements, weigh-ins — the whole spectrum of disordered eating behaviors.

The fallout was immediate. The bot was suspended, the headlines weren’t kind, and advocates asked a very important question: How could something like this slip through the cracks?

‍

The Problem? Generative AI is a Black Box.

Generative AI systems, especially large language models, process data in ways that are often unclear, even to their creators. We can see what we put in and what comes out, but the internal decision-making — the exact path the AI takes to arrive at its answer — is hidden inside a complex web of billions of parameters. Researchers are working on “explainable AI” to shed light on these inner workings, but even the most recent reasoning models of AI seem unable to explain their own reasoning consistently.

This lack of transparency makes it hard to fully trust the output, especially when dealing with sensitive topics like health, safety, or ethics. Without clear visibility, errors can slip by unnoticed until they cause harm.

‍

Three Guardrails for Using AI Safely

So how do you get around this AI-trust problem?  A few guidelines and guardrails:

‍

1. RAG Architecture

One of the most effective ways to reduce hallucinations and boost AI reliability is to ground answers in verified, relevant information. That’s exactly what Retrieval-Augmented Generation (RAG) does.

With RAG, the AI doesn’t rely on vague “memories” from training data. Instead, it first retrieves documents from a trusted database, then uses those sources to generate its answer. This keeps responses fact-based and easy to update as new information becomes available.

Let’s say you want to build an interactive patient case, powered by AI.  But you want the outputs to be trustworthy.  How do you do that?

At ReelDx, the RAG approach means:

• The AI pulls only from a curated list of possible diagnoses
• The AI is told to ignore any conditions not in the database
• The AI is instructed to link every output to a specific case file or reference source

The result: higher accuracy and transparency. It’s not perfect — AI isn’t capable of such a thing right now — but it’s a heck of a lot more accurate than out-of-the-box Large Language Models.

Want to see RAG in action?

We’d be happy to show you what we built and how we built it. Book a meeting with us to see our AI in action and get a free trial for you and your students.

‍

2. Disclaimers and Training

I tried the MyCity Chatbot Beta for myself to see whether it still spat out wrong information. I discovered a smart addition from the New York City government: a friendly disclaimer.

This is a great example of how you might use AI tools in your own classroom. It highlights both the strengths and limitations of the technology, and it helps learners see that AI is not perfect, while also teaching them how to recognize those imperfections before you ask them to rely on it in their own work.

‍

3. Create a testing and oversight process

You can’t just “set it and forget it” with AI.

Even the most advanced AI tools will make mistakes. The key is to make sure that there’s a process of expert review both at the start of using a new tool and periodic testing throughout to catch mistakes and tweak the use.  Aim for continuous improvement in any tool that you choose.

‍

One Last Morsel

Hallucinations seem baked into this technology. That’s why students should practice catching them in the classroom. Better they get cultured in a safe environment than find out about AI’s flaws while melting under pressure in the field. (/end cheese puns)

Folks, we're living in a time where robots are recommending rodent-seasoned appetizers. If that doesn't convince you to teach critical thinking in the classroom, I don't know what will.

—Rob

‍

‍