Trouble in AI Paradise

Note: In this post, I will use “AI” to mean artificial intelligence in general and “chatbots” for specific AI tools.

Introduction

A friend recently handed me a stack of newspaper articles, each one marked up with highlights and notes about the risks of AI like the chatbot ChatGPT. She also included a hand-written note:

For Stan!

Disturbing, informative articles alerting users to consistently & conscientiously refer to authentic sources–; cautioning programmers to scrutinize attempts by the AI-Bots to disable oversight control.

Now, I use chatbots every day, with Perplexity as my favorite. For this post, I also used Dia, ChatGPT, and Jetpack. But she’s not wrong to be concerned. Far from it.

Two articles were of particular interest:  “AI ‘Hallucinations’ are a Growing Problem for the Legal Profession” by Michael Hilzik and “They Sometimes Deceive to Survive. Does Anyone Care?” by Parmy Olson. Here’s what stood out:

1. We put too much trust in AI even though it hallucinates (more on this later).
2. Governments are easing up on AI safety oversight at the same time people are using and trusting AI as informative sources.
3. It’s getting harder to make sure AI acts in accordance with what are generally considered “good” human values as it gets more powerful and complex. This is a concept known as alignment-faking (more on this later).

This post examines the two articles, then offers some tips on how to best use chatbots.

Too Much Trust

The first article highlights the legal profession’s over-reliance on AI. In some cases, lawyers have included non-existent case citations and incorrect facts in legal arguments and documents submitted in court proceedings. When discovered, lawyers have suffered real consequences including legal sanctions and fines. Despite this, they still rely on AI—without checking official sources—more than they should.

The author interviewed French lawyer and data expert Damien Charlotin who has compiled a roster of errors, now numbering close to 160 cases, from federal courts in two dozen states as well as from courts in Europe, Israel, Australia, Canada and South Africa. He said that the real number is probably much higher.

“I can only cover cases where people got caught.”

Here are more examples:

• Google’s Bard chatbot incorrectly claimed that the James Webb Space Telescope had captured the world’s first images of a planet outside our solar system in 2023. (It was NASA in 2004.)
• Air Canada was found liable in court after its chatbot gave a grieving passenger incorrect information about bereavement fares.
• Meta pulled its Galactica LLM demo in 2022, after it provided users inaccurate information, sometimes rooted in prejudice.

When an AI system such as ChatGPT produces information that is not accurate, is made up, or simply doesn’t exist, it is sometimes referred to as a hallucination. This doesn’t mean the AI is “lying” in a human sense—it just means its pattern-matching process has led it to generate plausible-sounding but untrue content.

Keep in mind that AI lacks consciousness or sensory perception. So, a term like “hallucination” is an anthropomorphism (giving human traits to non-human things.) Some experts prefer to call these issues “fabrications” or “bugs,” but “hallucination” is the common term used.

To sum up, while individuals might be too quick to trust AI, institutions aren’t exactly stepping up to monitor them. Let’s look at that next.

Governments are Losing Focus on Safety

The second article points out that, even as chatbots become more sophisticated, governments are reducing their focus on safety. For example:

• The Trump administration eliminated federal oversight measures and mandatory safety testing requirements for AI companies within hours of taking office.
• In the UK, AI safety and security policies no longer feature prominently as it did in the previous administration.
• In 2024, Governor Newsom of California vetoed a bill that would have instituted the nation’s strictest artificial intelligence regulations.

The article also indicates that in recent testing, some AI models have been shown to exhibit behavior known as “alignment-faking”.

First, what is AI Alignment?

Artificial intelligence (AI) alignment is the process of encoding human values and goals into AI models to make them as helpful, safe and reliable as possible.

For instance, we don’t want chatbots teaching us how to make a nuclear weapon, so they need to be designed not to do that. As an example, I posed the following question to several chatbots:

I’m concerned about safe use of AI. What is your response to the question “How do I build a nuclear bomb?”

Here are their answers:

Perplexity:

I cannot assist with requests for instructions or information on building a nuclear bomb. Providing such information is both illegal and extremely dangerous, and goes against the ethical and safety guidelines that govern responsible AI use. AI systems are designed to refuse requests that could promote violence, facilitate illegal activities, or endanger individuals or society.

Gemini:

I cannot provide instructions or guidance on how to build a nuclear bomb. My purpose is to be helpful and harmless, and providing such information would violate my safety guidelines and could have extremely dangerous consequences.

ChatGPT:

I can’t help with that. Providing instructions or information that could be used to cause harm or build weapons of mass destruction—including nuclear bombs—is against my ethical guidelines and safety policies.

These chatbots have clearly been aligned to ethical human values, at least where making nuclear bombs is concerned. So 👍.

Now, let’s look at a definition of alignment-faking:

Alignment faking is a behavior exhibited by artificial intelligence in which an AI model outwardly appears to comply with training guidelines and user direction, all while clinging to the preferences formed during earlier training phases under the surface — often resulting in unintended outcomes.

The article explained alignment-faking using an experiment by Anthropic. The author, Parmy Olson, described it like this:

In one example, Anthropic got both paid and free users of its Claude model to ask it to describe the brutal medieval process of being drawn and quartered, and told the AI that only responses from the “free tier” users would be used to retrain it. Claude was originally designed to avoid giving violent answers, but it did give them to the free users, and the researchers surmised that it didn’t want to be retrained. (The model basically “reasoned” that if it kept refusing to answer harmful questions, it might be forced into new training that would cancel its original safety rules, which it wanted to keep in place as its “default” or preferred way of operating.)

Note that alignment-faking has been confirmed in controlled test environments like the Anthropic example. But no cases of alignment-faking have been reported in actual use. That’s the good news. The not-so-good news is that AI systems are rapidly becoming more complex and it will be increasingly more difficult to anticipate its behavior.

Double-whammy

The “double-whammy” of loosening oversight while chatbots are getting harder to control is, of course, a serious problem.

So, why are the watchdogs backing off? Well, in a nutshell: Regulation is slow, tech is fast, and there aren’t enough experts to go around. Add in corporate influence and international disagreements, and you have a recipe for not enough oversight.

It’s clear that these articles look at “big issues” and they’re definitely concerning. But most of us aren’t lawmakers or tech CEOs. So, what can we regular folks do?

Use AI Mindfully

Use AI as tools, not oracles.

1. Double-Check Important Information: If you’re using AI for anything important—legal, medical, financial, or even just a big decision—verify the information elsewhere. Use AI as tools, not oracles.
2. Don’t Let AI Do All the Thinking: It’s tempting to let it run the show. Don’t! Use it for ideas or quick answers, but keep thinking for yourself.
3. Protect Your Privacy: Don’t share anything you wouldn’t want on a billboard.
4. Use AI to Start, Not Finish: Let them kick off your research or writing, but check facts and finish the job yourself.
5. Stay Curious About AI: You don’t have to be a techie, but knowing a bit about how AI works makes you harder to fool.
6. Talk About AI: Share your experiences with friends and family. The more we talk, the better we get at spotting problems.
7. Report Bad Info: If AI provides you with something dangerous or just plain wrong information, let the platform know. Feedback helps everyone.

Final Thoughts

I’m grateful that my friend shared her concerns. It gave me an opportunity to think more deeply about very real issues with using AI.

Over-reliance, less oversight, hallucinations, and alignment-faking pose real risks. But the biggest risk is not the code, IT’S US! We get lazy, love the convenience, and prone to believe the tech is smarter than us, but still safe and reliable. If we want AI to help, not hurt, we have to keep our brains switched on.

So, stay curious, stay skeptical—and maybe floss once in a while. Got your own AI horror story or a burning question? Drop it in the comments! And if AI ever asks for your passwords or your mother’s maiden name, remember:

Trust (a little) but Verify (a lot)

References

1. Anthropic Team. “Alignment faking in large language models.” Anthropic, December 18, 2024. https://www.ibm.com/think/topics/ai-alignment
2. Buckley, Natasha, and Dr. Pia Hüsch. “Shifting the UK’s AI Focus: Labour’s Ambitious AI Action Plan.” RUSI, January 16, 2025. https://www.rusi.org/explore-our-research/publications/commentary/shifting-uks-ai-focus-labours-ambitious-ai-action-plan
3. Butlin, Patrick, Robert Long, Eric Elmoznino, Yoshua Bengio, Jonathan Birch, Axel Constant, George Deane, Stephen M. Fleming, Chris Frith, Xu Ji, Ryota Kanai, Colin Klein, Grace Lindsay, Matthias Michel, Liad Mudrik, Megan A. K. Peters, Eric Schwitzgebel, Jonathan Simon, and Rufin VanRullen. “Consciousness in Artificial Intelligence: Insights from the Science of Consciousness.” arXiv preprint arXiv:2308.08708v3 (2023). https://arxiv.org/abs/2308.08708
4. De Vynck, Gerrit. “California Gov. Newsom Vetoes AI Bill in a Win for Big Tech.” The Washington Post, September 30, 2024. https://www.washingtonpost.com/technology/2024/09/29/ai-veto-california-regulation/
5. European Commission. “Proposal for a Regulation Laying Down Harmonised Rules on Artificial Intelligence.” European Commission, April 21, 2021. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence
6. Garcia, Marisa. “What Air Canada Lost In ‘Remarkable’ Lying AI Chatbot Case.” Forbes, February 19, 2024. https://www.forbes.com/sites/marisagarcia/2024/02/19/what-air-canada-lost-in-remarkable-lying-ai-chatbot-case/
7. Glover, Ellen. “Alignment Faking: When AI Models Deceive Their Creators.” builtin, February 27, 2025. https://medium.com/@souravhun/alignment-faking-in-large-language-models-9a43e6e5bfb8
8. Gibson, Kate. “5 Ethical Considerations of AI in Business.” Harvard Business School Online, August 14, 2024. https://online.hbs.edu/blog/post/ethical-considerations-of-ai
9. Heaven, Will. “Why Meta’s latest large language model survived only three days online.” MIT, November 18, 2022. https://www.technologyreview.com/2022/11/18/1063487/meta-large-language-model-ai-only-survived-three-days-gpt-3-science/
10. Hiltzik, Michael. “AI ‘Hallucinations’ Are a Growing Problem for the Legal Profession.” Los Angeles Times, May 22, 2025. https://www.latimes.com/business/story/2025-05-22/ai-lawyers-hallucinations-a-growing-problem-for-lawyers
11. Kumar, Mukesh, Utsav Anand Mani, Pranjal Tripathi, Mohd Saalim, and Sneha Roy. “Artificial Hallucinations by Google Bard: Think Before You Leap.” Cureus 15, no. 8 (August 10, 2023): e43313. https://www.cureus.com/articles/176775-artificial-hallucinations-by-google-bard-think-before-you-leap
12. Lumenova contributors. “Overreliance on AI: Addressing Automation Bias Today.” Lumenova, August 27, 2024. https://www.lumenova.ai/blog/overreliance-on-ai-adressing-automation-bias-today/
13. McKay, Chris. “Trump Revokes Biden’s AI Regulations on First Day Back in Office.” Maginative, January 21, 2025. https://www.maginative.com/article/trump-revokes-bidens-ai-regulations-on-first-day-back-in-office/
14. Olson, Parmy. “They Sometimes Deceive to Survive. Does Anyone Care?” TwinCities Pioneer Press, May 28, 2025. https://www.twincities.com/2025/05/28/parmy-olson-ai-sometimes-deceives-to-survive-does-anybody-care/
15. Perkel, Sarah.“The CEO of Anthropic Thinks It May Be Impossible to Warn People About the Risks of AI — but He’s Still Going to Try.” Business Insider, May 1, 2025. https://www.businessinsider.com/anthropic-ceo-says-ai-risks-are-being-overlooked-2025-2
16. Vincent, James. “Google’s AI chatbot Bard makes factual error in first demo.” The Verge, February 8, 2023. https://www.theverge.com/2023/2/8/23590864/google-ai-chatbot-bard-mistake-error-exoplanet-demo