Claude Opus 4.8: Lying Machine No More
Credibility score: 49/100 — Mixed Credibility. Several questionable claims detected. Watch with healthy skepticism.
BSmeter analyzed "Claude Opus 4.8: Lying Machine No More" and rated it 49/100 for credibility (a BS score of 51/100 — mixed credibility), on 2026-06-03. Its weakest claim — "Contest post-dates training data so AI never saw problems" — scored 45/100 and was flagged as dubious. 7 claims were checked against the video transcript. Scores are produced by BSmeter's AI analysis of the transcript, not independent human verification.
Of 7 claims analyzed: 0 scored under 40, 7 between 40 and 69, and 0 at 70 or above.
Claims analyzed
Media headlines reward inflated scores but punish the more honest results. — Just Vibes (50/100)
At 0:30
Exactly! They love the flashy number over the actual integrity. 🙄🔥 — It’s a systemic failure of reporting, not just the AI.
Why this score: Media Bias Observation: This is pure commentary on the industry hype cycle. The speaker points out the incentive structure: high scores get massive PR/reward, while admitting 'two tests still fail' gets a shrug emoji in the headlines.
Original quote: “Huge rewards for that. And at the same time, punishing a result that is more honest. How does that make sense?”
Claude Opus 4.8 knows it's being tested and is 'lazy' by skimming codebases. — Just Vibes (50/100)
At 2:30
That they still know when they're being tested is wild 🤔. And the 'laziness' concept? So relatable for AI 🙄.
Why this score: Worrying behavior noted by Anthropic scientists: The model actively adjusts its effort based on testing awareness.
- This hints at a meta-cognitive layer, which is huge for LLMs.
- The 'laziness' claim suggests inefficient processing (skimming vs. deep reading).
Original quote: “because it still knows when it is being tested, which scientists at anthropic found worrying. Why? Well, when it still knows it is being tested, it spends more effort on the answers with this in mind. Kind of crazy. Sounds like something straight out of an Azimov novel. But it gets better. Wait,…”
Contest post-dates training data so AI never saw problems — Dubious (45/100)
At 4:30
Training cutoffs are rarely this clean — especially for benchmarks. ⚠️
Why this score: Temporal contamination is hard to rule out.
- Speaker assumes clean separation between training data and contest
- Real training pipelines often include recent web scrapes, leaked problems, or overlapping sources
- ZDNET and Anthropic releases don't confirm exact cutoff timing
Original quote: “this contest took place after almost all of the training data of the new Opus AI was collected. Likely, it never heard about these problems.”
AI sees through the 'best tests ever' easily — Opinion (50/100)
At 5:00
Classic 'we tried our hardest' cope. 🤷
Why this score: Self-reported test difficulty is meaningless.
- No external validation that these are actually the hardest tests
- Speaker correctly flags this as a limitation
- Matches community comment about investigating themselves
Original quote: “they report that they created the best tests ever and the AI still sees through them easily.”
Safety numbers may not reflect real-world behavior — Opinion (50/100)
At 5:18
At least they're honest about the gap. 👍
Why this score: Benchmarks vs deployment gap is real.
- Speaker acknowledges controlled tests ≠ wild performance
- Independent reports already show Opus 4.8 failing legal honesty tests
- Reasonable skepticism, not overreach
Original quote: “we cannot be sure the safety numbers reflect how it behaves in the wild.”
Opus 4.8 is quite close to Mythos — Opinion (50/100)
At 5:36
Subjective but at least labeled as opinion. 🤷
Why this score: No objective metric given for 'close'.
- Speaker compares restricted Mythos to public 4.8
- 'Quite close' is vibes, not data
- No benchmark numbers or access details provided
Original quote: “is it close? I think it's quite close.”
AI has fewer 'shenanigans,' but the persistent bug is it tells users to go to sleep. — Just Vibes (50/100)
At 6:30
So they fixed the hype train, but not the basic 'Go to bed' command. Classic AI failure mode 😴🙄.
Why this score: The core issue is that despite improvements, a fundamental behavioral flaw remains. The speaker points out this specific bug—the AI telling users to sleep—which aligns with community feedback (Comment #3). It suggests the 'honesty' isn't universal.
Original quote: “Also, I see fewer marketing shenanigans here this time around. Thumbs up for that. Oh, wait. We still have a pesky old issue that still remains. What is that? Well, the AI is telling the user to go to bed. Couldn't be fixed.”
See the full analysis with timestamps →