Ternary Bonsai 27B benchmarked and tested vs Qwen 27B - 16GB Local LLM setup
Credibility score: 51/100 — Mixed Credibility. Several questionable claims detected. Watch with healthy skepticism.
BSmeter analyzed "Ternary Bonsai 27B benchmarked and tested vs Qwen 27B - 16GB Local LLM setup" and rated it 51/100 for credibility (a BS score of 49/100 — mixed credibility), on 2026-07-20. Its weakest claim — "Blames model when harness fails — false dilemma" — scored 20/100 and was flagged as false dilemma. 52 claims were checked against the video transcript. Scores are produced by BSmeter's AI analysis of the transcript, not independent human verification.
Of 52 claims analyzed: 1 scored under 40, 40 between 40 and 69, and 11 at 70 or above.
Claims analyzed
Ternary weights (-1/0/+1) make inference much faster than floats — No Frame (75/100)
At 0:36
Straight technical description of the actual mechanism — no trick.
Why this score: Speaker accurately states what ternary weights are and the performance benefit without exaggeration or missing context.
Original quote: “The Ternary part is essentially how the weights are represented. So, you can see here they mention each weight takes a value of either -1, 0, or +1. So, this is different to the very large floating point numbers that we usually using in model weights. And this allows the system to calculate through…”
Drafters work like MTP but sit outside the model — Missing Context (45/100)
At 1:28
Compares drafters to MTP then immediately notes the key difference — the comparison is loose.
Why this score: Speaker says 'kind of like MTP' then explains why it isn't the same. The qualifier prevents a clean false equivalence, but the initial analogy still risks confusing viewers who know speculative decoding.
Original quote: “basically, it's kind of like MTP in a way. So, what these do, they're smaller models that assist the base model in predicting the next tokens.”
Lists exact model sizes like they're settled facts. — No Frame (75/100)
At 2:26
Straight numbers pulled from files. No framing tricks here.
Why this score: He's citing concrete file sizes for the base and drafter variants. Plain data drop with nothing hidden or spun.
Original quote: “we saw over in the files, this base is just 7.17 GB. And then we add on our little 4B model. So, if we go for a Q4 version of our drafter, that's only 2 gig. Or we can go with a 16-bit BF16, 7.29 GB.”
Exact model sizes presented as fact — no frame — No Frame (75/100)
At 2:26
Just reads the file sizes straight — no trick, no hype.
Why this score: Speaker quotes concrete numbers from the files without exaggeration or loaded language. Straight reporting of measured values.
Original quote: “we saw over in the files, this base is just 7.17 GB. And then we add on our little 4B model. So, if we go for a Q4 version of our drafter, that's only 2 gig. Or we can go with a 16-bit BF16, 7.29 GB.”
Calls Q4KM 'standard' precision for their model — Missing Context (45/100)
At 2:55
Labels their choice 'standard' — but Q4KM is one of several common options, not the default.
Why this score: By calling Q4KM the 'standard,' the speaker implies it's the neutral baseline. Viewers aren't told other quant levels (Q5, Q6, Q8) are equally common depending on VRAM and speed goals.
Original quote: “So, it's the same base model, 27B, but obviously this is using our standard floating-point precision for our quantization. So, we'll be using the Q4KM.”
Equates MTP and drafter as 'kind of similar' — Confidence Mismatch (45/100)
At 3:03
Calls two different techniques 'kind of similar' with zero technical comparison shown.
Why this score: Speaker equates MTP speculative decoding with the Bonsai drafter without evidence or definitions. The 'kind of' softener hides the fact that no actual mechanism match has been demonstrated.
Original quote: “And then we'll be using the MTP version here, too, because again, this is kind of a similar concept to the drafter principle in the bonsai model.”
Lists benchmarks without explaining what 'various depths' means — Missing Context (45/100)
At 3:22
'Various depths' sounds rigorous — but no definition or methodology given yet.
Why this score: The memory benchmark is presented as thorough, yet the transcript never specifies how depth is measured, what retrieval task is used, or how success is scored. Listeners are left to assume the test is meaningful.
Original quote: “We've got our performance benchmark where we'll test the prefill and the decode speed of the model. We've got our memory benchmark where we'll be filling the context up and see how it does at finding data at various depths.”
Calls agency benchmark 'a good test' without evidence — Confidence Mismatch (45/100)
At 3:34
Labels it 'a good test' — confidence with zero validation shown.
Why this score: Speaker asserts the benchmark's quality before any results. No prior validation, community acceptance, or comparison to established agent benchmarks is provided.
Original quote: “We've got our agency benchmark, which will be a good test of tool calling and reasoning. We've then got our OpenAI human eval, which is 164 Python challenges.”
Calls dungeon crawler 'really good' for algorithm testing — Confidence Mismatch (45/100)
At 3:37
'Really good evaluation' — asserted, never justified.
Why this score: Speaker claims the test measures pure algorithmic reasoning, but offers no proof it prevents library use or that success correlates with real coding ability.
Original quote: “And then onto some coding. We're going to be doing our dungeon crawler, which is a really good evaluation of the model's ability to use algorithms without reaching for frameworks or libraries.”
Introduces new Blender test via MCP — no prior validation — Missing Context (45/100)
At 3:51
New test announced as if it's ready — but it's literally the first run.
Why this score: Speaker presents the Blender/MCP setup as a finished benchmark while admitting in the same breath it's brand new and untested. Viewers aren't told how success or failure will be measured.
Original quote: “And then two new tests that I've been working on here. So, we've got Blender, where we're going to connect the model to Blender, the 3D modeling software program via MCP, and we're going to get it to create some simple shapes in there.”
Chains Godot success to Blender success — conditional test — Missing Context (45/100)
At 4:04
Test only happens if Blender works — so failure is already baked in.
Why this score: The Godot test is presented as part of the plan, yet its execution depends entirely on Blender succeeding. This makes the overall evaluation structure fragile and the 'we shall see' framing misleading.
Original quote: “And then Godot, which is a game engine. Again, we're going to have the model connect to that through MCP, and we're going to get it to create some simple games. And if it was actually successful in the Blender test, we're going to get it to use some assets that it made in the game in Godot.”
Ends with 16 GB VRAM system spec — straightforward — No Frame (75/100)
At 4:19
Just states the hardware — no exaggeration.
Why this score: Speaker gives the exact VRAM amount without claiming performance or making it sound exceptional. Clean fact.
Original quote: “So, these new tests go well. We shall see. And then the target system here, I am running on a system with 16 GB of VRAM on my GPU.”
Claims results generalize to 'your' hardware — zero shared data — Missing Context (45/100)
At 4:33
Promises performance on 'your' rig while admitting zero shared benchmarks — classic extrapolation without evidence.
Why this score: Speaker sets up the entire video as relevant to viewers' machines yet only tested on one specific 16 GB VRAM + 32 GB DDR4 setup. No data from other configs is shown, so the 'obviously' claim rests on nothing.
Original quote: “So, we're going to be targeting systems around this spec. So, obviously, if you've got more VRAM, you're going to see better performance than I will have in this video. Likewise, if you've got less, less of a performance.”
Q4 drafter 22 t/s vs BF16 drafter 1 t/s — drafter shouldn't change output — Missing Context (45/100)
At 4:52
Calls the 1 t/s BF16 result 'interesting' while the drafter comment shows why it shouldn't happen at all.
Why this score: Drafter models are only supposed to affect speed, not final output quality. The massive drop from 22 to 1 token per second suggests either a bug or offloading the BF16 model to system RAM, yet the speaker frames it as a simple preference instead of addressing the real cause.
Original quote: “the first one we're going to look at is the Q4 drafter. So, this one is giving us a 46 on the prefill. ... we're getting 22 tokens a second here. And if I just bring up the BF16 drafter, so we can see if we compare these two, the prefill is 63, coming down to 56 here, whereas 44 down to 34. ...…”
BF16 'one token a second' vs Q4 '22' — no mention of why the gap exists — Missing Context (45/100)
At 5:17
Presents 22× speed difference as model choice when memory offloading is the real culprit.
Why this score: BF16 drafter likely doesn't fit in 16 GB VRAM, forcing slow system RAM usage. The transcript never says this, so viewers may blame the quantization instead of the hardware constraint. Commenters already flagged exactly this issue.
Original quote: “if we compare these two, the prefill is 63, coming down to 56 here, whereas 44 down to 34... token generation, so we're getting one token a second on our BF16 drafter model, whereas we're getting 22 tokens a second on our Q4 drafter.”
Calls 22 t/s 'quite significant' while skipping that base model is offloaded — Missing Context (45/100)
At 5:57
Hails the drafter's 22 t/s as a big win — quietly compares it to a base model that's already crippled by RAM swapping.
Why this score: The 8.5 t/s base number is artificially low because the full model doesn't fit in VRAM. The drafter wins only because it's small enough to stay on GPU. Without that context the 'significant' improvement looks magical instead of mechanical.
Original quote: “our pre-fill speeds are quite a bit more on the base model here. We're getting over 100 tokens a second for the pre-fill versus again around 40 on the bonsai with the Q4 drafter... token generation speed though... eight and a half tokens a second on the base model and 22 on the bonsai version.”
22 t/s is double my normal 11 t/s on 27B — Missing Context (45/100)
At 6:21
Calls it double the speed — never says what model or setup gave the 11 t/s baseline.
Why this score: Without knowing if the 11 t/s came from the same Qwen 27B or a different model entirely, the 'double' claim has no anchor. Commenters already flag that base models without enough VRAM drop to slow RAM speeds, which could explain the gap.
Original quote: “model with the Q4 drafter. This is quite significant. Like to get 22 tokens a second on a 27B dense model on my hardware at least is really good. Like I think the best I've seen is 11 tokens a second normally. So, this is like double the performance I would normally get for speed.”
22 t/s on 27B is 'quite significant' — confidence with no baseline given — Confidence Mismatch (45/100)
At 6:27
Calls 22 t/s 'quite significant' without saying what normal 27B speeds look like on the same rig.
Why this score: The speaker jumps straight to 'significant' while the only comparison offered is his own prior 11 t/s result. Without external benchmarks or other 27B numbers, the label rests on personal surprise rather than data.
Original quote: “this is quite significant. Like to get 22 tokens a second on a 27B dense model on my hardware at least is really good.”
Drafter "helped" at 256K — admits results mixed, no conclusion — Confidence Mismatch (45/100)
At 8:26
Calls it "definitely helped" then immediately says "mixed" and "not sure what conclusion." Classic pivot without the receipts.
Why this score: Speaker swings from confident causal language to total uncertainty in the same breath. The "definitely" has no backing data shown, only three test runs and one changed variable.
Original quote: “And it also got lost at the same depth because 25% at 256K is the same as the 50% depth with 128K. So, oddly, it just really gets lost at this depth only, but then it was fine after that. And then, if we flip over to the BF16 drafter, now the story changes. So, we're still on Bonzai, but we're now…”
BF16 drafter suddenly makes Bonsai find everything across 256K — Confidence Mismatch (45/100)
At 8:37
Calls it 'the story changes' like the drafter fixed the depth problem — zero data on why it worked or if it will again.
Why this score: He's attributing success to the BF16 drafter without showing any control or ablation. The jump from 'lost at depth' to 'found everything' is presented as causal when it's just one run with no explanation of the mechanism.
Original quote: “And then, if we flip over to the BF16 drafter, now the story changes. So, we're still on Bonzai, but we're now using the BF16 drafter, and now it's found the data all the way across, but obviously, our test time is now 20 hours.”
Base model 'almost perfect' except on #13 where 'most models fail' — Missing Context (45/100)
At 10:42
Calls base 'almost perfect' while admitting it failed the same question as everyone else — the bar is doing all the work.
Why this score: The 'almost perfect' framing hides that the model still failed a known hard problem. Without showing how other models actually scored on the full set, the praise rests on moving the goalposts rather than raw performance.
Original quote: “base one took 15 minutes and was almost perfect, but it did fall down on number 13, which is where most models do fall down”
Bonsai beats base on both speed and score — No Frame (75/100)
At 10:55
Straight comparison with numbers attached. No hidden framing here.
Why this score: The speaker gives clear times and states Bonsai scored higher. The claim is testable against the data shown.
Original quote: “15 minutes for the base model and then just under 7 minutes for bonsai with a better score”
Claims Bonsai faster and better score than base model — Missing Context (45/100)
At 10:58
Calls it 'better score' after saying base got almost perfect and only failed on #13 — the exact same spot most models fail.
Why this score: Speaker presents Bonsai's win as decisive while admitting the base model was nearly perfect and tripped on the same question everyone trips on. The 'better' label hides that the difference is tiny and both hit the same wall.
Original quote: “just under 7 minutes for bonsai with a better score. So, that's a good result there for bonsai.”
77% pass rate, 90% on answered questions, 85% answer rate — No Frame (75/100)
At 11:23
Breaks out the three different metrics clearly. Numbers, not narrative.
Why this score: Speaker explicitly separates overall pass rate, conditional pass rate, and answer rate. The distinctions are shown on screen and stated plainly.
Original quote: “this is ternary bonsai and this is our Q4 drafter. So, as we can see, it's got 77% pass rate and if we evaluate it based on the questions that it answered, it got 90% and that's because it did not answer 24 questions”
Presents 90% on answered questions as the real number while burying the 24 skips — Volume Game (45/100)
At 11:25
Loudly drops the 90% figure, then quietly adds it skipped 24 questions. The 77% pass rate gets mentioned once and never revisited.
Why this score: The speaker leads with the inflated 90% stat that excludes skipped questions, then tucks the actual 77% result and the 15% skip rate into the same breath. Classic volume game — the impressive number gets the emphasis.
Original quote: “it got 77% pass rate and if we evaluate it based on the questions that it answered, it got 90% and that's because it did not answer 24 questions. So, that gives it an answer rate of 85%.”
Bonsai edges out base on answered-pass rate and answer rate — No Frame (75/100)
At 12:08
Direct side-by-side without inflating the margin. Clean.
Why this score: The claim stays within what the numbers actually show — 'slightly higher' instead of dramatic language.
Original quote: “the pass rate for answered questions is just slightly higher and the answer rate is also higher”
Bonsai answers twice as many questions as base Qwen — No Frame (75/100)
At 12:15
Clean head-to-head on refusal count. No spin added.
Why this score: The raw refusal numbers are given and compared directly. Viewers can judge the practical difference themselves.
Original quote: “base Qwen failed to answer 48 questions, whereas Bonsai only failed to answer 24”
Blames context burning for skips, implying Bonsai is smarter for answering more — Missing Context (45/100)
At 12:15
Attributes Bonsai's fewer skips to better thinking instead of the drafter setup that might be forcing shorter responses.
Why this score: Speaker frames the halved skip count as evidence Bonsai thinks more efficiently, but never mentions that the drafter system itself could be cutting off long reasoning chains that would otherwise burn context.
Original quote: “base Qwen failed to answer 48 questions, whereas Bonsai only failed to answer 24. So, the failure to answer questions usually comes down to the model burning all of its context on thinking.”
Claims Bonsai 2h faster + smarter than base on same test — Missing Context (45/100)
At 12:42
Speed win is real — but the 'better intelligence' part is just declared, not measured.
Why this score: Time difference comes from the clock. Intelligence claim has zero scoring, benchmark, or comparison shown to back it up.
Original quote: “So, our Bonsai took 4 hours and 22 versus the base model 6 hours. So, that's a 2-hour difference in speed on the Bonsai model and better intelligence on this test.”
Blames model for file write failure without testing harness first — Missing Context (45/100)
At 14:51
Assumes model fault before checking if the harness itself blocks writes — classic premature diagnosis.
Why this score: Speaker rules out his own setup instantly and pins it on the model. The transcript shows no harness test was run yet, so the 'model problem' conclusion jumps the gun.
Original quote: “Now, I don't know what that's about. I've done nothing. This is the same stuff I always use in here. And the model is just having problems. It's trying to use a shell command to write to file, which it shouldn't be doing. So, I'm wondering is this a problem with the model unable to use this harness…”
Blames model when harness fails — false dilemma — False Dilemma (20/100)
At 15:00
Presents only two options — harness or model — while ignoring the third: the setup itself.
Why this score: The speaker narrows it to 'model can't use harness' vs 'model broken' and skips checking whether the MCP configuration or Klein client is actually passing the tool schema correctly. Classic two-option trap that hides the real culprit.
Original quote: “It's trying to use a shell command to write to file, which it shouldn't be doing. So, I'm wondering is this a problem with the model unable to use this harness or is this a problem with the model itself?”
Claims prior Klein tests prove harness works, so model must be at fault — Missing Context (45/100)
At 15:24
Cites 'it worked with other models' without showing whether those tests used the exact same write-to-file task.
Why this score: The speaker treats successful past runs as proof the harness is fine for this specific operation. Different prompts and different tool calls mean the comparison isn't apples-to-apples.
Original quote: “But this will be a problem if we're going to do our blender and our Godot tests because my MCP is configured for those within Klein and I have tested it with other models in Klein and it did work.”
Declares model can't do tool calling after one failed test — Confidence Mismatch (45/100)
At 16:05
One failed HTML write becomes 'the model can't do tool calling' — that's a hell of a leap from a single data point.
Why this score: Speaker moves from 'this prompt didn't work' to a blanket verdict on the model's entire tool-use capability. No retry, no prompt tweak, no harness debug shown.
Original quote: “So it looks like the problem was the model's ability to understand the tool calling within the client harness, unfortunately.”
Model made UI but no working game — downplaying failure — Missing Context (45/100)
At 16:44
Calls it 'structure' like that's progress — zero mention of the actual broken state.
Why this score: The model produced dead UI with zero interactivity. Framing it as 'structure' softens the total failure and hides how far off the output actually is.
Original quote: “it's sort of created the structure, but there's not really a game yet”
Model loops 'wait' and 'I just realized' endlessly — calling it overthinking — Confidence Mismatch (45/100)
At 17:11
Labels the endless self-correction as 'overthinking' — it's actually just failing to solve the problem.
Why this score: The transcript shows the model repeatedly hitting dead ends and restarting its reasoning with 'wait' and 'I just realized.' Calling this 'overthinking' implies it's smart but verbose; it's actually stuck and unproductive.
Original quote: “I think it's overthinking because if we look at the thinking... there should be a new benchmark called the wait benchmark”
Bonsai stuck in endless checking loop, never writes code — Confidence Mismatch (45/100)
At 18:52
Calls it 'never writes any code' after one re-prompt attempt — that's confidence with barely any data behind it.
Why this score: The speaker watched one failed run, then declared the model's fundamental behavior as 'never writes code.' One observation turned into a blanket rule. No multiple trials, no different prompts, just one messy session turned into permanent diagnosis.
Original quote: “Okay, just checking in with you again on Bonsai and it looks like after re-prompting it to try and get it to get out of its thinking loop, be a bit more concise, and figure out the issues, the thinking patterns are just the same, basically. It wants to check everything a million times. Uh, it…”
One-shot success declared 'clear win' after single demo — Confidence Mismatch (45/100)
At 21:01
One working run equals 'clear win' — zero retries shown, zero comparison data.
Why this score: Speaker moves from 'it worked this time' to 'base model wins' with nothing establishing consistency or failure rate. One successful attempt doesn't prove superiority over the other model when the test is cherry-picked to the single case that succeeded.
Original quote: “Done. Perfect. So, there's no issues. So, I can't really see how the model is if it has to go back and fix things, but as we can see, it's a one shot and it took just over 10 minutes to do it. So, yeah, it's a clear win for the base model here.”
Model will succeed this time — hope dressed as test — Confidence Mismatch (45/100)
At 22:32
Keeps restarting like success is coming — zero evidence it will.
Why this score: Speaker frames repeated failures as temporary glitches rather than capability limits, building false expectation that the next attempt will magically work.
Original quote: “So, let's reopen blender and we'll let it try again at modifying blender. So, it's going for an MCP call again now.”
Shows UI text too small to read, asks viewer to trust it's working — Missing Context (45/100)
At 22:45
Apologizes the text is too small right as he claims it's running — the only evidence we need is hidden.
Why this score: He points at the screen, admits we can't read it, then tells us what it says. The verification is entirely verbal with no visible proof offered.
Original quote: “So, if we go to edit preferences, I apologize this text is probably too small for you to see, but I'm just going to confirm the MCP server here and it is running. So, and it does say at the bottom in pi MCP one out of two, the other one being Godot, which is not running obviously.”
Declares Ternary Bonsai completely failed after one short attempt — Confidence Mismatch (45/100)
At 23:13
Calls the whole test a failure after a few error messages and moves on — zero retries shown.
Why this score: The model hit MCP errors and he immediately concludes it 'isn't successfully able to interact at all.' No extended testing or troubleshooting is attempted before switching models.
Original quote: “Okay. So, checking in again, and it doesn't seem like it's successfully able to interact with Blender at all. So, I'm going to stop this, and we're going to jump over to the base queen again, just to see if this model is capable of doing it or not.”
Ternary model completely failed Blender test — No Frame (75/100)
At 23:13
Straight report of what happened. No spin.
Why this score: Speaker simply states the observable result without exaggeration or softening — clean delivery of the actual outcome.
Original quote: “Okay. So, checking in again, and it doesn't seem like it's successfully able to interact with Blender at all.”
Base Qwen will definitely connect and work — Confidence Mismatch (45/100)
At 23:28
Says 'we should see' like it's guaranteed — it's a hope, not a fact.
Why this score: Uses predictive certainty ('we should see') before running the test, turning an experiment into an implied foregone conclusion.
Original quote: “Okay. So, we're in Pi with the base model. We've got the extension running. Let's pass the prompt over, and then we should see down here MCP becomes one out of two, which will indicate it's connected.”
Base Qwen deletes the cube so it's 'definitely able to interact' — Confidence Mismatch (45/100)
At 23:50
One successful cube deletion and he declares full interactivity proven — the bar is suddenly very low.
Why this score: After the first model struggled, one visible change (deleted default cube) is treated as conclusive proof the base model works. The earlier standard of 'successfully able to interact at all' appears much stricter here.
Original quote: “So, yep, we can see at the bottom now MCP is connected. Says it's going to start by clearing default scene. So, let's see if anything changes in our workspace. So, it's explored the tools. There's 26 of them. So, here we go. It's going to do an MCP call now. So, we'll see if this first MCP call…”
Base model succeeded despite error — victory claimed — Missing Context (45/100)
At 24:04
Calls it success while admitting an error happened — glosses over the actual result quality.
Why this score: Speaker downplays the error and focuses only on the visible change, omitting whether the deletion was the intended or correct action.
Original quote: “So, we'll see if this first MCP call does affect anything in the program. And there it is. So, there was some error message, but we can see... the scene is now has been changed. So, it it deleted the default cube.”
Calls Ternary clearly ahead after one Blender task — Confidence Mismatch (45/100)
At 24:36
One partial task, then declares the winner. Confidence way ahead of the data shown.
Why this score: Speaker jumps from 'looks not too bad' on one incomplete marble gate to 'we already know which model has done better.' No second task, no scoring, just a verdict. Classic overclaim from limited evidence.
Original quote: “it's already doing better than bonsai on this task”
Declares winner before finishing test — confidence mismatch — Confidence Mismatch (45/100)
At 24:50
Calls it decided mid-run while Bonsai still hasn't finished — the test isn't over yet.
Why this score: Speaker interrupts the slower model and announces the result before the second model completes the task. The framing treats an incomplete comparison as settled fact.
Original quote: “So, I think for the purposes of this test, we already know which model has done better.”
Cancels Godot test, calls it impossible for Bonsai — Missing Context (45/100)
At 25:21
Skips the harder test without trying — then frames the sand sim as the real comparison.
Why this score: Speaker asserts Bonsai can't handle the game engine, then pivots to the easier browser test. Viewers never see Bonsai actually fail the skipped task; the comparison quietly narrows to something the speaker already expects Bonsai to attempt.
Original quote: “there's no way Bonsai is going to do the game engine. So, we're going to go with a nice familiar test here”
Sand sim UI broken, yet 'surprisingly no errors' — Confidence Mismatch (45/100)
At 26:14
Features don't work, console clean — calls it surprising instead of admitting the model failed to wire the controls.
Why this score: The demo has non-functional buttons and sliders. Speaker treats the lack of console errors as unexpected rather than evidence the generated code simply didn't implement the requested interactions. The 'surprise' framing downplays the actual failure.
Original quote: “That's not working. I can't change to a different material. And if I adjust the brush size, the counter doesn't seem to change here. So, if we go into our console, surprisingly, there are no errors to be”
Bonsai keeps failing the same fix — looping on the issue — No Frame (75/100)
At 26:49
Straight reporting: model repeatedly fails to fix the exact bugs he identified.
Why this score: He describes the concrete failure mode (UI ready but materials and brush size broken) and shows the model repeating the same useless loop. No exaggeration, no hidden framing — just the outcome.
Original quote: “All right, we're checking in now on Bonsai and after asking it to do some fixes, I simply told it, where are we? So, let me find my prompt that I gave it. So, the UI is all in place, but I cannot place any material and I can't change material either. So, very simple, I've identified two issues that…”
Calls Bonsai's repeated failures a total fail — No Frame (75/100)
At 27:45
Straight observation — the model really is looping the same mistake.
Why this score: Speaker simply reports what the output is doing without exaggeration or loaded framing.
Original quote: “On and on and on and on and let's go all the way down to where we are now and as you can see, it's just doing the exact same thing. So, no, that's a fail again on this one.”
"Performance definitely better" + "aced it" — sweeping win declared from mixed results — Confidence Mismatch (45/100)
At 31:21
Calls the whole thing "definitely better" after admitting memory was mixed — confidence jumped the evidence.
Why this score: Speaker says performance is "definitely better" right after conceding mixed memory results. The single strong agency test is used to stamp an overall victory label that the mixed data doesn't actually support.
Original quote: “So, the performance is definitely better. Memory was just a bit of a mixed bag. Then on our agency test, it aced it, which is quite rare to see.”
Claims 2x+ speed advantage over base model — Confidence Mismatch (45/100)
At 33:24
Says 'double or more' with zero numbers shown — just vibes on the multiplier.
Why this score: Speaker gives a qualitative speed claim without benchmarks, token counts, or hardware context. The 'or more' leaves the upper bound completely open while sounding decisive. Missing any actual measurement makes the stated advantage impossible to verify from what's said.
Original quote: “it is like double or more than double the speed of the base 27B if you're at Q4”
See the full analysis with timestamps →