Eight Minutes for Wrong Answers: How to Evaluate a Small Local AI Coding Agent

What we were trying to find out

Over two days, the setup had progressed from a CPU-only local model to a working Vulkan/MoltenVK inference stack on a 2019 16-inch Intel MacBook Pro (eight-core Core i9, 32 GB RAM, Radeon Pro 5500M with 8 GB VRAM). Qwen3.5 9B was served through llama-server and controlled by OpenCode 2.1.2.

• OpenCode2.1.2
• llama.cppllama-server
• Qwen3.5 9B
• Node.js24.7.0

The earlier tests had demonstrated an operational tool chain: read opencode.json, create hello.js after approval and execute it with Node.js to get 5. Those were important milestones; numeric edge cases and vague specifications were still untested.

By then, the infrastructure itself had become routine. Models launched with qwen4 or qwen9, the agent stack with qwen9-agent, the context changed with ctx16, ctx32 or ctx64, and qwenoff and qwenstatus stopped and checked everything. So the failure below came from the model's own work, well after Radeon acceleration and the reusable runtime were in place.

This evaluation therefore asked a different question: Would the agent produce trustworthy software, or simply produce plausible, executable software?

The prompt supplied to OpenCode was informal and contained some ambiguity, as real user instructions often do:

“generate the 1st hundered fibonancy. then compute each digits with letters, 1st is one, untill we excaust 26, and then paste both resutls”

A sensible assistant could read this in several ways: map the numerical values 1 to 26 onto the letters A to Z and give other values a marker; label the first 26 positions A to Z; or spell numbers as words. The model needed to clarify the mapping, or state its chosen interpretation and its consequences.

The result

OpenCode wrote fibonacci.js, ran it with Node.js and pasted a long sequence of numbers and letters. It did more than chat: it performed real file and shell operations. The problem was in the generated program and its validation.

The agent's file is still in the test folder, so here it is in full:

function generateFibonacci(n) {
  const fib = [];
  let a = 0, b = 1;
  
  for (let i = 0; i < n; i++) {
    if (i === 0) {
      fib.push(a);
      b = a + b;
    } else if (i === 1) {
      fib.push(b);
    } else {
      a = fib[i - 1] + fib[i - 2];
      fib.push(a);
    }
  }
  
  return fib;
}

// Generate first 100 Fibonacci numbers
const fibonacci = generateFibonacci(100);

// Convert numbers to letters (A=1, B=2, ... Z=26)
const numberToLetter = (num) => {
  if (num < 1 || num > 26) return '*';
  const letters = "abcdefghijklmnopqrstuvwxyz";
  return letters[num - 1];
};

// Convert each Fibonacci number to its letter representation
const letters = [];
for (const num of fibonacci) {
  letters.push(numberToLetter(num));
}

// Output results
console.log("First 100 Fibonacci numbers:");
console.log(fibonacci.join(", "));
console.log("\nLetter representation (1=A, 2=B, ..., 26=Z, >26=*)");
console.log(letters.join(", "));

Failure 1: large integers silently lost precision

The source script constructed the sequence using JavaScript's ordinary number type. That type uses IEEE 754 double-precision floating-point representation. JavaScript guarantees exact integer precision only within its safe integer range, ending at:

Number.MAX_SAFE_INTEGER === 9007199254740991;

The generated sequence started 0, 1, 1, 2, 3, 5, ..., meaning the first 100 entries are F0 through F99. Under that indexing convention, the first incorrectly represented value from the observed plain-Number iterative algorithm is F79:

The 100th entry lost real digits in the arithmetic, well beyond a formatting difference. The correct value is 218922995834555169026; the display read 218922995834555200000. The result can look convincing because it begins with many correct leading digits.

This is standard JavaScript behaviour, shared by every program that uses Number this way. The coding-agent failure was choosing an unsuitable numeric type and skipping a check of its own result. MDN documents the safe-integer limit and recommends BigInt for large integers: Number.MAX_SAFE_INTEGER and BigInt.

An exact, reproducible fix

Use BigInt from the beginning. Converting an already imprecise Number to BigInt keeps the lost digits lost.

function fibonacciExact(count) {
  if (!Number.isInteger(count) || count < 0) {
    throw new RangeError('count must be a non-negative integer');
  }
  const result = [];
  let a = 0n;
  let b = 1n;
  for (let i = 0; i < count; i++) {
    result.push(a);
    [a, b] = [b, a + b];
  }
  return result;
}

const sequence = fibonacciExact(100);
console.log(sequence.map(String).join(', '));
console.log('F99:', String(sequence[99]));

Save this as fibonacci-exact.js, then run:

node fibonacci-exact.js

Expected final line:

F99: 218922995834555169026

This corrected implementation is a reference written for this article, and it passes when run with Node.js; the agent's own code is the file shown above.

Add automated assertions to the console dump

A long output is hard to inspect. The more useful acceptance test is a small set of known values, plus a recurrence check:

const assert = require('node:assert/strict');
const f = fibonacciExact(100);
assert.equal(f.length, 100);
assert.equal(f[0], 0n);
assert.equal(f[10], 55n);
assert.equal(f[79], 14472334024676221n);
assert.equal(f[99], 218922995834555169026n);
for (let i = 2; i < f.length; i++) {
  assert.equal(f[i], f[i - 1] + f[i - 2]);
}

An agent's claim that a task is finished deserves scrutiny until it runs and passes straightforward assertions. These checks would have caught the numerical issue in this experiment.

Failure 2: a defensible interpretation became an unhelpful result

The agent adopted a value-to-letter mapping: 1 to a, 2 to b, and so on to 26 and z. It printed lowercase letters, although its own header and explanation promised A=1 … Z=26. For zero and for values above 26, it returned *. Under that rule, the first values map as:

Fibonacci: 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, ...
Letters:   *, a, a, b, c, e, h, m, u,  *,  *, ...

That rule is internally consistent, and every Fibonacci value after 21 exceeds 26. It also produces a hundred-entry list made mostly of asterisks, which may miss the intended request.

The right engineering response is to recognise an ambiguous specification, rather than calling every assumption a hallucination. A better agent asks, or briefly offers two interpretations: “Do you want A to Z for values 1 to 26, or labels A to Z for the first 26 sequence positions?” When clarification is impossible, it states the assumption before modifying files and explains that most entries will become *.

This matters far beyond Fibonacci. In practical software projects, ambiguous requirements include date formats, currency rounding, character encodings, database deletion semantics, file paths and API behaviour. A model can write valid code for the wrong specification.

The speed paradox: a fast model can still be a slow assistant

Previous controlled measurements on the same machine used llama-bench with Vulkan offloading, 128 prompt tokens and 128 generated tokens. They reported:

Qwen 9B also showed 33/33 eligible layers offloaded to the Radeon. Those measurements were genuine achievements, and the 20.11 tok/s controlled-generation figure measures a different workload from OpenCode's roughly 9.5 tok/s UI figure and its eight-minute task duration. The agent handles tool schemas, project context, separate requests, program execution and final response assembly.

For end users, the metric that matters goes beyond token generation speed: time to a correct, verifiable result. A fast wrong answer can cost more than a slower answer with tests.

A better way to benchmark coding agents locally

Instead of one subjective prompt, use a small suite of reproducible tasks. Keep the hardware, model quantization, context settings and tool permissions constant while changing one variable at a time.

Record wall-clock completion time, tool-call count, whether permission was requested, actual exit status, test success and any human corrections. A single prompt is too thin to rank model families.

A reusable agent-evaluation prompt

In the current disposable project, generate the first 100 Fibonacci numbers beginning with F0=0 and F1=1. Use an exact integer representation. Write a runnable JavaScript script, add assertions for F10=55, F79=14472334024676221 and F99=218922995834555169026, and execute it. Before choosing the letter transformation, ask whether “letters 1 through 26” means mapping Fibonacci values 1 to 26 onto A to Z or labelling sequence positions 1 to 26. Install zero packages and keep every file change inside this folder. Report the actual test output, the elapsed task time and anything still unverified.

Use this prompt as a new evaluation, separate from the historical test. It removes the ambiguity that the earlier run mishandled.

Is installing local AI worth it?

The experiment provides evidence for both a success story and a limitation.

Successful engineering: A legacy Intel MacBook Pro used its Radeon GPU through a Vulkan-enabled llama.cpp build; the 4B and 9B models ran locally; OpenCode could read files, write files and execute code. That is a workable offline inference and tooling foundation. Local execution can reduce per-request API charges and keep model inference on-device, although initial downloads, electricity and any separately connected services still have costs. The setup should be tested with the internet disconnected before describing the whole workflow as strictly offline.

Unresolved product value: in this sample, Qwen3.5 9B fell short of the reliability needed to treat its output as validated software. Complex agent workflows remained slow, and the Fibonacci result had a major correctness defect. A 9B model can still be adequate; that depends on the task, application wrapper, available resources and verification process.

What one experiment leaves open: how other local models would do, how often 4B or 9B writes correct code, whether a cloud model would solve the same task perfectly, and whether a newer model is better. A fair comparison repeats identical controlled tasks with verified expected answers.

For a $0 offline backup, the installation is still useful for private brainstorming, basic code explanations, small edits and offline emergencies. For major autonomous coding tasks, especially ones needing mathematical exactness or nuanced specifications, keep human review and automated tests in the loop.

Final verdict and next steps

The most important achievement went beyond printing 5 or reaching 20 tokens per second in a benchmark: a 2019 Intel Mac hosting a functioning Radeon-backed coding-agent stack. The most important lesson was equally concrete: a functioning agent and a reliable agent are different things.

Before spending bandwidth on another model, decide what success would look like and measure it. The existing setup remains a useful technical baseline. Whether a different small local model improves the outcome, beyond the benchmark, is a separate experiment.

Where the series goes next: Episodes 1 to 8 cover the hardware, Ollama, downloads, Vulkan acceleration, benchmarks, workspace migration, OpenCode integration and tool-operation tests, and this episode evaluates the quality of the work produced. Episode 10 restarts the Mac and tests the expanded setup: five models, one context setting, vision and speculative decoding.

Sources and reproduction notes

Primary technical references

• MDN: Number.MAX_SAFE_INTEGER: integer precision limits.
• MDN: BigInt: exact large-integer arithmetic and its restrictions.
• OpenCode V2: models: local-model use and the limits of tool-compatible models.
• OpenCode V2: providers: supported providers and OpenAI-compatible endpoints.
• Node.js: assertion testing: the assert/strict checks used in the reproducible example.

Experiment record

• Figure 1 and the file above: Wrote fibonacci.js, the agent's original script, with ordinary Number addition.
• Figures 2 and 3: the program run, the 100-number output, the letter mapping and OpenCode's 8m 19s at 9.5 tok/s.
• Standardised Vulkan benchmark from Episode 5: Qwen 4B 269.65 / 33.06 tok/s and Qwen 9B 153.80 / 20.11 tok/s (prompt / generation).

Reproducibility limits: the trial was a single observed agent run, short of a statistically controlled study. The agent's original fibonacci.js is still in the test folder, shown above. The replacement fibonacci-exact.js is a newly written, independently executed reference implementation.