One number, no logs: five days on the AMD hackathon leaderboard

The AMD Developer Hackathon's Track 1 judge takes a Docker image, runs 19 hidden tasks against it, and gives you back one number. Accuracy. That's the whole interface. No logs, no stderr, no exit code, no per-task breakdown. You submit, you wait somewhere between ten minutes and an hour and a half, and a percentage shows up on a leaderboard. Then you try to figure out what it means.

I spent five days reading that one number. This is what it cost me, and the two or three things it taught me that I actually want to keep.

Day one: a perfect score that meant nothing

We submitted the first version on July 8. Our own evaluation said 24 out of 24. A clean sweep. I remember being a little smug about it.

The judge said 52.6 percent.

That gap is the entire story in miniature, and I didn't understand it yet. Our local test set was something we wrote, so of course we passed it. The judge's set was hidden, and it was measuring something our test never touched. The only measurement that mattered was the one I couldn't watch being taken. I would relearn that sentence about four more times over the next week, each time wearing a different costume.

The probing era

So we did the only thing you can do against a box that won't talk to you. Change one thing, submit, read the delta, keep it or revert it. Slow science.

The second version scored worse, 47.4. Turned out the code I'd added to stop the model from leaking its reasoning was also eating the answer sometimes. Fixing that got a couple of tasks back. Aligning the prompts with what each task actually wanted got one more. I tried making the math solver show its work, and it cost a task, so I reverted it. Giving the code-generation answers a cleaner structure got one back. Fixing how we handled mixed-sentiment reviews, the ones that say something nice and something mean in the same paragraph, jumped us two tasks at once, up to 15 of 19.

Each of those probes cost a submission, and early on a submission could take an hour to come back. You learn patience or you learn nothing.

Then I found the thing that broke my confidence. The same image, byte for byte, scored 15 one day and 14 the next. The score wobbles. Somewhere around two tasks of variance, just from noise. Which meant every neat conclusion I'd written down from a single submission was suspect. A week of careful bookkeeping, and the ground under it moved by two tasks depending on the weather.

The rules changed under us

On July 10 the organizers flipped the whole competition. No more pass or fail. Now it was a live leaderboard ranked by fewest paid API tokens, with an accuracy gate sitting around 80 percent. Qualify above the gate, then win by spending less.

Everything I'd built to chase accuracy was now working against me. The generous token limits, the answer-and-then-explain contracts, all of it was spending tokens I no longer wanted to spend. And most of the bill wasn't even the answers. It was hidden chain-of-thought from the reasoning models, hundreds of tokens per task that the judge never reads and never scores. Pure overhead.

Two levers came out of this. First, setting reasoning effort to none cut the token bill by more than half, because it killed that invisible thinking step. Second, and this is the one that shaped the rest of the week: the guide said local compute was free. Anything my container could answer on its own CPU cost zero tokens and still counted for accuracy.

Zero tokens. That number will do things to your judgment.

The wrong conclusion I lived inside for days

We started moving tasks off the paid models and onto small models bundled inside the image. The first one worked beautifully. A little BERT tagger for named-entity tasks, the "pull the company and person names out of this text" category. It scored 28 of 30 on our oracle and it ran on the judge without a complaint. Zero tokens, real accuracy. Encouraging.

So I got greedy and tried moving more categories local. Code generation. Summaries. Math and logic. Sentiment. The board punished almost every one. Math and logic local came back 10 of 19. An all-local experiment, everything bundled, scored 3 of 19 despite a 96 percent oracle. I wrote memory files with titles like "math/logic-local dead" and "local oracle is blind too," shrugged, and concluded the small models just weren't good enough. Back to the paid models for anything hard.

Hold onto that conclusion. It was wrong, and the specific way it was wrong is the part of this I keep thinking about.

On July 11 the organizers rotated the hidden task set and re-enforced the gate. Our best image scored 78.9 percent. That's 15 of 19. One task short of qualifying. One. I went at the three categories that had no public examples, since those were the only places a hidden defect could hide, and found two real ones: a logic task that wanted a short justification and was failing because we shipped a bare answer, and a local coder that produced code that compiled and was wrong. Fixed both. The next version scored 17 of 19 at about 5,650 tokens and finally ranked. Somewhere around 36th out of 69 qualified teams. Mid-table. Safe. Solid.

I did not want mid-table. There were teams sitting at zero tokens.

The bet, and the night

Before the real gamble there was a smaller one, and I lost it. I'd convinced myself that an older, cheaper version had genuinely ranked and that its later terrible re-grade was a leaderboard caching bug. The evidence actually pointed at a window where the gate was briefly off. I bet on my version anyway. It came back 52.6 percent and didn't qualify, exactly like the boring explanation predicted. I put the 17 of 19 image back and got my rank returned. Lesson, again: test the thing, don't trust the story you like.

While I watched the board, I had an agent chasing the zero-token dream in parallel. Bundle one real model, answer everything on it, pay nothing. It picked a 4B Qwen build. The oracle numbers were gorgeous, 104 of 109 across seven categories. The CI run answered all 19 smoke tasks correctly, inside the judge's exact time and length limits. It went in around 7:20 in the evening.

The judge said PULL_ERROR. I checked every layer of the image, confirmed it was publicly pullable, reminted the tag, resubmitted.

This time it scored. 15.8 percent. Three of nineteen.

Here is the thing about three of nineteen. Our BERT answers exactly three tasks. A 4B model that actually ran would not contribute a flat zero across the other sixteen. It might do badly. It would not do nothing, twice, across two different model sizes. That's not a quality problem. That's a fingerprint. Everything downstream of BERT was never executing.

The reason was a safety check I'd written weeks earlier and forgotten about. In local.py:

def _cpu_supports_wheel():
    try:
        with open("/proc/cpuinfo") as f:
            return "avx2" in f.read()
    except OSError:
        return True

The prebuilt llama.cpp wheel is compiled with AVX2. On a CPU that doesn't have AVX2, llama.cpp doesn't raise a clean error. It runs an illegal instruction and the whole process dies, exit code 0xc000001d, no Python traceback, nothing. A dead container scores zero, so this guard refused to load the model and let the pipeline fall back to junk answers instead. It was doing exactly what I built it to do.

The judge's CPU has AVX but not AVX2. Probably a VM with a masked CPU model. So on every single submission, for days, that guard tripped silently, the model never loaded, and I read the resulting scores as proof that local models were bad. My "local is dead" memory file was an autopsy of a patient who was never in the room. The models I'd written off had never produced a single token on that machine.

Six hours

I gave myself a hard rule before this part, because it was past midnight and I was going to be making calls while tired: get to zero tokens, or fall back to the 17 of 19 image at the last safe moment. One or the other. Nothing clever in between.

The first fix attempt built a portable wheel with AVX2 turned off and a flag to bypass the guard. I reviewed it and found the bug in a couple of minutes: the bypass line had landed in the wrong function, one below the check it was supposed to override, so the loader still consulted the old unbypassed version. 3 AM energy. That image was going to score three or four no matter what, and it did, 21.1 percent. I moved the bypass to the right place and wrote unit tests to pin the behavior down, because CI physically cannot catch this class of bug. Every GitHub runner has AVX2. The defect only exists on a CPU we didn't own.

Next version went in at 00:52. INFRA_ERROR, twelve minutes later. A faster failure is still information. I resubmitted it as a flakiness check and dug into the wheel while it ran. The cause was one line in ggml's CMake file:

option(GGML_BMI2 "ggml: enable BMI2" ${INS_ENB})

When you build non-native, that default flips on for every instruction extension. Our "portable" wheel had AVX2, FMA, and F16C disabled, but BMI2 was still baked in. A CPU old enough to lack AVX2 also lacks BMI2, so the first pdep instruction killed the container. The guard I had just correctly bypassed had been protecting me from this exact crash the whole time.

BMI2 off, next version in at 01:50. It ran. It survived. It scored 10.5 percent. Two of nineteen. The container lived and the model still produced nothing useful.

At that point I stopped submitting blind and built the thing I should have built on day one. A CI job that exports the image's filesystem and runs its llama stack under qemu, emulating an old CPU that matches the judge's profile: AVX yes, AVX2 no, BMI2 no.

docker export $(docker create $IMAGE) | tar -x -C rootfs
cp /usr/bin/qemu-x86_64-static rootfs/
chroot rootfs /qemu-x86_64-static -cpu SandyBridge \
  /usr/local/bin/python3 /smoke.py

The verdict came back clean. The wheel loaded in ten seconds and generated the string OK, correctly, under the judge's exact instruction limits. Which meant the instructions were fine. Which left only the ugly answer: the judge's box is too slow or too memory-starved to push even a 2B model through sixteen tasks inside the time budget. I halved the context window and the output caps and sent it one more time at 03:01. It scored 21.1 percent at 04:15.

That was the whole answer. Not a bug, not a flag. The bet itself was wrong. The five teams sitting at zero tokens almost certainly ran stacks of tiny specialist models, one BERT-sized model per category, not one general model doing everything. The proof was in my own scores all week. My BERT ran on that judge every single night without a complaint. The winning architecture was sitting in my own row, three tasks wide, and I'd spent my whole runway trying to make one big model work instead of seven small ones.

The fallback

At 04:30, on the rule I'd set for myself before the night started, I queued the 17 of 19 image. A browser extension I'd wired up walked the submission wizard, clicked update, and returned to the team page. The board was slow and loud by then, other teams flashing TIMEOUT and PULL_ERROR everywhere as everyone scrambled before the deadline. The re-check was still pending when I finally stopped.

Not the ending I chased all night. But a real one, picked on purpose, with the reasons written down.

What I keep

Three things.

The score was the only debugger I ever had, and it was barely enough, and the reason it was enough at all is that different failures produced different numbers. "Model never ran" scored 3 because BERT still answered its three tasks. "Model ran badly" would have scored something else. If every fallback layer in my stack had failed to the same number, I'd have had no debugger at all. So if I ever build for a black-box judge again, I'm designing those fingerprints in on purpose from the first commit, not discovering by luck that I happened to have one.

Wrong conclusions calcify fast. "Local models are bad" sat in my notes for days and steered real decisions, and it rested entirely on a model that never executed. Writing something down does not make it true. It just makes it feel true, which is worse.

And the guard gets the last word. That tiny cpuinfo check was written to prevent a crash, and it worked perfectly, so perfectly that it hid the crash it was preventing, and the hiding cost me more than the crash ever would have. If it had just let the container die, I'd have seen a dead container and known within a day. Instead it handed me plausible wrong answers for a week. Sometimes the safety net is the exact thing you most need to watch fail.