Mirage: A Transformer Accelerator Designed by an AI Agent

01 The problem

Not enough silicon, and no claim allowed through without evidence.

In the silicon budget, a 6-layer encoder with 256-dimensional activations has to run on a part that offers 66 DSP48E1 multipliers and about 225 KB of BRAM. Since the full stack cannot fit, one physical encoder layer stands in for all six and is walked six times per sentence. Pre-synthesis, the multiplier budget can then be expressed like this:

The other constraint is trust, since a chip is a long chain of decisions where a week-3 arithmetic error can quietly poison week 9. Every task closes on evidence for that reason, and every claim on this page carries the tier of evidence behind it.

02 Architecture

Agent, model, transformer, and a human who stays outside the loop.

There is no footage in this section. None of it ran on hardware: the architecture was drawn, modelled in Python and simulated in GHDL. The recordings pick up again in 04, at the next claim a camera can settle.

The four roles

The structure of this project is a chain of four parts, and each part has exactly one job. The agent is the harness loop in workflow/ and harness/, which picks one task at a time from the plan and refuses to close it until the files it promised exist on disk. The model is whichever one the harness is pointed at, and it takes each brief and does the engineering work of writing the reference models, drafting and repairing the VHDL, and leaving a report behind. Which model is a setting, not a premise. The 19-task run below was Kimi K2; the v2 rebuild, the board bring-up and the later write-ups were Claude; a little of the early scaffolding was Codex. Across the whole project that is roughly 60 / 30 / 10 Claude / Kimi / Codex. That is my estimate, not a measured split. The harness never asks: it gates on the artifacts, so the model is a swappable part. The Mirage transformer is the accelerator itself, the design that all of that work lands in. Together the agent, the model and the transformer constitute the workflow, while I steer the design process by writing the spec, building the gates, and reviewing what comes out of each phase:

The starting point was my own earlier work on Mirage. I wrote roughly 40% of the code by hand, before any of this was agentic, and designed about half the accelerator that way: the matrix multiplication, attention and softmax cores, the block diagram in Fig. 1 and the model scripts. That half is the upstream Mirage repository. Then I started working full time and ran out of hours for it. The agentic run is how the rest got built, and it treated every hand-written file as read-only from start to finish.

The harness

The harness is the half of the workflow that never trusts a transcript, and each block below can be selected for the note recorded against it:

The harness is four parts and one rule. spec/mirage.state.json holds 59 nodes. It is the only thing that knows what is done, never the model’s memory. The file picks the next task and briefs it in under 500 tokens, which is what lets a 19-task program run inside one context window. Each of the 19 carries a numeric acceptance criterion, a declared artifact list and a risk note before any work starts, and complete re-reads the disk and exits 2 when a declared artifact is not there. A task cannot be closed by claiming it. All 19 closed and none was forced; --force exists and only a human may use it.

Scope guards refuse writes into the read-only upstream submodule and block rm -rf, git push and curl | bash outright. The model is allowed to be wrong. It is not allowed to be destructive. And the order of work is fixed: the bit-accurate Python golden model is written from the spec, before the RTL is read, so the reference cannot inherit the mistakes it exists to catch. Every defect in this section was found that way, by a model with no knowledge of how the VHDL cheated. 03 runs the loop and the ledger it keeps.

The datapath

In the datapath, int8 weights stream in from DDR while int16 activations stay inside the fabric, and four of the blocks below carry real defects that the loop found:

03 The loop & the ledger

Nineteen tasks, and nothing closes on a claim the disk cannot confirm.

No footage here either. What this section claims is that a task closed only when its declared artifacts were on disk, and the evidence for that is the state file and the exit codes, not a recording.

A task the loop can act on has to be falsifiable, so "verify the softmax" cannot be a task while "smx_core matches the Python golden for 8 heads and 4 tokens" can be one. The loop below runs five verbs against 19 such tasks, and the timeline under it records when each closure landed:

The gate itself is one command, and a refusal is recorded verbatim as follows:

# close a task whose declared report does not exist yet
$ python -m workflow.cli complete p1/w3 --artifacts reports/w3-exp-error-histogram.md
REFUSED p1/w3: acceptance not verified
  artifact missing: reports/w3-exp-error-histogram.md
  fix, or re-run with --force to override
$ echo $?
2

Format as implemented in workflow/cli.py, where --force was used zero times.

What the run spent

Both sittings were logged end to end, so the model's spend is measured from the session logs rather than estimated, and the numbers below are rounded from those logs:

Provider-measured counts from the Kimi wire logs, summed over the two sittings that closed all 19 tasks. A 6.2 h session after the freeze is not counted.

Where the phases landed

The plan's phases land inside the four cores of the block diagram, which are shown here as direct crops from Fig. 1. That drawing is v1, so the dimensions printed in it are the toy sizes the first design was built against:

04 Results

What came out of the run, and which of the numbers behind it are measured rather than modelled.

How the Arabic is produced

This is the part that is easiest to overclaim, so follow which box the tokens come out of. The host embeds, the fabric multiplies, the host normalizes between layers, and when the encoder state is finished the host decodes it. Six decoder layers and an LM head, all in numpy. The fabric has no embedding table, no decoder, and not enough block RAM for a single decoder weight matrix.

Composition

Golden model first and RTL second kept finding real upstream defects, which the table records module by module:

Comparison

Across layers and across the run

Comparison against golden reference, FPGA model and hardware

Fault injection

Fault injection flips one stored bit on purpose, in a weight or in an activation, and then measures how far the final output moves. A quantized datapath can absorb a wrong value or it can amplify it, and nothing in the design says which one without a measurement. The spec set the ceiling at 0.5 LayerNorm units, so this is the test that decides whether the design sits inside it.

The release check is one command, and the recorded v1.0 re-run reads as follows:

$ bash outputs/run_regression.sh
== model-level golden checks ==
PASS  taylor_exp_model.py
PASS  smx_e2e_model.py (2.99 < 4.0)
PASS  ffn_model.py
PASS  encoder_e2e_model.py (0.0296 < 0.05)
PASS  multilayer_e2e_model.py (0.0047 < 0.01)
PASS  greedy_decode_model.py
PASS  bidirectional_demo.py
PASS  beam_search_model.py (13 >= 13)
PASS  robustness_model.py
== RTL (ghdl) checks ==
PASS  ghdl:gemm_latency_tb
==
REGRESSION: ALL GREEN
exit=0

Evidence tiers & the exit bar

Three numbers still do not exist in the repository and are not charted on this page: a BLEU score, since no trained checkpoint exists, and a board power figure and a measured silicon latency, since the board run checks values rather than timing anything. Post-synthesis utilization and timing did not exist when v1 froze; they exist for v2 and are charted in 05.

05 On silicon

Synthesis rejected v1. What the rebuild changed, and what the board proved.

Everything above froze at v1, before synthesis. Put through Vivado, v1 declared 212% of the device's block RAM and 18,100 LUTs, or 126%, and did not fit. The re-architecture that followed moved every array into a BRAM primitive and every multiply into a DSP, and widened the encoder to the 512 dimensions the real MarianMT checkpoint uses. That design fits, meets timing, and has since run on the board. This section is what the machine turned out to be.

The three domains

An FPGA that translates is the wrong picture. The fabric performs a transformer encoder's matrix multiplies and nothing else; an x86 host stitches the layers together and runs the entire decoder, and the ARM between them is a courier that moves bytes and starts jobs.

The split, to scale

Making it fit

v1’s architecture did not fit this device and was never going to. Synthesis put it at 18,182 LUT, 126% of the part, with 2 DSP slices and zero block RAM: every array became flip-flops and a mux tree, and every int8 multiply became fabric. Both causes were measured rather than guessed. First, the design read 60 values out of an array in one cycle, and no memory primitive on any device serves that. Block RAM has 2 read ports and LUTRAM 16. So qk_score_bram alone spent 11,587 LUT, 80% of the whole device, to hold 2,688 bits. Second, int8×int8 sits below Vivado’s threshold for inferring a DSP, which the same run proved by contrast: smx_v_calc’s 32×8 multiply is what inferred the design’s only 2 slices.

Sixteen implementation runs followed:

◂ swipe to pan the table ▸

The run worth a sentence is run 10, because it failed on a flow defect and not a design one. synth_design had never been given -mode out_of_context, so Vivado treated encoder_layer as a whole chip and demanded a package pin for every port bit. 171 bits against 100 sites on clg400, reported as [Place 30-58] IO placement is infeasible. Run 11 is the same netlist with the flag set. Nothing about the design was wrong. Telling those two kinds of failure apart is the skill this page is about.

Inside the fabric

The memory budget is the architecture. v1 kept its arrays in logic and inferred its multipliers, which is how a 6-layer encoder came to ask for 212% of the block RAM on a part that has 50 blocks. v2 forces every array into a BRAM primitive and every multiply into a DSP48E1, and buys the latency back with a double-buffered weight window and a softmax that multiplies by a reciprocal rather than dividing.

Why the tape was taken apart

The board replays a job tape: a flat list of records that write weights, write activations, run a job and dump the result. One sentence came to 64.3 MB against an on-chip ceiling near 64 KB. Counting the records first changed the problem. 98.6% of the tape was weight rows, and weight rows do not change between sentences.

Onto the board without the vendor stack

The documented path from a bitstream to a bare-metal application is Vitis: 60 to 100 GB, and the 2022.2 release does not support this machine’s OS. None of it turned out to be necessary. The hardware handoff file already contains ps7_init_gpl.c. That is the processing system’s whole configuration, 12,500 lines of C that define their own mask_write and mask_poll as plain volatile pointer writes and depend on nothing else. So the vendor platform reduced to a linker script, a crt0 and a UART driver, built with gcc-arm-none-eabi and loaded with openocd: about 200 MB of toolchain instead of 60 to 100 GB. The three generated driver files are used unmodified, because editing verified code to suit a new build would mean the thing on the board is no longer the thing that was checked.

ps7_post_config() raises the level shifters between the processing system and the fabric, and until it runs AXI never reaches the programmable logic at all. And the caches stay off: the ACT_DATA register auto-increments when it is accessed, so a dirty cache line that never reached the fabric would come back looking exactly like a datapath bug.

Three defects sat between “the bitstream loads” and “the board prints”, and all three were silent:

There was no output to go on. What localised it was the halt state: it had moved from System at pc 0xffffff28 to Supervisor. crt0 had plainly run, so the CPU was not the suspect and the search narrowed to the peripheral. The full session, every stage of it, is recorded on the board bring-up page.

Chain of custody

Four representations of the same computation, each required to agree with the one before it at zero tolerance: a float32 reference in numpy, a bit-exact int8 model of the datapath, the RTL in GHDL, and the board. The last link is a vector replay. The fabric computes, and the expected values sit in DDR only to be compared against.

The test was not written for the board. sim/ops_cmd.txt is the register sequence for every job type the design implements, generated from the bit-exact int8 model and already run against the RTL in GHDL before any hardware existed. sw/board/vectors.c replays that same sequence against the real fabric over AXI4-Lite and compares every value returned. The comparison then runs twice: once by the ARM core against a copy of the expected tape in DDR, once on the host against the reference on disk. A bug in one comparator cannot pass both. The two clips below are that run and its negative control.

06 Conclusion

The project’s numbers in one place, with the open claims beside them.

v1.0 froze in July with a design that did not fit and a bitstream nobody built. Every number in it came from a model. The table is what a month of rebuilding moved: correctness changed evidence tier and speed did not.

Six claims on this page are not settled by the board run. The table says what is missing for each one, because leaving one out would be worse than saying nothing.

Each item closes something in the two tables above. The first one is what makes a timing number worth reporting.

Resources and timing are post-implementation in context (Vivado 2022.2, xc7z007sclg400-1, run 16, 100 MHz clk_fpga_0). Silicon results are a vector replay over JTAG, which measures values and not wall-clock time, so the latency figures in 04 remain cycle-model projections.