← projects

VHDL Transformers Agentic Workflows

Mirage — A Transformer Accelerator Designed by an AI Agent

Mirage is an open-source transformer accelerator designed by an AI agent, specifically Kimi K3. It runs Transformer encoder inference for English to Arabic and Arabic to English machine translation. It targets the Digilent Cora Z7-07S (Zynq-7007S), a board that carries 14,400 LUTs, 66 DSP48E1 slices, 50 RAMB36 blocks and one 667 MHz ARM Cortex-A9. This page documents the agentic workflow that built it against a 19-task plan, a week-by-week schedule of module-specific features in which each core was verified before the next, from the matrix multiply array through multi-head attention to the fully integrated encoder.

Fahad Tajiki Jul 2026 GitHub ↗ sim + model evidence · no silicon

Mirage top-level block diagram, four quadrants: a 12 by 5 DSP matrix multiplication array, multi-head attention with Q K V weight BRAMs and a finite state machine, the softmax path with max finder, Taylor approximation and quantizers, and the feed-forward network with projection, hidden and output layers behind BRAM caches. — Fig. 1 · The provided Mirage top-level block diagram, cropped to the diagram body from `Mirage/images/mirage-top-level-bd.png`. The four cores it names are where the plan's audits and drafts land. Select any image on this page for its full-resolution file.

01 The problem

In the silicon budget, a 6-layer encoder with 256-dimensional activations has to run on a part that offers 66 DSP48E1 multipliers and about 225 KB of BRAM. Since the full stack cannot fit, one physical encoder layer stands in for all six and is walked six times per sentence. Pre-synthesis, the multiplier budget can then be expressed like this:

◂ swipe to pan the diagram ▸

Fig. 2 · 65 of 66 DSP48E1 slices allocated before synthesis (reports/w11-resource-map.md).

The other constraint is trust, since a chip is a long chain of decisions where a week-3 arithmetic error can quietly poison week 9. Every task closes on evidence for that reason, and every claim on this page carries the tier of evidence behind it.

02 Architecture

The four roles

The structure of this project is a chain of four parts, and each part has exactly one job. The agent is the harness loop in workflow/ and harness/, which picks one task at a time from the plan and refuses to close it until the files it promised exist on disk. The model is Kimi K3, which takes each brief and does the engineering work of writing the reference models, drafting and repairing the VHDL, and leaving a report behind. The Mirage transformer is the accelerator itself, the design that all of that work lands in. Together the agent, the model and the transformer constitute the workflow, while I steer the design process by writing the spec, building the gates, and reviewing what comes out of each phase:

◂ swipe to pan the diagram ▸

Fig. 3 · agent → model → Mirage transformer, together the workflow, with the human steering from outside it.

Substantial work on Mirage predated my involvement, since the starting point was a provided minimal codebase that was roughly 40% complete. That base contributed the matrix multiplication, attention and softmax cores, the block diagram in Fig. 1 and the model scripts, and it is credited directly as the upstream Mirage repository. The workflow treated all of that code as read-only for the entire run.

The harness

The harness is the half of the workflow that never trusts a transcript, and each block below can be selected for the note recorded against it:

◂ swipe to pan the diagram ▸

harness Select any block. The artifact gate is the one that matters.

data · artifacts gate checks purple = the model

Fig. 4 · The harness (workflow/, harness/hooks.py). Briefs come from the state file and evidence comes from ghdl and spec-first goldens, while completion is decided by a gate that reads the disk.

Write-up for this section is still in progress.

The datapath

In the datapath, int8 weights stream in from DDR while int16 activations stay inside the fabric, and four of the blocks below carry real defects that the loop found:

◂ swipe to pan the diagram ▸

datapath Select any block for the audit note recorded against it.

data flow control · prefetch heavy edge = layer reuse ×6 dashed outline = model-drafted

Fig. 5 · Datapath from spec/mirage.state.json and the W6 to W11 reports. Reusing one layer six times is what makes it fit.

03 The loop & the ledger

A task the loop can act on has to be falsifiable, so "verify the softmax" cannot be a task while "smx_core matches the Python golden for 8 heads and 4 tokens" can be one. The loop below runs five verbs against 19 such tasks, and the timeline under it records when each closure landed:

◂ swipe to pan the diagram ▸

Fig. 6 · The only path from "I think I'm done" to closed runs through the gate, and the only way past a block is a human. Contract: harness/loop.py, workflow/query.py.

◂ swipe to pan the timeline ▸

ledger Select any task, or step through all 19 in closure order.

Fig. 7 · All 19 closures from spec/mirage.state.json in order. Select a task, or step through the run, for what it found and when it closed.

The gate itself is one command, and a refusal is recorded verbatim as follows:

the gate, refusing

# close a task whose declared report does not exist yet
$ python -m workflow.cli complete p1/w3 --artifacts reports/w3-exp-error-histogram.md
REFUSED p1/w3: acceptance not verified
  artifact missing: reports/w3-exp-error-histogram.md
  fix, or re-run with --force to override
$ echo $?
2

Format as implemented in workflow/cli.py, where --force was used zero times.

What the run spent

Both sittings were logged end to end, so the model's spend is measured from the session logs rather than estimated, and the numbers below are rounded from those logs:

live session time

≈9.2 h5.6 h on Jul 24 and 3.6 h on Jul 25, around the overnight pause

model calls

308Kimi K3 requests across the 19 tasks

fresh tokens

≈583k≈416k read in as prompts and ≈167k written out

cached reads

≈30Mprompt tokens re-read from cache to keep context warm

Provider-measured counts from the Kimi wire logs, summed over the two sittings that closed all 19 tasks. A 6.2 h session after the freeze is not counted.

Where the phases landed

The plan's phases land inside the four cores of the block diagram, which are shown here as direct crops from Fig. 1:

Multi-head attention quadrant of the Mirage block diagram: Q K V weight BRAMs, token caches and a finite state machine. — Fig. 8 · The multi-head attention core, where P1 opened. W1 found the `score_sum` overflow, and W6 integrated all 8 heads at worst 3.98 LSB.

Softmax quadrant of the Mirage block diagram: max finder, Taylor approximation and quantizers behind BRAM caches. — Fig. 9 · The softmax path, where W3 exposed the clamp defect and W4 and W5 drafted `smx_cache` and `smx_core`.

Feed-forward network quadrant of the Mirage block diagram: projection, hidden and output layers behind BRAM caches with a finite state machine. — Fig. 10 · The feed-forward network core, where P2 ran. W7 found it failing 10 of 10 trials and closed it with a redesign spec.

Matrix multiplication quadrant of the Mirage block diagram: a 12 by 5 array of multiply-accumulate DSP cells. — Fig. 11 · The 12×5 DSP array every phase leans on, measured in W14 at 36 of 36 dot products with start to done in 1 clock.

04 Results

Golden model first and RTL second kept finding real upstream defects, which the table records module by module:

Defects found in upstream RTL (reports W1 to W9)
Module	As found	Root cause	After the loop
`taylor_exp`	64/64 sampled points ≥100% error	implicit clamp swallowed the input (Q-format)	0.9683% max error (Q9, \|x\| ≤ 0.5352)
`smx_core` e2e	attention 87–107 int8 LSB from golden	exp input scaling off by ≈2¹⁸	worst 2.99 LSB, RMS 0.52
`ffn_core`	10/10 trials fail and it won't compile	`no declaration for "mac_array"`, 278 KB vs 225 KB BRAM	redesign spec: 16-bit activations, streamed weights
6-layer stack	drift >1 LN unit by layer 1	softmax chaotic under int8 inter-stage quantization	int16 inter-stage: 0.0062 LN worst over 3 layers

Composition

Fig. 12 · Where the 66 DSP multiplier slices go: 60 to the matrix tile, 4 to the exponential, 1 to the drafted softmax and 1 free (reports/w11-resource-map.md).

Fig. 13 · Where the 50 block RAMs go: 42 allocated and 8 free. The split only holds because the FFN's 262 KB of hidden and output weights stream from DDR instead of sitting on chip.

Comparison

Fig. 14 · The baseline came from the software planning model, and the W15 pass re-priced it with the gemm rate verified in GHDL while overlapping weight streaming with compute (reports/optimization-report-v1.md).

Fig. 15 · Real-time NMT needs about 15 tok/s, so the optimized figure carries roughly 24× headroom, and the pipeline is now DDR-bound with the crossover near 192 MB/s.

Fig. 16 · DSP at 98% is expected to fail the acceptance check (W15 · GHDL), and the documented mitigation time-multiplexes the gemm as 2×30 (reports/w11-resource-map.md).

Across layers and across the run

Fig. 17 · Drift is how far the hardware's numbers wander from the software reference as rounding errors compound across stacked layers, measured in LayerNorm units. Every int8 format crosses the 0.01 budget by layer 1, while int16 stays flat, which is why the datapath keeps int16 between stages (reports/w9-multilayer.md).

purple band = Kimi K3 live session unshaded gap = my pause · review

Fig. 18 · Each step is one task closing, timestamped in spec/mirage.state.json. The shaded bands are Kimi K3's two live sessions, and the unshaded gap is where I paused the run overnight and reviewed.

Comparison against golden reference, FPGA model and hardware

Fig. 19 · The software side is the pure Python golden reference timed on the host CPU, and the FPGA side prices the design's cycle count at 200 MHz using the gemm rate measured in GHDL simulation. The 1,700× is against interpreted Python, and the report itself puts an optimized int8 CPU baseline at roughly 10 to 20× (reports/evaluation-report-draft.md).

Fig. 20 · The bitstream build for the Cora Z7 exists and was deliberately not fabricated, so no silicon number exists in the repository. The gray bar carries the cycle model's projection and is marked not measured, since no board run exists.

Fig. 21 · Fault injection flips one stored bit at a time, in a weight or an activation, and measures how far the output moves. Both cases absorb the hit at least 10× inside the 0.5 limit, and the matrix's one real finding is an FFN accumulator that can overflow int32 at adversarial all-max input (reports/robustness-matrix.md).

The release check is one command, and the recorded v1.0 re-run reads as follows:

reports/final-regression.log

$ bash outputs/run_regression.sh
== model-level golden checks ==
PASS  taylor_exp_model.py
PASS  smx_e2e_model.py (2.99 < 4.0)
PASS  ffn_model.py
PASS  encoder_e2e_model.py (0.0296 < 0.05)
PASS  multilayer_e2e_model.py (0.0047 < 0.01)
PASS  greedy_decode_model.py
PASS  bidirectional_demo.py
PASS  beam_search_model.py (13 >= 13)
PASS  robustness_model.py
== RTL (ghdl) checks ==
PASS  ghdl:gemm_latency_tb
==
REGRESSION: ALL GREEN
exit=0

Evidence tiers & the exit bar

ghdl-simulated

The gemm tile at 36 of 36 dot products with start to done measured at 1 clock, and every model-drafted VHDL file analyzing and elaborating clean.

Model-level golden

All accuracy claims, from LSB drift to token-identical decode, come from bit-accurate Python models against float references, not from full-pipeline RTL sim.

Estimate / projection

Latency, throughput, the 1,700× comparison and the resource map are cycle-model or pre-synthesis figures, and nothing was measured on silicon.

Spec exit criteria and where each stands
Exit criterion	Standing	Status
EN→AR & AR→EN encoder translation	10/10 token-identical vs golden on toy random weights, and no trained checkpoint exists in the repo	model-level
≥3× latency vs CPU baseline	the cycle model clears it with margin, and no int8-CPU comparison exists in the repo	modeled
<90% LUT & <95% BRAM, timing met	LUT 25% and BRAM 84% pre-synth, timing unmeasured, DSP 98% against the task's 90% bar	pre-synth
Open-source repo + build/sim docs	the regression script and per-week reports exist, and no LICENSE file exists	blocked
Demo script + pitch deck	both exist	done

Four numbers do not exist in the repository and are not charted on this page: a BLEU score, since no trained checkpoint exists, a board power figure and a measured silicon latency, since no board run exists, and post-synthesis utilization and timing, since no Vivado run exists.