What we publish

Every ARCHSLOTH build arrives with its own evaluation attached: the sample count, the harness, the exact file it was measured against, and the failures it produced. Not a claim that it is accurate — the numbers, so you can check.

If we cannot show the numbers, we do not publish the file.

Three rules we hold ourselves to

1 · Q4 is the floor

We do not ship below 4-bit. Sub-4-bit builds make a model small, and they also make it unreliable in ways perplexity will not show you. If a model does not fit your memory at Q4, that is a hardware answer, not a quantization answer — and we would rather tell you that than hand you a file that loads and then quietly fails.

2 · Calibration is not language-neutral

An importance matrix built on English text protects English.

We calibrate across Korean, English, and reasoning data, and we report all three separately. A build that holds its English scores while losing Korean looks healthy in aggregate and is not. We have measured exactly that failure in publicly available expert-pruned builds of a 124B model:

Axis	Original weights	Expert-pruned build
Korean knowledge	12 / 14	0 / 14
English knowledge	8 / 8	8 / 8

The English column is why the failure goes unnoticed. Asked a Korean question, the pruned build answered in Chinese, then repeated itself until it hit the token limit. Aggregate accuracy barely moved.

3 · Failures get names

"78% accurate" hides what actually broke. Our evaluations separate empty output, language collapse, runaway repetition, malformed tool calls, wrong arguments, and plainly wrong answers — because a model that answers Korean questions in Chinese and a model that gets a date wrong need completely different fixes, and one number cannot tell them apart.

How we measure

Denominators are always printed. A score with no denominator is not a score.
Controls are always included. We report a domain we did not optimize for, so you can tell a real gain from a shifted baseline.
The measuring tool is tested before the model is. Our graders are mutation-tested: we feed them known failures and confirm each is caught under a different name. Two real defects in our own grader were caught this way before a single model was scored.
Every comparison names the exact file. Repository, filename, bit width, harness, flags. If you cannot reproduce it, we should not have printed it.

Roadmap

Korean-calibrated Q4 builds of widely used open models. Each one published with a side-by-side evaluation against the most-downloaded existing quant of the same model at the same bit width.

Same model, same bits, same harness. The only variable is the calibration.

We will publish the ones that win. We will not publish the ones that do not.

Who we are

A team in Korea that builds inference acceleration and on-device models. We ship weights that run on CPU-only mini PCs and on Android tablets, with no GPU — which is how we learned that a quant can pass every aggregate benchmark and still be broken in the language our users actually speak.

We are new here. This page has no download counts on it yet, and we would rather it stayed that way until the tables underneath are ours.

_{Open weights, permissively licensed. Reproductions welcome — if a number here does not replicate on your hardware, open a discussion and we will correct it.}