IFEval
52.3
prompt-level, strict
The first model of AhoosAI's own is available. Better for everyone, not only for code.
It answers in the language you wrote in, and it is at home in Persian. Most coding models are not: they write layouts that look right in English and come apart the moment the text runs the other way, and they slip back into English halfway through an answer. This one was trained out of both habits.
A right-to-left page that needs no fixing afterwards, a form a screen reader can use, a container that does not run as root. Small things, each of which someone otherwise notices later.
HumanEval
61.6
pass@1, greedy
MBPP
67.2
pass@1, 3-shot
Public benchmarks
Ours were run with lm-evaluation-harness. Beside each is the figure its own maker published. These were not measured under the same conditions — every maker uses its own harness and its own settings — so read the column as context rather than a ranking.
Our base publishes 88.4 on HumanEval and the same weights with this adapter scored 61.6 here. The two were taken differently: the published one asks the model as you would in a chat and reads the code out of its answer, ours asks it to continue a half-written function. A model tuned for conversation does worse at the second, adapter or no adapter.
One run settles it: the same harness and the same settings with the adapter switched off. Until that exists, 61.6 is a number from this setup rather than a verdict.
IFEval has no published figure for this base, so it stands on its own above. It is also the one a behaviour fine-tune has any business moving.
What it changed
Our own suite: 105 tasks, each with a rule a machine can check, put to the same weights with the adapter off and on. This one is a comparison — both sides, same tools, same day.
Frontend
+6
Before
9/20
After
15/20
Backend
+1
Before
14/20
After
15/20
Debugging
0
Before
19/20
After
19/20
Language
0
Before
24/25
After
24/25
Image specs
-3
Before
18/20
After
15/20
84 to 88 overall. The first fine-tune this project ran scored one task behind its base; this is four ahead.
The gain is where the base was weakest. Interface work went from 45% to 75%. Nothing else in the suite was that weak, and nothing else moved that far.
Nothing moved at the ceiling. Debugging at 19/20 and language at 24/25 stayed exactly there. A fine-tune that has learned the wrong thing shows up here first.
The one area that lost ground is the one with no training data behind it. The image set was deferred. It cost three tasks — in the score, not guessed at.
How it was trained
| Base | Qwen2.5-Coder-7B-Instruct |
| Method | QLoRA · r=16 · alpha=32 |
| Trained parts | 40,370,176 of 7,655,986,688 · 0.53% |
| Data | 3,462 examples — 2,262 written for this, 1,200 public |
One pass through the data was enough, and we checked
Training can read the data once or many times. Each extra pass doubles the cost, so the question is whether the second one buys anything. We measured the error on examples the model never saw, five times during the single pass:
| step 50 | 0.7029 | — |
| step 100 | 0.6621 | +0.0408 |
| step 150 | 0.6433 | +0.0188 |
| step 200 | 0.6389 | +0.0044 |
| step 210 | 0.6391 | -0.0002 |
Each stretch improved less than half as much as the one before it, and the last ten steps made it very slightly worse. The model had finished learning before the pass ended. A second one would have cost twice as much for nothing.
The same number, twice
IFEval was run twice — two different rented machines, days apart, installed from scratch each time.
| first run | 0.5231 | 0.6163 |
| second run | 0.5231 | 0.6163 |
Identical to four decimal places. That matters: a benchmark that gives a different answer each time it runs is measuring the weather, not the model. This one is measuring the model.
Trademarks and sources
OpenAIAnthropicGoogle GeminiQWenDeepSeekMeta
The names and marks above belong to their owners and appear here only to identify whose published figure each row describes. Nothing on this page is endorsed by, affiliated with, or produced with any of them.
Brand marks come from Simple Icons, whose icon data is released under CC0 while the marks themselves remain their owners' trademarks.
Published figures are each maker's own, from their model cards and technical reports: the Qwen2.5-Coder technical report (arXiv:2409.12186), the DeepSeek-Coder paper (arXiv:2401.14196), OpenAI's GPT-4o and o1-preview system cards, Anthropic's Claude 3.5 Sonnet model card, Google's Gemini 1.5 technical report, and Meta's Llama 3.1 model card.
Our own numbers come from lm-evaluation-harness and from this project's own suite, whose full output — every prompt, every answer, every check, both sides — is published alongside the code.