korean on 0hye

Budget LLMs on Korean

Sat, 03 Oct 2026 00:00:00 +0900

Five similarly priced models ($0.05–0.15 in, $0.20–0.60 out per 1M tokens) on four Korean tests: linguistic knowledge (KoBALT), named-entity recognition (KLUE-NER), picking the case name of a court judgment (LBox), and following verifiable instructions written in Korean (HanIFEval). The write-up is in this post, and how HanIFEval was built is in another.

Made in Korea is not best at Korean: five budget LLMs on four Korean tests

Sat, 03 Oct 2026 00:00:00 +0900

I ran five budget-tier LLMs on four Korean tests. The short version: the two Korean-made models are not the best at Korean. DeepSeek V4.1 Flash and GPT-6 Luna lead. Upstage’s Solar Pro 4 is significantly behind DeepSeek on all four tests, and Solar Mini 4 is last on all four.

The numbers are kept current on the benchmark page. This post explains what was run and how to read it.

TL;DR

Model	KoBALT	KLUE-NER (F1)	LBox	HanIFEval	Cost, all four
DeepSeek V4.1 Flash	81.7%	78.6	84.6%	97.2%	$8.23
GPT-6 Luna	82.4%	74.8	84.5%	93.2%	$1.00
GLM 5.3 Flash	71.0%	76.4	80.7%	95.3%	$10.22
Solar Pro 4	72.7%	76.4	81.4%	93.0%	$3.61
Solar Mini 4	59.1%	65.1	77.0%	92.5%	$3.66

Reasoning on, one run per model, measured 2026-09-29 to 10-03. Bold marks the gaps this post is about, not a winner.

Translate the checker too: building HanIFEval, a Korean IFEval

Sat, 03 Oct 2026 00:00:00 +0900

IFEval is scored by code, not by a judge model. Each item is a prompt plus the arguments its checker uses: “at least 300 words” is the prompt’s text, and relation: at least, num_words: 300 is what the code tests. Translate only the prompt and the two can drift apart. The Korean prompt then asks for one thing while the checker scores another.

I audited an existing Korean translation and found that in 1 item out of 13. So I built HanIFEval, which translates each prompt together with its checker arguments and then checks the pair with code. This post covers why that was needed, how it was built, what Korean does to a substring-based checker, and how five budget LLMs did on it.