Budget LLMs on Korean
Budget-tier LLMs from Korea, the US and China on Korean linguistics, named-entity recognition, legal-case classification and instruction following.
Five similarly priced models ($0.05–0.15 in, $0.20–0.60 out per 1M tokens) on four Korean tests: linguistic knowledge (KoBALT), named-entity recognition (KLUE-NER), picking the case name of a court judgment (LBox), and following verifiable instructions written in Korean (HanIFEval). The write-up is in this post, and how HanIFEval was built is in another.
Summary
Score (in % unless the test's metric says otherwise), with the 95% interval below it. Click a column to sort. A dash means the test was not run in this configuration.
| Model | LinguisticsKoBALT | ExtractionKLUE-NER | LegalLBox casename | Instruction followingHanIFEval |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | 81.7 78.7–84.4 | 78.6 76.8–80.4 | 84.6 82.2–86.7 | 97.2 95.2–98.4 |
| GPT-6 Luna | 82.4 79.4–85.1 | 74.8 72.9–76.6 | 84.5 82.1–86.6 | 93.2 90.5–95.3 |
| GLM 5.3 Flash | 71.0 67.5–74.2 | 76.4 74.5–78.3 | 80.7 78.1–83.0 | 95.3 92.9–97.0 |
| Solar Pro 4 | 72.7 69.3–75.9 | 76.4 74.4–78.2 | 81.4 78.9–83.7 | 93.0 90.2–95.1 |
| Solar Mini 4 | 59.1 55.5–62.7 | 65.1 62.6–67.6 | 77.0 74.3–79.5 | 92.5 89.7–94.7 |
KoBALT Linguistics · Accuracy · n = 700
700 expert-written ten-option questions on Korean linguistics (syntax, semantics, pragmatics, phonetics/phonology, morphology), released 2025-05. A truncated or unparsed reply counts as not correct. Source
- GPT-6 Luna 82.4% · 95% 79.4–85.1 · 577/700 · $0.42
- DeepSeek V4.1 Flash 81.7% · 95% 78.7–84.4 · 572/700 · $4.35
- Solar Pro 4 72.7% · 95% 69.3–75.9 · 509/700 · $1.50
- GLM 5.3 Flash 71.0% · 95% 67.5–74.2 · 497/700 · $6.8291 replies truncated at 64k tokens; on the 609 it answered, 81.6% correct.
- Solar Mini 4 59.1% · 95% 55.5–62.7 · 414/700 · $1.61
KLUE-NER Extraction · Entity-level micro-F1 × 100 · n = 1,000
1,000 KLUE-NER v1.1 dev sentences (news and movie reviews), six entity types. The model copies the sentence and marks each entity; a reply whose copy differs from the sentence scores as no entities found. Interval: sentence-level bootstrap. Source
- DeepSeek V4.1 Flash 78.6 · 95% 76.8–80.4 · n = 1000 · $2.00
- GLM 5.3 Flash 76.4 · 95% 74.5–78.3 · n = 1000 · $1.03
- Solar Pro 4 76.4 · 95% 74.4–78.2 · n = 1000 · $1.23
- GPT-6 Luna 74.8 · 95% 72.9–76.6 · n = 1000 · $0.14
- Solar Mini 4 65.1 · 95% 62.6–67.6 · n = 1000 · $0.95
LBox casename Legal · Accuracy · n = 1,000
Pick the case name of a Korean court judgment from its statement of facts, among 100 case names (1,000 items). In 282 items the case name appears in the facts; the models separate on the other 718. Source
- DeepSeek V4.1 Flash 84.6% · 95% 82.2–86.7 · 846/1000 · $1.15
- GPT-6 Luna 84.5% · 95% 82.1–86.6 · 845/1000 · $0.28
- Solar Pro 4 81.4% · 95% 78.9–83.7 · 814/1000 · $0.41
- GLM 5.3 Flash 80.7% · 95% 78.1–83.0 · 807/1000 · $1.46
- Solar Mini 4 77.0% · 95% 74.3–79.5 · 770/1000 · $0.76
HanIFEval Instruction following · Prompt-level strict accuracy · n = 429
429 verifiable instructions (word counts, forbidden words, formats, repeat-the-request) from a checker-consistent Korean translation of IFEval, v1.1. An item counts only if every instruction in it is followed. Source
- DeepSeek V4.1 Flash 97.2% · 95% 95.2–98.4 · 417/429 · $0.73
- GLM 5.3 Flash 95.3% · 95% 92.9–97.0 · 409/429 · $0.91
- GPT-6 Luna 93.2% · 95% 90.5–95.3 · 400/429 · $0.16
- Solar Pro 4 93.0% · 95% 90.2–95.1 · 399/429 · $0.47
- Solar Mini 4 92.5% · 95% 89.7–94.7 · 397/429 · $0.34
Models
| Model | Org | Served by | Price in / out |
|---|---|---|---|
| DeepSeek V4.1 FlashServed by a third-party fp8 endpoint, not DeepSeek's own. | DeepSeek (CN) | StreamLake (fp8, third-party) | $0.15 / $0.60 |
| GPT-6 Luna | OpenAI (US) | OpenAI | $0.10 / $0.50 |
| GLM 5.3 FlashReasoning cannot be turned off. 91 of 700 KoBALT replies hit the 64k-token limit and count as not correct. | Z.AI (CN) | Z.AI (fp8) | $0.15 / $0.50 |
| Solar Pro 4 | Upstage (KR) | Upstage | $0.09 / $0.36 |
| Solar Mini 4Run after the other four (2026-10-03); its pairwise tests form a separate family, so the others' significance results are unchanged. | Upstage (KR) | Upstage | $0.05 / $0.20 |
Price: USD per 1M tokens (input / output), OpenRouter list price for the pinned provider, checked 2026-09-29 (Solar Mini 4: 2026-10-03). Cost figures above are for the scored rows only.
Read before comparing
- One configuration per model: one pinned provider, reasoning on at the provider default, temperature 0, max_tokens 64,000, one sample per item. Run-to-run variance is not measured.
- Contamination is not controlled: KLUE (2021), LBox (2022) and KoBALT (2025-05) were public with answers before these models were released. HanIFEval is new (2026-10), but its English source, IFEval, is from 2023.
- Non-answers count as wrong. GLM 5.3 Flash's KoBALT score is low mainly because 91 of 700 replies were still reasoning at 64k tokens.
- KLUE-NER is scored on the dev split (test labels are withheld), with a strict copy rule: a reply that changes the sentence, even by an added final period, scores zero entities. GPT-6 Luna loses most under this rule.
- HanIFEval is near ceiling for these models (92–97%) and is model-validated, not human-validated. Its translator was Gemini and its reviewers Claude and GPT-6; GPT-6 Luna is a GPT-6-family model.
- Significance: McNemar exact tests (bootstrap for NER) with Bonferroni over the 6 pairs of the first four models; Solar Mini 4, added later, is tested in its own family of 4 pairs.
- Cost covers the scored rows only, at OpenRouter prices on the run dates.
Changelog
- First published: 5 models, 4 Korean tests (KoBALT, KLUE-NER, LBox casename, HanIFEval v1.1).