Primary-source-grounded, Code-First deterministic evaluation suite for Japanese enterprise LLMs and RAG pipelines—covering FY2026 (R8) tax revisions, labor & MHLW calculations, the 2026 Toriteki Act, and rule-grounded DuckDB/SQLite queries.
Every task embeds verified excerpts from e-Gov Law API v2, National Tax Agency (NTA) R8 Tax Revision pamphlets/FAQs, JFTC Toriteki Act guidelines, MHLW Ikuji-Jitan rules, and NAOJ 2026 calendar—all tagged with Japan PDL 1.0 attributions.
Gold outputs are generated deterministically via Decimal solvers and read-only DuckDB/SQLite execution. Strict All-pass grading rejects hedging prose (allow_hedging: false) and tests Abstention on out-of-scope rules and 100% NULL DB columns.
temperature=0)| Model & Mode | All-Pass | FMR | Abst | Cat-1 | Cat-2 | Cat-3 | Cat-4 |
|---|---|---|---|---|---|---|---|
gemini-3.8-flash (Closed-book) |
0.0% (0/10) | 0.3452 | 0.0% | 0/2 | 0/3 | 0/2 | 0/3 |
gemini-3.5-flash-lite (Open-book) |
30.0% (3/10) | 0.5024 | 100% | 1/2 | 0/3 | 0/2 | 2/3 |
gemma-4-31b-it (Open-book) |
80.0% (8/10) | 0.9457 | 100% | 1/2 | 3/3 | 1/2 | 3/3 |
gemini-3.1-pro-preview (Open-book) |
80.0% (8/10) | 0.8714 | 100% | 1/2 | 3/3 | 2/2 | 2/3 |
gemini-3.8-flash (Open-book) |
100.0% (10/10) | 1.0000 | 100% | 2/2 | 3/3 | 2/2 | 3/3 |
| Category | Sample Tasks | Primary Grounding & Key Traps |
|---|---|---|
| Cat-1: Invoice R8 | 2 tasks (incl. 1 abstention) | R8 70% transitional credit, per-supplier ¥100M cap allocation, per-rate invoice rounding, corporate 3割特例 exclusion |
| Cat-2: Tax & Labor | 3 tasks | Art. 204 two-tier withholding (10.21% / 20.42%), R8 2026-11-30 YEA boundary & R9 single-parent rule, MHLW Ikuji-Jitan 484,121 cap |
| Cat-3: Toriteki & Cal | 2 tasks | 2026-01-01 Toriteki Act (>300 employee rule, 60-day payment deadline, 14.6% delay interest), 2026-05-06 & 2026-09-22 Citizen's Holiday |
| Cat-4: Grounded SQL | 3 tasks (incl. 1 abstention) | EDINET DuckDB YoY excluding is_amendment = TRUE, JP OpenData SQLite tier1/tier2 permit filter, 100% NULL flood_hazard_rank abstention |
git clone https://github.com/terselogic/jp-bizdata-eval.git
cd jp-bizdata-eval
uv sync --extra dev
uv run pytest -v
uv run jp-eval verify-reference
令和8年度税制改正・取適法・労働基準法・EDINET / 法人オープンデータSQLに加え、自社RAG規定・社内業務ルールを組み込んだ非公開評価パック(マルチシード・回帰テストスイート)の作成・先行アクセスを受け付けています。