JP BizData Eval (Japanese Regulatory & SQL Benchmark)

Primary-source-grounded, Code-First deterministic evaluation suite for Japanese enterprise LLMs and RAG pipelines—covering FY2026 (R8) tax revisions, labor & MHLW calculations, the 2026 Toriteki Act, and rule-grounded DuckDB/SQLite queries.

Request Expanded Pack / Custom Eval GitHub (10-Task Sample & Scorer) Hugging Face Sample

Benchmark Design (Zero-Compromise Verification)

Primary Sources Embedded (Not Formula Hints)

Every task embeds verified excerpts from e-Gov Law API v2, National Tax Agency (NTA) R8 Tax Revision pamphlets/FAQs, JFTC Toriteki Act guidelines, MHLW Ikuji-Jitan rules, and NAOJ 2026 calendar—all tagged with Japan PDL 1.0 attributions.

Code-First Gold + Anti-Hedging + Abstention

Gold outputs are generated deterministically via Decimal solvers and read-only DuckDB/SQLite execution. Strict All-pass grading rejects hedging prose (allow_hedging: false) and tests Abstention on out-of-scope rules and 100% NULL DB columns.

Verified Baseline Results (Curated 10-Task Sample, temperature=0)

Model & Mode All-Pass FMR Abst Cat-1 Cat-2 Cat-3 Cat-4
gemini-3.8-flash (Closed-book) 0.0% (0/10) 0.3452 0.0% 0/2 0/3 0/2 0/3
gemini-3.5-flash-lite (Open-book) 30.0% (3/10) 0.5024 100% 1/2 0/3 0/2 2/3
gemma-4-31b-it (Open-book) 80.0% (8/10) 0.9457 100% 1/2 3/3 1/2 3/3
gemini-3.1-pro-preview (Open-book) 80.0% (8/10) 0.8714 100% 1/2 3/3 2/2 2/3
gemini-3.8-flash (Open-book) 100.0% (10/10) 1.0000 100% 2/2 3/3 2/2 3/3

4 Covered Domains in the Public 10-Task Sample

Category Sample Tasks Primary Grounding & Key Traps
Cat-1: Invoice R8 2 tasks (incl. 1 abstention) R8 70% transitional credit, per-supplier ¥100M cap allocation, per-rate invoice rounding, corporate 3割特例 exclusion
Cat-2: Tax & Labor 3 tasks Art. 204 two-tier withholding (10.21% / 20.42%), R8 2026-11-30 YEA boundary & R9 single-parent rule, MHLW Ikuji-Jitan 484,121 cap
Cat-3: Toriteki & Cal 2 tasks 2026-01-01 Toriteki Act (>300 employee rule, 60-day payment deadline, 14.6% delay interest), 2026-05-06 & 2026-09-22 Citizen's Holiday
Cat-4: Grounded SQL 3 tasks (incl. 1 abstention) EDINET DuckDB YoY excluding is_amendment = TRUE, JP OpenData SQLite tier1/tier2 permit filter, 100% NULL flood_hazard_rank abstention

Quick Start (Run Locally in Under 1 Minute)

git clone https://github.com/terselogic/jp-bizdata-eval.git
cd jp-bizdata-eval
uv sync --extra dev
uv run pytest -v
uv run jp-eval verify-reference

日本語概要・拡張パック(60〜100問規模)/カスタムEvalのご相談

令和8年度税制改正・取適法・労働基準法・EDINET / 法人オープンデータSQLに加え、自社RAG規定・社内業務ルールを組み込んだ非公開評価パック(マルチシード・回帰テストスイート)の作成・先行アクセスを受け付けています。