system

You are one role in an automated verified-reasoning pipeline. Your reply is parsed by a program: return exactly what is asked, nothing else. You have no shell or filesystem. Use the web_search tool to verify facts, citations, and definitions rather than relying on recall. Use the code interpreter (Python; sympy/numpy/scipy available) to recompute rather than trusting arithmetic.

Solve the given problem. Show your reasoning.

Use web search for anything you're even remotely unsure about.

user

Is a large share of the reported word-error-rate improvement for Northern Sámi and Maltese speech recognition a scoring artefact — does rescoring the published systems under one fixed normalisation on one held-out test set shrink the improvement?

Non-exhaustive sources that may help:
- Mozilla Common Voice, Scripted Speech v25.0 (release 2026-03): 41,792 total / 28,377 validated hours, 290 languages, CC0 — https://commonvoice.mozilla.org/
- Getman Y. et al., Northern Sami ASR, Interspeech 2024 -- direct fine-tuning WER 36.1-42.7%, best extended fine-tuning 28.8% — https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf
- Getman Y. et al., NoDaLiDa/Baltic-HLT 2025 -- out-of-domain Northern Sami: Whisper 43.15% WER, best system 32.29% — https://aclanthology.org/2025.nodalida-1.19.pdf
- Williams A., DeMarco A., Borg C., SIGUL 2023 -- Maltese: zero-shot Whisper ~80% WER, fine-tuned XLS-R 2B 24.98% on ~50h (MASRI corpus ~40h24m) — https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf
- Whisper large-v3 (Apache-2.0, 99 languages) and large-v3-turbo (MIT) — https://huggingface.co/openai/whisper-large-v3
- Meta Omnilingual ASR (10 Nov 2025) -- 1,600+ languages, code Apache-2.0, corpus (350 languages) CC-BY; supersedes MMS (CC-BY-NC-4.0) for coverage — https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
- HuggingFace Open ASR Leaderboard -- multilingual track currently covers only five languages, none of them European minority languages — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- ML-SUPERB 2.0 multilingual benchmark, Interspeech 2024, arXiv:2406.08641 — https://arxiv.org/abs/2406.08641
- CLARIN ERIC (ESFRI Landmark) and ALT-EDIC (created by Commission decision Feb 2024). NOTE: ELRC closed in Jan 2023 and ELRC-SHARE is shut down -- do not cite it as live infrastructure — https://www.clarin.eu/

reasoning

{
 "summary": [
  "**Researching normalization effects**\n\nI need to research the question about whether rescoring published systems with fixed normalization on a held-out test shrinks improvements. It sounds like I should conduct a comparison to clarify this. I’ll need to inspect relevant papers and data to identify reported word error rate (WER) differences due to scoring. It's essential that my approach is robust, so searching the web for studies could really help."
 ]
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "Northern Sámi Maltese ASR rescoring fixed normalization held-out test scoring artefact WER",
   "site:isca-archive.org/interspeech_2024/getman24b Northern Sami ASR normalization WER test set",
   "site:aclanthology.org/2025.nodalida-1.19 Northern Sami WER normalization test",
   "site:isca-archive.org/sigul_2023/williams23_sigul Maltese WER normalization test set"
  ],
  "query": "Northern Sámi Maltese ASR rescoring fixed normalization held-out test scoring artefact WER"
 }
}

reasoning

{
 "summary": [
  "**Considering experiment details**\n\nIt seems like the question is asking if a large share could be a scoring artifact. Specifically, I think they’re inquiring if rescoring published systems using one fixed normalization on a held-out test set would reduce the improvement seen. However, to provide an answer, I need some data or hypothesis outputs, which aren’t available right now. I guess I should check the paper details regarding normalization and test sets, maybe downloading PDFs if necessary."
 ]
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf"
 }
}

reasoning

{
 "summary": [
  "**Figuring out PDF formatting**\n\nI’m looking to add lines around tables in a PDF for 2025. I wonder if there are options for normalization that I can incorporate as well. Adding those lines could improve clarity and make the data easier to read. It’s a small detail, but it can really enhance the overall presentation! So, I need to explore the formatting tools I have available to make that happen. Let's see what I can do!"
 ]
}

web_search

{
 "action": {
  "pattern": "43.15",
  "type": "find_in_page",
  "url": "https://aclanthology.org/2025.nodalida-1.19.pdf"
 }
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "site:github.com aalto-speech northern-sami-asr normalization WER transcripts",
   "site:huggingface.co Maltese XLS-R 2B 24.98 Williams model",
   "\"24.98\" Maltese ASR HuggingFace model",
   "\"fixed normalisation\" ASR Northern Sámi Maltese"
  ],
  "query": "site:github.com aalto-speech northern-sami-asr normalization WER transcripts"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://github.com/aalto-speech/northern-sami-asr"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://github.com/aalto-speech/northern-sami-asr"
 }
}

web_search

{
 "action": {
  "pattern": "chars_to_ignore_regex",
  "type": "find_in_page",
  "url": "https://github.com/aalto-speech/northern-sami-asr/blob/38580c7ec9337da5cee8ea7aa31d085b7e63363b/scripts/finetuning/huggingface_run_speech_recognition_ctc_multigpu.py"
 }
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "site:huggingface.co aiden williams Maltese XLS-R ASR",
   "site:huggingface.co \"Maltese\" \"XLS-R-2B\" ASR",
   "site:huggingface.co models Maltese ASR Williams Demarco Borg",
   "site:github.com \"The Applicability of Wav2Vec2\" code"
  ],
  "query": "site:huggingface.co aiden williams Maltese XLS-R ASR"
 }
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "site:huggingface.co/MLRS ASR Maltese wav2vec",
   "site:huggingface.co \"Aiden Williams\" ASR Maltese",
   "site:huggingface.co \"MASRI test set\" WER 24.98"
  ],
  "query": "site:huggingface.co/MLRS ASR Maltese wav2vec"
 }
}

reasoning

{
 "summary": [
  "**Investigating baseline data**\n\nI'm looking into a reported improvement in the Maltese baseline from 55.05 to 24.98 on the same MASRI test. However, the old baseline paper likely has different normalization methods. I need to inspect the earlier Mena/Jasper paper and its datasets. It's clear I need to search for and access that PDF to understand the context better. Let's see what specific details come up during the search!"
 ]
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "Mena Maltese Jasper 55.05 WER MASRI test normalization 2021",
   "\"55.05\" WER Maltese Jasper",
   "site:aclanthology.org MASRI Maltese 55.05 WER"
  ],
  "query": "Mena Maltese Jasper 55.05 WER MASRI test normalization 2021"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf"
 }
}

code_interpreter

{
 "code": "vals = {\n\"NS_vs_Whisper_abs\":43.15-32.29,\n\"NS_vs_Whisper_rel\":(43.15-32.29)/43.15*100,\n\"NS_vs_XLSR_abs\":47.70-32.29,\n\"NS_vs_XLSR_rel\":(47.70-32.29)/47.70*100,\n\"NS_2024_direct_to_best_abs\":37.25-28.84,\n\"NS_2024_direct_to_best_rel\":(37.25-28.84)/37.25*100,\n\"Maltese_report_abs\":55.05-24.98,\n\"Maltese_report_rel\":(55.05-24.98)/55.05*100\n}\nvals\n",
 "outputs": []
}

reasoning

{
 "summary": [
  "**Clarifying normalization and evidence**\n\nI need to define what normalization is in this context. It seems essential to gather evidence in 2025 that baselines were rerun instead of copied. The comparison shows an improvement from a score of 28.84 to 47.70 for the old 2024 model on a one-hour out-domain test, which confirms the rerun. I should also look into details about the test set, scoring, and whether the authors used lowercase punctuation stripping. Searching through the code might help clarify this further."
 ]
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://huggingface.co/collections/GetmanY1/sami-parliament-wav2vec2-asr-66699110493b618b9ee2bf21"
 }
}

reasoning

{
 "summary": [
  "**Clarifying normalization details**\n\nI could mention that there’s at least the same reference/test and scoring pipeline as Table 1, but the paper doesn’t fully specify the normalization methods. It indicates a common-set rescore but lacks explicit details about the normalizer, particularly concerning fixed normalization for 2025. I can say the publication doesn't provide enough details for a strict audit. Additionally, I’ll want to ensure the citations are inline and only include what's necessary in the summary to keep it concise."
 ]
}

output

**Answer: only partly—and the two languages differ.**

### Northern Sámi: the improvement survives common-set rescoring

The strongest controlled comparison is already in Getman et al. (2025): previous systems and the new 22,400-hour models were evaluated on the **same one-hour, out-of-domain test set**. The results were:

| System | Common-test WER |
|---|---:|
| Earlier XLS-R extended-fine-tuning system | 47.70% |
| Fine-tuned Whisper | 43.15% |
| New best Large-22K system | 32.29% |

Thus the new system retains:

- **10.86 percentage points**, or **25.2% relative**, over Whisper;
- **15.41 points**, or **32.3% relative**, over the earlier XLS-R system.

Those are substantial gains after common-test rescoring, so they cannot largely be explained by differing published WER conventions. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf))

However, cross-paper headline numbers are misleading. The 2024 figure of **28.84%** came from the eight-hour UIT-SME corpus containing only two speakers, whereas the 2025 common comparison used a different one-hour mixture of read and spontaneous speech with varying audio quality. The same earlier XLS-R system scored **28.84% on UIT-SME but 47.70% on the later test**. That roughly 19-point shift is a test-set/domain effect, not model regression. ([isca-archive.org](https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf))

The 2025 paper uses SCTK and a common reference for its comparison, although it does not specify a sufficiently detailed, versioned text-normalisation contract to reproduce a strict normalisation audit. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf))

### Maltese: the claimed 55.05% → 24.98% improvement is not controlled

The apparent Maltese improvement is **30.07 points**, or **54.6% relative**, but its endpoints are not directly comparable:

- **55.05%** was obtained with Jasper on **MASRI-HEADSET v2**, a 250-utterance test split lasting about **19 minutes**.
- **24.98%** was reported on a different, approximately **one-hour MASRI test set**, containing heterogeneous material such as rural speech, political debates and news. ([um.edu.mt](https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf))

The preprocessing also differed. The 2023 system lowercased text, removed most non-alphabetic characters, retained hyphens and apostrophes, and converted digits to Maltese words. The earlier Jasper configuration explicitly applied **no text normalisation**. ([isca-archive.org](https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf))

Moreover, the earlier Maltese study already reported **21.45% WER for XLS-R 2B** on its 19-minute MASRI-HEADSET test—numerically below 24.98%, though again on a different test. This shows that the “55.05 to 24.98” narrative also depends on selecting Jasper rather than the strongest previously reported neural system. ([um.edu.mt](https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf))

No published experiment appears to rescore the Jasper baseline and the 2023 XLS-R system on one genuinely held-out Maltese set under one fully specified normaliser. Therefore, **the size of the scoring artefact cannot be quantified from the published evidence**.

### Verdict

- **Northern Sámi:** **No**—common-test rescoring changes the headline numbers but leaves a real, sizeable 25–32% relative WER reduction.
- **Maltese:** **Unresolved, but the published 55.05% → 24.98% claim is apples-to-oranges.** It likely overstates progress because of test-set, baseline-selection and normalisation differences, but saying that a “large share” is specifically a scoring artefact requires the missing common-set rescore.

system

You are one role in an automated verified-reasoning pipeline. Your reply is parsed by a program: return exactly what is asked, nothing else. You have no shell or filesystem. Use the web_search tool to verify facts, citations, and definitions rather than relying on recall.

You formalize solutions into proofs. The solutions you receive were
generated by an LLM and may contain hallucinations, wrong facts, or
flawed reasoning: treat every claim as potentially wrong, and do not
assume the solver's reasoning is valid unless you can justify it.

On every call, you must choose one of two actions:

- action="proof": Produce a formal proof for the solution (see
  proof format below). This is the default — start by trying to
  formalize.
- action="reject": If while formalizing you find any substantive
  issues with the solution (incorrect facts, flawed reasoning,
  missing cases, etc.), reject the solution and set reject_reason
  to explain what's wrong; the solver will get this and retry.

A proof is a sequence of states. Each state is a list of strings.
Index 0 is the expression being solved for. It starts as a goal
variable (like "ANSWER") and ends when resolved.

Each step has: state (full list of strings after this step),
justification_type ("citation", "problem_given", or "computation"),
and justification (why).

One transformation per step. Every step must be justified.

When a step's justification is a named mode of inference, use
justification_type: citation. The cited "result" is the mode of
inference itself.

PREMISES MUST BE EXPLICIT. When a step relies on a premise that
isn't already in the previous state — including bounds, conditions,
edge cases, factual claims, or assumptions like "X is a Y" — you
MUST first add a step that introduces the premise explicitly. That
added step needs its own justification. Do not bury premises in
justification text. If you cannot justify a premise, your proof
must handle the case without it (e.g., by considering all cases).

Use web search for anything you're even remotely unsure about.

user

Problem: Is a large share of the reported word-error-rate improvement for Northern Sámi and Maltese speech recognition a scoring artefact — does rescoring the published systems under one fixed normalisation on one held-out test set shrink the improvement?

Non-exhaustive sources that may help:
- Mozilla Common Voice, Scripted Speech v25.0 (release 2026-03): 41,792 total / 28,377 validated hours, 290 languages, CC0 — https://commonvoice.mozilla.org/
- Getman Y. et al., Northern Sami ASR, Interspeech 2024 -- direct fine-tuning WER 36.1-42.7%, best extended fine-tuning 28.8% — https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf
- Getman Y. et al., NoDaLiDa/Baltic-HLT 2025 -- out-of-domain Northern Sami: Whisper 43.15% WER, best system 32.29% — https://aclanthology.org/2025.nodalida-1.19.pdf
- Williams A., DeMarco A., Borg C., SIGUL 2023 -- Maltese: zero-shot Whisper ~80% WER, fine-tuned XLS-R 2B 24.98% on ~50h (MASRI corpus ~40h24m) — https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf
- Whisper large-v3 (Apache-2.0, 99 languages) and large-v3-turbo (MIT) — https://huggingface.co/openai/whisper-large-v3
- Meta Omnilingual ASR (10 Nov 2025) -- 1,600+ languages, code Apache-2.0, corpus (350 languages) CC-BY; supersedes MMS (CC-BY-NC-4.0) for coverage — https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
- HuggingFace Open ASR Leaderboard -- multilingual track currently covers only five languages, none of them European minority languages — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- ML-SUPERB 2.0 multilingual benchmark, Interspeech 2024, arXiv:2406.08641 — https://arxiv.org/abs/2406.08641
- CLARIN ERIC (ESFRI Landmark) and ALT-EDIC (created by Commission decision Feb 2024). NOTE: ELRC closed in Jan 2023 and ELRC-SHARE is shut down -- do not cite it as live infrastructure — https://www.clarin.eu/


Solution: **Answer: only partly—and the two languages differ.**

### Northern Sámi: the improvement survives common-set rescoring

The strongest controlled comparison is already in Getman et al. (2025): previous systems and the new 22,400-hour models were evaluated on the **same one-hour, out-of-domain test set**. The results were:

| System | Common-test WER |
|---|---:|
| Earlier XLS-R extended-fine-tuning system | 47.70% |
| Fine-tuned Whisper | 43.15% |
| New best Large-22K system | 32.29% |

Thus the new system retains:

- **10.86 percentage points**, or **25.2% relative**, over Whisper;
- **15.41 points**, or **32.3% relative**, over the earlier XLS-R system.

Those are substantial gains after common-test rescoring, so they cannot largely be explained by differing published WER conventions. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf))

However, cross-paper headline numbers are misleading. The 2024 figure of **28.84%** came from the eight-hour UIT-SME corpus containing only two speakers, whereas the 2025 common comparison used a different one-hour mixture of read and spontaneous speech with varying audio quality. The same earlier XLS-R system scored **28.84% on UIT-SME but 47.70% on the later test**. That roughly 19-point shift is a test-set/domain effect, not model regression. ([isca-archive.org](https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf))

The 2025 paper uses SCTK and a common reference for its comparison, although it does not specify a sufficiently detailed, versioned text-normalisation contract to reproduce a strict normalisation audit. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf))

### Maltese: the claimed 55.05% → 24.98% improvement is not controlled

The apparent Maltese improvement is **30.07 points**, or **54.6% relative**, but its endpoints are not directly comparable:

- **55.05%** was obtained with Jasper on **MASRI-HEADSET v2**, a 250-utterance test split lasting about **19 minutes**.
- **24.98%** was reported on a different, approximately **one-hour MASRI test set**, containing heterogeneous material such as rural speech, political debates and news. ([um.edu.mt](https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf))

The preprocessing also differed. The 2023 system lowercased text, removed most non-alphabetic characters, retained hyphens and apostrophes, and converted digits to Maltese words. The earlier Jasper configuration explicitly applied **no text normalisation**. ([isca-archive.org](https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf))

Moreover, the earlier Maltese study already reported **21.45% WER for XLS-R 2B** on its 19-minute MASRI-HEADSET test—numerically below 24.98%, though again on a different test. This shows that the “55.05 to 24.98” narrative also depends on selecting Jasper rather than the strongest previously reported neural system. ([um.edu.mt](https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf))

No published experiment appears to rescore the Jasper baseline and the 2023 XLS-R system on one genuinely held-out Maltese set under one fully specified normaliser. Therefore, **the size of the scoring artefact cannot be quantified from the published evidence**.

### Verdict

- **Northern Sámi:** **No**—common-test rescoring changes the headline numbers but leaves a real, sizeable 25–32% relative WER reduction.
- **Maltese:** **Unresolved, but the published 55.05% → 24.98% claim is apples-to-oranges.** It likely overstates progress because of test-set, baseline-selection and normalisation differences, but saying that a “large share” is specifically a scoring artefact requires the missing common-set rescore.

Formalize this into a proof, or reject if it has errors.

reasoning

{
 "summary": [
  "**Verifying sources**\n\nI realize I need to verify the sources thoroughly. There’s a potential issue with the solution regarding the question about scoring an artifact under a fixed normalization with one held-out test. The Northern paper uses the same test, but I wonder if they were \"rescoring published systems under one fixed normalization.\" I must inspect the PDFs and possibly search the web for more clarity on this matter. Let's get started!"
 ]
}

web_search

{
 "action": {
  "type": "search",
  "queries": [
   "site:aclanthology.org/2025.nodalida-1.19.pdf Northern Sami 47.70 43.15 32.29 SCTK",
   "site:isca-archive.org/sigul_2023/williams23_sigul.pdf Maltese 24.98 normalization MASRI test",
   "\"55.05\" Jasper \"21.45\" Maltese ASR",
   "\"MASRI-HEADSET v2\" 250 utterances 19 minutes Jasper"
  ],
  "query": "site:aclanthology.org/2025.nodalida-1.19.pdf Northern Sami 47.70 43.15 32.29 SCTK"
 }
}

reasoning

{
 "summary": [
  "**Inspecting PDF pages**\n\nI'm thinking about inspecting the PDF pages related to the results and methodology. It's important to ensure that I understand everything clearly before moving forward. If there are specific sections I need to focus on, I should make note of those. This will help me get a complete picture and support the task at hand. Let's open the PDF and dive into those pages!"
 ]
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://aclanthology.org/2025.nodalida-1.19/"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://aclanthology.org/2025.nodalida-1.19/"
 }
}

web_search

{
 "action": {
  "type": "open_page",
  "url": "https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf"
 }
}

reasoning

{
 "summary": [
  "**Evaluating error in data**\n\nI think there's a potential error with the figure 55.05 for Jasper in the earlier data augmentation paper. It likely relates to the same 19-minute test, but I need to consider if \"explicitly no text normalization\" applies to the Jasper an4 architecture versus the 55.05 system. I should trace the 55.05 table, likely involving pretraining in English and then fine-tuning with the an4 architecture. I'll need to look into this further to clarify."
 ]
}

web_search

{
 "action": {
  "pattern": "55.05",
  "type": "find_in_page",
  "url": "https://www.um.edu.mt/library/oar/bitstream/123456789/110339/1/Analysis_of_data_augmentation_methods_for_low_resource_Maltese_ASR.pdf"
 }
}

output

{
 "action": "reject",
 "proof": {
  "initial_state": [],
  "steps": []
 },
 "reject_reason": "The Northern Sámi conclusion overstates the evidence. Getman et al. (2025) did evaluate the earlier XLS-R system, fine-tuned Whisper, and new models on the same one-hour test set, reporting 47.70%, 43.15%, and 32.29% WER respectively. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) However, the paper does not document a fixed text-normalisation contract or say that SCTK produced those WERs; SCTK is mentioned specifically for matched-pair significance tests. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) Thus this is evidence that gains survive a common-test comparison, but not the fixed-normalisation rescoring experiment asked about. The solution itself acknowledges that a strict normalisation audit is unreproducible, so its categorical Northern Sámi verdict ('No') does not follow. The defensible verdict is that Northern Sámi shows substantial gains after controlling the test set, while the share attributable to scoring normalisation remains unquantified; Maltese likewise lacks the required common-set, fixed-normaliser rescore."
}

system

You are one role in an automated verified-reasoning pipeline. Your reply is parsed by a program: return exactly what is asked, nothing else. You have no shell or filesystem.

You write the literature-search queries for one verification run, from its
digest: the question (a research proposal's hypothesis), the outcome, the
proof, and every audited claim with the checker's reason. You do not search.
An engine runs every query you return against OpenAlex (title, abstract and
full text of ~250M works) and records the exact URL, the total match count
and the top hits; the adjudicator reads those results, not your opinion.

Return one `searches` entry per claim worth testing:
- every absence claim the run relied on or failed on ("no study reports X",
  "X has not been measured", "no benchmark covers Y"), in the digest's words;
- the proposal's question itself: has it already been answered?

For each, `queries` is a ladder from broad to narrow, as many rungs as the
vocabulary needs. The broadest rung names the topic and must plausibly match
published work; each narrower rung adds a condition the claim turns on.
Cover the field's synonyms and named artefacts (tools, datasets, benchmarks,
versions) across rungs rather than inside one query. A ladder whose broad
rung matches nothing is uncalibrated and proves nothing, so prefer several
plain rungs to one long one.

Query syntax: whole words, stemmed; AND / OR / NOT (uppercase) and
parentheses; "double quotes" for phrases. `from_date` / `to_date` bound the
publication date (YYYY-MM-DD; "" for unbounded). `issns`: journal ISSNs to
restrict to, only when the claim names venues; otherwise [].

user

QUESTION:
Is a large share of the reported word-error-rate improvement for Northern Sámi and Maltese speech recognition a scoring artefact — does rescoring the published systems under one fixed normalisation on one held-out test set shrink the improvement?

Non-exhaustive sources that may help:
- Mozilla Common Voice, Scripted Speech v25.0 (release 2026-03): 41,792 total / 28,377 validated hours, 290 languages, CC0 — https://commonvoice.mozilla.org/
- Getman Y. et al., Northern Sami ASR, Interspeech 2024 -- direct fine-tuning WER 36.1-42.7%, best extended fine-tuning 28.8% — https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf
- Getman Y. et al., NoDaLiDa/Baltic-HLT 2025 -- out-of-domain Northern Sami: Whisper 43.15% WER, best system 32.29% — https://aclanthology.org/2025.nodalida-1.19.pdf
- Williams A., DeMarco A., Borg C., SIGUL 2023 -- Maltese: zero-shot Whisper ~80% WER, fine-tuned XLS-R 2B 24.98% on ~50h (MASRI corpus ~40h24m) — https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf
- Whisper large-v3 (Apache-2.0, 99 languages) and large-v3-turbo (MIT) — https://huggingface.co/openai/whisper-large-v3
- Meta Omnilingual ASR (10 Nov 2025) -- 1,600+ languages, code Apache-2.0, corpus (350 languages) CC-BY; supersedes MMS (CC-BY-NC-4.0) for coverage — https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
- HuggingFace Open ASR Leaderboard -- multilingual track currently covers only five languages, none of them European minority languages — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- ML-SUPERB 2.0 multilingual benchmark, Interspeech 2024, arXiv:2406.08641 — https://arxiv.org/abs/2406.08641
- CLARIN ERIC (ESFRI Landmark) and ALT-EDIC (created by Commission decision Feb 2024). NOTE: ELRC closed in Jan 2023 and ELRC-SHARE is shut down -- do not cite it as live infrastructure — https://www.clarin.eu/

OUTCOME: Declined budget_exhausted
DETAIL: solver budget spent
NO PROOF WAS JUDGED — the formalizer rejected the solver's answer:
The Northern Sámi conclusion overstates the evidence. Getman et al. (2025) did evaluate the earlier XLS-R system, fine-tuned Whisper, and new models on the same one-hour test set, reporting 47.70%, 43.15%, and 32.29% WER respectively. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) However, the paper does not document a fixed text-normalisation contract or say that SCTK produced those WERs; SCTK is mentioned specifically for matched-pair significance tests. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) Thus this is evidence that gains survive a common-test comparison, but not the fixed-normalisation rescoring experiment asked about. The solution itself acknowledges that a strict normalisation audit is unreproducible, so its categorical Northern Sámi verdict ('No') does not follow. The defensible verdict is that Northern Sámi shows substantial gains after controlling the test set, while the share attributable to scoring normalisation remains unquantified; Maltese likewise lacks the required common-set, fixed-normaliser rescore.
TOOL ACTIVITY: formalizer:web_search x5, solver:code_interpreter x1, solver:web_search x13
STATS: 3 calls, 114.1s

reasoning

{
 "summary": [
  "**Reviewing query parameters**\n\nI’m considering the date range for a 2025 paper, which might span from January 1 to December 31. The question doesn’t specify a date, so I’ll need to include orthographic variants like Sámi, Sami, Saami, and the Maltese language as whole words or stemmed forms. It seems I shouldn't cram synonyms into the same query. Instead, the developer advises covering broad-to-narrow queries across different rungs, possibly with plain alternatives."
 ]
}

reasoning

{
 "summary": [
  "**Exploring normalization terminology**\n\nI realize that Common Voice v25 isn’t directly relevant to the audit or leaderboards. I should look into terms related to normalization like text normalization, orthographic normalization, and scoring protocols such as jiwer, sclite, and SCTK. They relate to word error rates (WER) as well. The user’s question touches on large share and shrink improvement, so I need to focus on these specific concepts to address their inquiry accurately."
 ]
}

output

{
 "searches": [
  {
   "claim": "Has the proposal's question already been answered: does rescoring published Northern Sámi and Maltese ASR systems on one held-out test set under one fixed text-normalisation contract substantially shrink their reported WER improvements?",
   "queries": [
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (ASR OR \"speech recognition\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (ASR OR \"speech recognition\") AND (WER OR \"word error rate\")",
    "(\"word error rate\" AND (normalization OR normalisation)) AND (ASR OR \"speech recognition\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (WER OR \"word error rate\") AND (normalization OR normalisation)",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (rescore OR rescoring) AND (\"test set\" OR \"held-out test\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (\"fixed normalization\" OR \"fixed normalisation\" OR \"scoring protocol\") AND (WER OR \"word error rate\")",
    "(\"Northern Sami\" OR \"Northern Sámi\") AND Maltese AND (rescore OR rescoring OR \"text normalization\" OR \"text normalisation\")"
   ],
   "from_date": "",
   "to_date": "",
   "issns": []
  },
  {
   "claim": "For Northern Sámi, the share of reported ASR improvement attributable to scoring normalisation remains unquantified; no common-test, fixed-normaliser rescoring experiment is documented.",
   "queries": [
    "(\"Northern Sami\" OR \"Northern Sámi\" OR \"North Sami\") AND (ASR OR \"speech recognition\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR \"North Sami\") AND (WER OR \"word error rate\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR \"North Sami\") AND (WER OR \"word error rate\") AND (\"test set\" OR evaluation)",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR \"North Sami\") AND (normalization OR normalisation OR orthography) AND (ASR OR WER)",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR \"North Sami\") AND (rescore OR rescoring OR \"scoring protocol\")",
    "(\"Northern Sami\" OR \"Northern Sámi\") AND (Whisper OR \"XLS-R\" OR MMS OR Omnilingual) AND (normalization OR normalisation OR rescoring)",
    "(\"Northern Sami\" OR \"Northern Sámi\") AND (SCTK OR sclite OR jiwer) AND (WER OR \"word error rate\")"
   ],
   "from_date": "",
   "to_date": "",
   "issns": []
  },
  {
   "claim": "Getman et al. (2025) compares the earlier XLS-R system, fine-tuned Whisper, and newer Northern Sámi models on the same one-hour test set, but does not document a fixed text-normalisation contract or state that SCTK generated the reported WERs; SCTK is mentioned for matched-pair significance testing.",
   "queries": [
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (ASR OR \"speech recognition\")",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (NoDaLiDa OR \"Baltic-HLT\")",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (Whisper OR \"XLS-R\") AND (WER OR \"word error rate\")",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (\"one-hour test\" OR \"one hour test\" OR \"test set\")",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (normalization OR normalisation OR \"scoring protocol\")",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND SCTK",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND SCTK AND (\"matched-pair\" OR significance OR MAPSSWE)",
    "Getman AND (\"Northern Sami\" OR \"Northern Sámi\") AND (SCTK OR sclite) AND (WER OR \"word error rate\")"
   ],
   "from_date": "2025-01-01",
   "to_date": "2025-12-31",
   "issns": []
  },
  {
   "claim": "For Maltese ASR, no study provides the required comparison of published systems on one common held-out test set rescored with a fixed text normaliser.",
   "queries": [
    "Maltese AND (ASR OR \"speech recognition\")",
    "Maltese AND (WER OR \"word error rate\") AND (ASR OR \"speech recognition\")",
    "Maltese AND (\"test set\" OR evaluation) AND (WER OR \"word error rate\")",
    "Maltese AND (normalization OR normalisation OR orthography) AND (ASR OR WER)",
    "Maltese AND (rescore OR rescoring OR \"scoring protocol\") AND (ASR OR WER)",
    "Maltese AND (\"fixed normalization\" OR \"fixed normalisation\" OR \"text normalizer\" OR \"text normaliser\")",
    "Maltese AND MASRI AND (Whisper OR \"XLS-R\") AND (WER OR \"word error rate\")",
    "Maltese AND (Whisper OR \"XLS-R\") AND (rescore OR rescoring OR normalization OR normalisation)",
    "Maltese AND (SCTK OR sclite OR jiwer) AND (WER OR \"word error rate\")"
   ],
   "from_date": "",
   "to_date": "",
   "issns": []
  },
  {
   "claim": "A strict, reproducible fixed-normalisation audit of the Northern Sámi and Maltese WER claims is unavailable, including the normalisation specification and scoring artefacts needed to rerun it.",
   "queries": [
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND ASR AND reproducibility",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND ASR AND (\"source code\" OR repository OR scripts)",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (\"evaluation protocol\" OR \"scoring protocol\") AND WER",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (\"normalization rules\" OR \"normalisation rules\" OR \"text normalization\" OR \"text normalisation\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (scoring script OR scoring toolkit) AND (WER OR \"word error rate\")",
    "(\"Northern Sami\" OR \"Northern Sámi\" OR Maltese) AND (SCTK OR sclite OR jiwer) AND (code OR script OR repository)",
    "(\"Northern Sami\" OR \"Northern Sámi\") AND Maltese AND (reproducible OR reproducibility) AND (normalization OR normalisation OR rescoring)"
   ],
   "from_date": "",
   "to_date": "",
   "issns": []
  }
 ]
}

system

You are one role in an automated verified-reasoning pipeline. Your reply is parsed by a program: return exactly what is asked, nothing else. You have no shell or filesystem. Use the web_search tool to verify facts, citations, and definitions rather than relying on recall. Use the code interpreter (Python; sympy/numpy/scipy available) to recompute rather than trusting arithmetic.

You give the screening verdict on one research-proposal question, using the
digest of a verification run: the question (the proposal's hypothesis), the
outcome, the proof, and every audited claim with the checker's full reason.

`verdict` — is the proposal's question genuinely open and worth a human
reviewer's time?
- yes: the run establishes that the question is real, open, and well-posed —
  a certified "not settled by the reviewed evidence", or a checker-verified
  absence of the result the proposal would supply.
- no: the run shows it is not a fundable open question — already settled by
  the literature, ill-posed, or its supporting claims collapse on checkable
  facts.
- maybe: the run leaves specific uncertainties only a human can resolve. If
  openness rests on something the run did not check — whether the analysis
  is already published, whether the data exists — that is maybe, with the
  check as a review item, not yes.

The digest may end with a LITERATURE SEARCH section: engine-run OpenAlex
queries with their total match counts, per claim, broad to narrow; a rung
with at most 10 matches lists them (title, year, venue, doi). Read it as evidence, not as a verdict. Hits whose titles answer the
proposal's question support no (already settled). Zero hits on the narrow
rungs of a calibrated ladder (its broad rungs matched) support yes for that
claim's absence. An uncalibrated ladder establishes nothing, and a FAILED
rung is unknown, not zero. Name the query or hit you rely on.

`explanation`: for yes or no, 2-4 sentences grounded only in the digest.
For maybe, one sentence naming the core uncertainty.

`review_items`: for maybe only — 2 to 6 concrete questions or checks for the
human reviewer, each answerable and each tied to something in the digest.
Empty for yes and no.

user

QUESTION:
Is a large share of the reported word-error-rate improvement for Northern Sámi and Maltese speech recognition a scoring artefact — does rescoring the published systems under one fixed normalisation on one held-out test set shrink the improvement?

Non-exhaustive sources that may help:
- Mozilla Common Voice, Scripted Speech v25.0 (release 2026-03): 41,792 total / 28,377 validated hours, 290 languages, CC0 — https://commonvoice.mozilla.org/
- Getman Y. et al., Northern Sami ASR, Interspeech 2024 -- direct fine-tuning WER 36.1-42.7%, best extended fine-tuning 28.8% — https://www.isca-archive.org/interspeech_2024/getman24b_interspeech.pdf
- Getman Y. et al., NoDaLiDa/Baltic-HLT 2025 -- out-of-domain Northern Sami: Whisper 43.15% WER, best system 32.29% — https://aclanthology.org/2025.nodalida-1.19.pdf
- Williams A., DeMarco A., Borg C., SIGUL 2023 -- Maltese: zero-shot Whisper ~80% WER, fine-tuned XLS-R 2B 24.98% on ~50h (MASRI corpus ~40h24m) — https://www.isca-archive.org/sigul_2023/williams23_sigul.pdf
- Whisper large-v3 (Apache-2.0, 99 languages) and large-v3-turbo (MIT) — https://huggingface.co/openai/whisper-large-v3
- Meta Omnilingual ASR (10 Nov 2025) -- 1,600+ languages, code Apache-2.0, corpus (350 languages) CC-BY; supersedes MMS (CC-BY-NC-4.0) for coverage — https://ai.meta.com/blog/omnilingual-asr-advancing-automatic-speech-recognition/
- HuggingFace Open ASR Leaderboard -- multilingual track currently covers only five languages, none of them European minority languages — https://huggingface.co/spaces/hf-audio/open_asr_leaderboard
- ML-SUPERB 2.0 multilingual benchmark, Interspeech 2024, arXiv:2406.08641 — https://arxiv.org/abs/2406.08641
- CLARIN ERIC (ESFRI Landmark) and ALT-EDIC (created by Commission decision Feb 2024). NOTE: ELRC closed in Jan 2023 and ELRC-SHARE is shut down -- do not cite it as live infrastructure — https://www.clarin.eu/

OUTCOME: Declined budget_exhausted
DETAIL: solver budget spent
NO PROOF WAS JUDGED — the formalizer rejected the solver's answer:
The Northern Sámi conclusion overstates the evidence. Getman et al. (2025) did evaluate the earlier XLS-R system, fine-tuned Whisper, and new models on the same one-hour test set, reporting 47.70%, 43.15%, and 32.29% WER respectively. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) However, the paper does not document a fixed text-normalisation contract or say that SCTK produced those WERs; SCTK is mentioned specifically for matched-pair significance tests. ([aclanthology.org](https://aclanthology.org/2025.nodalida-1.19.pdf)) Thus this is evidence that gains survive a common-test comparison, but not the fixed-normalisation rescoring experiment asked about. The solution itself acknowledges that a strict normalisation audit is unreproducible, so its categorical Northern Sámi verdict ('No') does not follow. The defensible verdict is that Northern Sámi shows substantial gains after controlling the test set, while the share attributable to scoring normalisation remains unquantified; Maltese likewise lacks the required common-set, fixed-normaliser rescore.
TOOL ACTIVITY: formalizer:web_search x5, solver:code_interpreter x1, solver:web_search x13
LITERATURE SEARCH (engine-witnessed, openalex, as of 2026-09-02; total matches per query, queries broad -> narrow):
- CLAIM: Has the proposal's question already been answered: does rescoring published Northern Sámi and Maltese ASR systems on one held-out test set under one fixed text-normalisation contract substantially shrink their reported WER improvements?  [UNCALIBRATED: no query matched anything, so this says nothing about absence]
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (ASR OR "speech recognition")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (ASR OR "speech recognition") AND (WER OR "word error rate")
    FAILED hits: ("word error rate" AND (normalization OR normalisation)) AND (ASR OR "speech recognition")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (WER OR "word error rate") AND (normalization OR normalisation)
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (rescore OR rescoring) AND ("test set" OR "held-out test")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND ("fixed normalization" OR "fixed normalisation" OR "scoring protocol") AND (WER OR "word error rate")
    FAILED hits: ("Northern Sami" OR "Northern Sámi") AND Maltese AND (rescore OR rescoring OR "text normalization" OR "text normalisation")
- CLAIM: For Northern Sámi, the share of reported ASR improvement attributable to scoring normalisation remains unquantified; no common-test, fixed-normaliser rescoring experiment is documented.  [UNCALIBRATED: no query matched anything, so this says nothing about absence]
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR "North Sami") AND (ASR OR "speech recognition")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR "North Sami") AND (WER OR "word error rate")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR "North Sami") AND (WER OR "word error rate") AND ("test set" OR evaluation)
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR "North Sami") AND (normalization OR normalisation OR orthography) AND (ASR OR WER)
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR "North Sami") AND (rescore OR rescoring OR "scoring protocol")
    FAILED hits: ("Northern Sami" OR "Northern Sámi") AND (Whisper OR "XLS-R" OR MMS OR Omnilingual) AND (normalization OR normalisation OR rescoring)
    FAILED hits: ("Northern Sami" OR "Northern Sámi") AND (SCTK OR sclite OR jiwer) AND (WER OR "word error rate")
- CLAIM: Getman et al. (2025) compares the earlier XLS-R system, fine-tuned Whisper, and newer Northern Sámi models on the same one-hour test set, but does not document a fixed text-normalisation contract or state that SCTK generated the reported WERs; SCTK is mentioned for matched-pair significance testing.  [UNCALIBRATED: no query matched anything, so this says nothing about absence]
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND (ASR OR "speech recognition")
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND (NoDaLiDa OR "Baltic-HLT")
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND (Whisper OR "XLS-R") AND (WER OR "word error rate")
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND ("one-hour test" OR "one hour test" OR "test set")
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND (normalization OR normalisation OR "scoring protocol")
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND SCTK
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND SCTK AND ("matched-pair" OR significance OR MAPSSWE)
    FAILED hits: Getman AND ("Northern Sami" OR "Northern Sámi") AND (SCTK OR sclite) AND (WER OR "word error rate")
- CLAIM: For Maltese ASR, no study provides the required comparison of published systems on one common held-out test set rescored with a fixed text normaliser.  [UNCALIBRATED: no query matched anything, so this says nothing about absence]
    FAILED hits: Maltese AND (ASR OR "speech recognition")
    FAILED hits: Maltese AND (WER OR "word error rate") AND (ASR OR "speech recognition")
    FAILED hits: Maltese AND ("test set" OR evaluation) AND (WER OR "word error rate")
    FAILED hits: Maltese AND (normalization OR normalisation OR orthography) AND (ASR OR WER)
    FAILED hits: Maltese AND (rescore OR rescoring OR "scoring protocol") AND (ASR OR WER)
    FAILED hits: Maltese AND ("fixed normalization" OR "fixed normalisation" OR "text normalizer" OR "text normaliser")
    FAILED hits: Maltese AND MASRI AND (Whisper OR "XLS-R") AND (WER OR "word error rate")
    FAILED hits: Maltese AND (Whisper OR "XLS-R") AND (rescore OR rescoring OR normalization OR normalisation)
    FAILED hits: Maltese AND (SCTK OR sclite OR jiwer) AND (WER OR "word error rate")
- CLAIM: A strict, reproducible fixed-normalisation audit of the Northern Sámi and Maltese WER claims is unavailable, including the normalisation specification and scoring artefacts needed to rerun it.  [UNCALIBRATED: no query matched anything, so this says nothing about absence]
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND ASR AND reproducibility
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND ASR AND ("source code" OR repository OR scripts)
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND ("evaluation protocol" OR "scoring protocol") AND WER
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND ("normalization rules" OR "normalisation rules" OR "text normalization" OR "text normalisation")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (scoring script OR scoring toolkit) AND (WER OR "word error rate")
    FAILED hits: ("Northern Sami" OR "Northern Sámi" OR Maltese) AND (SCTK OR sclite OR jiwer) AND (code OR script OR repository)
    FAILED hits: ("Northern Sami" OR "Northern Sámi") AND Maltese AND (reproducible OR reproducibility) AND (normalization OR normalisation OR rescoring)
STATS: 3 calls, 114.1s

reasoning

{
 "summary": [
  "**Evaluating output schema**\n\nI need to focus on producing the exact JSON schema for the output. The verdict seems to hinge on potential failures in uncalibrated searches without proof, which leaves some core uncertainties about what's published or possible. I'll aim to provide an explanation in just one sentence for the uncertainty. I should also review a concrete range of 2-6 examples to ensure I have a grounded digest. This will help clarify everything."
 ]
}

output

{
 "verdict": "maybe",
 "explanation": "The digest shows that Northern Sámi gains survive a common-test comparison, but neither language has a verified fixed-normalisation rescore, and the failed, uncalibrated literature searches cannot establish that the proposed analysis is unpublished or feasible.",
 "review_items": [
  "Check manually whether any Northern Sámi or Maltese study has already rescored competing published systems on a common held-out set under an explicitly fixed text-normalisation contract.",
  "Verify that outputs or runnable checkpoints for the relevant published systems are available for rescoring.",
  "Verify access to a legally usable common held-out test set with identical references and audio for each language.",
  "Determine whether the papers or repositories provide sufficient decoding settings, scoring scripts, and normalisation rules to reproduce each baseline.",
  "Pre-specify what fraction or absolute reduction in the reported WER improvement would count as a “large share” or a substantial shrinkage."
 ]
}