Edit columns
Click a column header to sort.95% confidence interval via bootstrap resampling (5,000 resamples per model). Follows the filters above.
How much did fine-tuning help?
Each fine-tune shown next to the model it started from, on the Slovak Common Voice test set (– utterances).
Some of these models may have been trained on Common Voice, so this is not a perfectly fair comparison, but the fairest we can make for now. Limitations
To compare fine-tunes with each other: the full benchmark in Leaderboard → Fine-tuned, or by domain in Compare → Fine-tuned (e.g. parliament or customer service).
Which model fits your setup?
Set your limits; the page picks the most accurate model that fits.
Only the – open-weight base models are included: we ran them on one shared GPU, so their speed (RTFx) and size are measured.
The – closed APIs run on their providers’ servers, so they cannot be timed the same way. About the metrics
Base models with measured speed and model size are shown. Closed APIs without those measurements are excluded. Values are listed in the Leaderboard table.
Ranked separately from base models, it was trained on data from sources similar to the KInIT Slovak benchmark test set used for evaluation. Limitations
Compare models across domains
Pick 2–6 models. The leaderboard ranks overall scores; this view shows where each model wins or struggles.
Pick at least two models to compare.
How this benchmark works
Every model transcribes the same Slovak test set. Outputs are normalized the same way, then scored against human reference transcripts.
Open-weight models are also timed on the same machine, so speed is comparable too.
Metrics
Accuracy
lower is better- WER · CER
- Word and character error rate: edit distance between the model output and the reference, divided by the reference length. Computed over the whole test set, not averaged per sample.
- WER 95% CI
- Bootstrap confidence interval, with utterances resampled with replacement. Overlapping intervals mean two models are not clearly different.
- Insertions · Deletions · Substitutions
- The three parts of WER, each divided by the number of reference words. They add up to WER.
Numbers
lower is better- Numeric CER
- Character error rate on numbers: digit-level edit distance over all digits, divided by the number of reference digits.
- Numeric WER
- Word error rate on numbers: the same over whole digit sequences such as
2024or0903, divided by the number of reference sequences. - Numeric samples · Hyp/Ref ratio
- How many utterances contain digits in the reference and in the model output. A ratio above 1 means the model writes digits where the reference has none; below 1, it misses them.
Speed
higher is better- RTFx
- Inverse real-time factor: seconds of audio transcribed per second of processing, over the full test set. 700× means one hour of audio in about five seconds.
Normalization
Before scoring, references and model outputs go through the same normalization pipeline, so a model is not penalised for writing “25” instead of “dvadsať päť”.
The pipeline
Seven steps, in this order, applied to both the human reference and the model output.
- 1Unicode NFKC
- 2Lowercase
- 3Remove bracketed text
- 4Currency words → symbols
- 5Number words → digits
- 6Dates → compact digit form
- 7Strip punctuation at word edges
Example
One model output before and after the pipeline: the spoken number becomes digits, “eur” becomes “€”, and the bracketed tag and punctuation are removed.
Illustrative example, not taken from the test set.
Dataset
The KInIT Slovak benchmark test set is built from the test splits of public and in-house Slovak datasets, from read speech to council meetings and phone calls.
Only aggregate results are published here, never audio or transcripts.
Sources
Results can be split by
or compare models side by side in Compare.
KInIT fine-tuned models
KInIT publishes Slovak fine-tunes of eight open models. They were trained on the train splits of the same datasets this benchmark tests on, so ranking them next to base models would not be fair. They are ranked on their own in Leaderboard → Fine-tuned, together with third-party Slovak fine-tunes such as NaiveNeuron’s Whisper models, whose training data we do not know in detail. Each fine-tune is also compared with its base model in Fine-tuned vs. base, on the Slovak Common Voice test set. Why this is a limitation, and how we plan to fix it: Limitations.
| KInIT model | Fine-tuned from | Size |
|---|
How they were trained
- Full fine-tuning on a curated Slovak corpus: public datasets plus KInIT recordings, with personal data anonymised.
- Background noise, phone noise and packet loss added to 75% of training batches, for real-world recordings.
- Trained on the Devana HPC cluster.
Slovak only
Fine-tuning on Slovak makes the models much worse at other languages.
For multilingual audio, use the original base model.
We are working on this; see Limitations.
Limitations
What these results can and cannot tell you yet. We are working on all three.
Base and fine-tuned models cannot be compared fairly yet
LimitationThe test set draws on public Slovak datasets such as Common Voice and FLEURS, plus parliament speech and other recordings. KInIT’s fine-tunes were trained on the train splits of these same datasets: train and test splits do not share utterances, but they do share sources, domains, speaking styles and recording conditions. Third-party fine-tunes were also trained on Slovak data, but we only know what their model cards say. NaiveNeuron’s Whisper models, for example, list a Slovak parliament corpus, which may overlap with the benchmark’s parliament speech source. How much their training data overlaps with the sources we test on, we cannot check.
A fine-tune has therefore seen data similar to the test set, and the base models may not have. Its score here is likely better than it would be on new audio, so ranking it next to base models would favour it unfairly. That is why fine-tuned models are never ranked together with base models, and why they are measured on two different test sets:
Once it is ready, base and fine-tuned models can be ranked together, on data that none of them has seen.
Slovak is set, not detected
LimitationThis is a Slovak leaderboard, so every model is tested on Slovak speech only. Whenever a model or API lets us choose the transcription language, we set it to Slovak and do not use automatic language detection. Models without such an option run with their default settings.
The scores therefore show how a model does when it is told the audio is Slovak. With automatic detection, results may differ, for example when a model mistakes Slovak for a related language.
KInIT fine-tunes forget other languages
LimitationKInIT’s fine-tunes were trained on Slovak and have largely lost the other languages of their base models. This is known as catastrophic forgetting. This leaderboard scores Slovak only, so the loss does not show in these results.
For audio in other languages, or with several languages mixed, use the original base model.