Teaching Whisper to speak Australian at Everything Open 2026
- Published
- Filed under
- EverythingOpen, voice and speech technology, opensource, data science
- Reading time
- 2 min
Today I ran a tutorial at Everything Open 2026 at the University of Canberra called Make Whisper speak Australian with Mozilla Data Collective speech datasets.
Update, July 2026: the video is now online, thanks to Linux Australia, so if you missed it or want to work through it at your own pace, here it is.
Why Whisper needs to learn Australian
Many people in the Everything Open community run an open voice assistant of some kind, and these days that’s most likely Home Assistant, perhaps with the Voice Preview Edition hardware. If you run Home Assistant fully locally for privacy, then under the hood your speech is being transcribed by faster-whisper, a fast, efficient implementation of OpenAI’s Whisper speech recognition model.
But Whisper doesn’t always get it right, especially if you speak Australian, or with any accent that isn’t well represented in the data it was trained on. That’s not a new problem. My own research, through a Research Partnership with Mozilla’s Common Voice team, found that Whisper was much less accurate for many English accents. When your voice assistant doesn’t understand you, it isn’t much of an assistant.
So how might we make faster-whisper work better for voices like ours?
Fine-tuning, or teaching Whisper a new accent
The answer is fine-tuning. In machine learning, fine-tuning means taking a model that has already been trained, like Whisper, and adjusting its internal weights and biases (its internal mathematical representation of speech) using data that is closer to the task you actually want the model to do.
In plain terms, it means teaching Whisper to speak Australian. Bonza. Or, if you’re from Radelaide, heaps good.
Where the data comes from
Fine-tuning needs data, and where that data comes from matters.
The data for this tutorial came from Mozilla Common Voice, which is now published through Mozilla Data Collective. Common Voice is one of the largest open English speech datasets in the world, and it was built by and for its community, transparently and ethically: every clip was recorded by a volunteer who chose to contribute it. Mozilla Data Collective lets communities create data (for speech, through Common Voice), curate it, and control who can access it and for what purposes.
Since 2022, Common Voice contributors have been able to say which accent they speak with. About 700 speakers in the dataset have said they speak with an Australian accent, and that’s what makes it possible to fine-tune Whisper for Australian voices.
(Full disclosure: I work at Mozilla Data Collective, as Head of Applied AI & ML R&D.)
What the tutorial covers
Over the 100-minute session, we:
- downloaded the Common Voice English en-AU subset from Mozilla Data Collective;
- set up a notebook in Google Colab, building on Mozilla.ai’s blueprint for fine-tuning an ASR model with Common Voice data;
- prepared the audio and transcripts for training;
- fine-tuned Whisper on a GPU with the Australian English data;
- tested the fine-tuned model on our own voices; and
- converted the model to faster-whisper format, ready to use for local speech recognition in Home Assistant.
Thank you
Thank you to everyone who came along and worked through it with me, to the Everything Open 2026 organisers and volunteers, and to Linux Australia for recording and publishing the session. The video is licensed CC BY-NC-SA 4.0.
More about the tutorial, including links to the data and the blueprint, is in my portfolio.