Less has been written, however, about the market niche such a device serves.

Surely we don’t need another AI hardware device to speak into, given the failures of the Rabbit R1 and the Humane AI Pin? The market opportunity for such a device also seems limited given that we have chatbots on our phones - particularly with now-ubiquitous mobile internet coverage.

But if we examine the intended capabilities of the speculative device, its function and value become clearer. They’re not for the consumer, they’re for OpenAI.

Let me unpack.

The Verge reported that during the call in May, Altman

… dropped some hints during the call that shape our expectations, however, including that it will be unobtrusive, fully aware of a user’s life and surroundings, and will serve as a “third core device” a person would put on a desk after a MacBook Pro and an iPhone.

Fully aware of life and surroundings hints at an embodied view of intelligence

“Fully aware of a user’s life and surroundings”, when translated to technical functionality, means having sensors and recording capability to store things like geo-spatial data - where a person was, when, and what they were doing at that location. It means listening to conversations the person was having, who they were having them with, and what they talked about. And it might mean seeing what activities a person was doing, and what objects they were interacting with, and for what purpose.

This data could be used, hypothetically, to provide context for generative AI models, that are well-known for hallucinating - generating outputs that are provably false and misleading. A key way to address hallucination in LLMs is through retrieval augmented generation (RAG) - essentially a knowledge graph that an LLM checks its output against before sending it to the user. Thinking of it as a “think before you speak” brake for a person who lacks a filter between their brain and their mouth.

LLMs are probabilistic (the “stochastic” in Emily M. Bender et al’s paper “On the Dangers of Stochastic Parrots”¹). This means they work on probabilities - a “good” model will get the right answer or deliver a factually correct response most of the time. RAG aims to make LLMs more deterministic (rules-based) by “grounding” them in a map of reality.

Language theory (symbolic view) versus grounding theory (embodied view) of intelligence

Here, we reach a theoretical divide.

Some researchers believe that reality can be modelled purely through symbols. I’m over-simplifying here, but essentially this argument holds that our reality can be entirely represented through language.To better model reality, we need more language to model on. The opposing view is the grounded theory - that our reality is embodied and experienced, and requires sensori-motor input. Language isn’t enough to model our world - we need to know how the world looks, feels, hears and smells. Recent research seems to suggest that having multiple modalities provides better “conceptual alignment” between AI model and human representations of the world² - that is, the embodied view is gaining traction.

But how might we model those modalities?

Peak Token, the Token Wars, and the need for more data for an embodied representation of the world

In a recent talk given at both Everything Open and the Melbourne Machine Learning and AI Meetup, I introduced the concept of the “Token Wars”. Essentially, I argued, tokens - the chopped up, vectorised data used to train AI models - are not a public resource. They are both excludeable and rivalrous - and we’re running out of new, human-generated, real tokens of language data because of wholesale web scraping by bots. New LLMs are trained on nearly all the publicly available data in the world.

We’ve reached Peak Token.

So, if we’re leaning toward the embodied view of intelligence, and we’ve reached Peak Token with language data, this means that industry needs more multi-modal data to model the embodied view of intelligence.

A device, for example, that can sense where someone is, what they’re saying, the sounds around them and the objects they’re interacting with.

The OpenAI device’s primary purpose is for gathering multi-modal data for embodied AI models.

Any benefits it provides to the consumer in the form of better LLM responses are at best, a secondary feature, and at worst, incidental.

Will you pay to have the device gather more data about you?

Image: generated by ChatGPT using the prompt:
Generate an image of the speculative hardware device proposed by Sam Altman and Jony Ive for Open AI. It should contain the OpenAI logo and be screenless, in the form of something that could be clipped on to someone’s shirt.

Using examples from fieldwork in multiple geographies, Madianou first provides a brief history lesson on the logics of colonialism, demonstrating its far-reaching echoes into our societal and institutional configurations of today. She shows how humanitarianism joins capitalism through the desire to corporatise humanitarian organisations, leading to the need for accountability and audit.

Next, she shows how biometric infrastructures perform both data extraction and oppression, drawing on the work of Bowker and Star to illustrate how “categories have consequences” in the delivery of humanitarian aid - and how those categories are both created, and unable to be effectively contested.

Madianou then critiques accountability of humanitarian efforts, showing how feedback mechanisms are often performative and a poor way to evaluate the effectiveness of programs - mirroring in many ways broader themes within the project management literature of how to measure outcomes.

She proffers the term “surreptitious experimentation” to point to humanitarian aid sites as laboratories for testing new technologies on an audience unable to provide informed consent - and where the harms of experimentation are borne by those, paradoxically, that technology is intended to aid.

Painting digital humanitarianism as an inexplicable “black box” - a machine - she lays out an argument that shows how multiple stakeholders are imbricated in digital humanitarianism and its harms, however, in my view, stops short of identifying specific leverage points (in the parlance of Donella Meadows) for changing or re-configuring the machine.

Following, Madianou coins the term “mundane resistance” to point to actions of those who refuse to engage with the digital humanitarianism machine as it intends them to - by usurping feedback mechanisms to request songs from a radio station - for example. Here, I found the term “mundane” jarring - resistance in situations where one has little power are not acts of mundanity, but bravery, confidence, rebellion.

This book sits within infrastructure studies because technocolonialism is deeply infrastructural - technologies, institutions, funding mechanism, and the flow of data - combine to create a technocolonialist infrastructure in which digital humanitarianism operates.

The key weakness, in my view, in this book, is the lack of attention paid to how to undo the logics of technocolonialism; while Madianou points to concepts such as design justice and human-centred design, these by themselves are too vague. What is the prescription for de-technocolonialism? I see perhaps a confluence of Dan McQuillan’s desire for “decomputing” coupled with the de-colonial efforts of academics such as Timnit Gebru, Deb Raji and Joy Buolamwini. But where are the leverage points in the technocolonialist system? Madianou correctly points to funders - who themselves are not neutral actors. What I would have liked here instead is a blueprint for unwinding the six logics set out at the start - humanitarian accountability, audit, capitalism, technosolutionism, securitization and resistance - which would be another volume on its own - and one I hope Madianou is able to find the time and space to pen.

I’d recommend this book for scholars of science and technology studies (STS), and qualitative researchers who are looking for a way to structure arguments that intersect multiple concepts - as Madianou does this exceptionally.

What’s changed since the last report?

More work in Overleaf - an online editor for LaTeX

I’m still finishing my PhD at the Australian National University’s School of Cybernetics, and my primary tasks are writing up my thesis - which I’m doing in Overleaf in LaTeX. ANU now has a subscription to Overleaf, which makes logging on with single sign on (SSO) easier. Overleaf grew from two open source projects called ShareLaTeX and writeLaTeX, which both aimed to provide a collaborative environment for co-authoring papers in LaTeX. In 2017 they merged to become Overleaf.

I’ve been impressed with how the interface to Overleaf has been incrementally developed over the last year - I can backup my thesis to GitHub at the click of a button, and the ability to tag projects for easy retrieval has been great. A key feature I’m still missing in Overleaf is the ability to organise projects (papers) into folders, and the ability to share whole folders with collaborators. For example, I want to be able to put all my thesis writing in a folder and share that with my doctoral committee.

However, although Overleaf is billed as a social enterprise, it is wholly owned by Digital Science, an academic publishing services company, which in turn is a subsidiary of Holtzbrinck, a German publishing company - which also owns 53% of Springer Nature. I’ve noticed that there are several “experiments” being run (I’m in the Beta / Labs program), all of which encourage the user to “upgrade” to premium versions of plugins. This week Overleaf was pushing Writefull, a Grammarly-esque tool to help edit academic work - which it acquired in 2022. It was prominently displayed in the interface, and works well - except that you only get about 10 corrections before being prompted to upgrade to the Premium version, which is $USD 7.33 per month, with a current special being offered, bringing it down to $USD 4.40 per month. I figured out how to disable it, but it wasn’t intuitive, which I am sure is by design.

The play here, I speculate, is to get grad students hooked on Writefull, who in turn will pressure their institutions to buy the upgrade at an institutional level. I mean, kudos, it’s a smart GTM strategy - but it’s yet another instance of universities being beholden to publishing or software companies for increases in productivity and research outputs. Imagine instead if a group like AARNet or Universities Australia had invested self-hosting Overleaf (which is possible) - but then, that still costs money, and OpEx at that, which is always harder to justify than CapEx because of the prevailing view, in my experience, that software should look after itself.

The second-order piece here is that Overleaf / Digital Science / Holtzbrinck is collecting user experience data on how researchers use the Overleaf platform - rather than that being something that universities themselves can use to gain insights that feed into research development activities. From what I can tell just from the requests that Overleaf is sending, it’s using Google Analytics, Stripe and Growthbook (an open source feature flag and A/B testing suite, which collects UX research data) and BugSnag, which to be fair, is not that many and feels very boot-strappy (e.g. there’s no Adobe type CX cruft). Another GTM play I see here for Overleaf is then selling these “insights” back to universities (or not - and using them to create more premium plugins to upsell).

Even with these downsides, Overleaf is a game-changer for collaborative LaTeX document editing, so I suspect I’ll be using it for the foreseeable future.

Dark Visitors - protecting my blogs from web scrapers

I’ll be giving a talk at Everything Open in January on the topic of “Token Wars” - essentially an overview of tokens, transformer architecture and the broader efforts by various actors to either harvest tokens from the public web, or to protect their data from being scraped.

Dark Visitors is a tool that fits into the latter category - and I discovered it when it was featured on The Sizzle newsletter, then featured in a Mozilla Data Futures Lab talk. Its key features are identifying, then blocking (if you have the premium version) scrapers and harvesters.

It’s been a revelation looking at the visuals from Dark Visitors to realise just how much non-human agent traffic there is to my blog. For example, here’s an overview of November - and you can see that one scraping agent - in this case a Scrapy instance - hit my blog over 1200 times in one day. Keep in mind this is a tiny, cpanel-hosted blog - it’s not CDN’d, and it’s not intended to serve high volumes of traffic. Scrapers and bots are now capable of basically DDOSing entire sites when they try to harvest data.

The other key insights Dark Visitors has given me is just which agents are disrespecting robots.txt, and therefore deserved to be banned. Dark Visitors has a WordPress plugin which makes this easier - essentially if a user agent makes a request, and it’s blocked, then it gets served a 403 Forbidden http response (instead of 200 OK and the content). For example, AppleBot is a key culprit:

I do know of folks who are less forgiving and instead use nginx to serve a 10Tb binary 😈

A work-focused Android e-paper tablet: The Onyx Boox Tab Pro X

The key hardware change this year has been using adopting an Onyx Boox Tab Pro X Android e-paper tablet. I wanted a tablet because I’m still reading a lot of papers, which quickly gets tiring on an LCD screen, and I’ve also had to elevate my dodgy leg this year a lot more because $REASONS, and a tablet is much more comfortable .

I settled on the Onyx Boox Tab Pro for several reasons:

• It uses stock Android (AOSP), which means that you can use the Google Play Store to install software
• It is black and white - I wanted this, so that colour was less distracting
• The A4 size was a key feature for me for reading primarily A4 or Letter sized documents
• The handwriting recognition notes feature was also attractive
• Bluetooth means that you can connect a keyboard and mouse and use it to type documents

So far, I’ve been super happy with the Boox Tab Pro X - it’s fantastic for note-taking meetings, and for marking up PDFs. The key downside I see is that the markup of PDFs is not converted to text from handwriting - so the saved image size is a lot larger. I have been taking this to meetings where previously I would take my laptop - and I find that handwriting on the tablet has a different cognitive “feel” - my brain works in a different way when using a table - a phenomenon that is also backed up by recent research.

Still trying to find an alternative to the paper-based Passion Planner

In last year’s State of My Toolchain, I expressed my frustration with the cost of the Passion Planner, and my efforts to find an alternative, settling on the Leuchtturm1917 A4 planner.

This has been a “just OK” choice. The paper is thinner, the hardback binding makes it less flexible, and the layout is not as good as the Passion Planner. But I still refuse to pay the exorbitant postage Passion Planner wants.

So, for 2025, I bought the digital edition, which is a set of PDFs. They are promoted as working well with most PDF readers, but tested best with GoodNotes. I installed GoodNotes on the Onyx Boox, intending that the tablet would become my new planner - and it wouldn’t even load! So, fail on that front. I am still trying to get the Passion Planner PDF to render well in NeoReader - the native PDF reader for Onyx Boox, and you’ll have to stay tuned for how I go.

Making friends with Claude, but still enemies with generative AI more broadly

While I am still very sceptical of generative AI, and aghast at the amount of natural resources such as water and electricity it consumes - to the point where people want to re-commission nuclear power stations - I am also aware of their productivity benefit.

I’ve settled on Claude from Anthropic - mostly because I think the values on which Anthropic are founded are vastly different to OpenAI, if still problematic (the whole existential risk, effective altruism, open philanthropy nexus is problematic, although I find the TESCREAL naming unhelpful in trying to tackle it).

I have several detailed prompts specific - such as the philosophy professor who helps me to refine methodological arguments, and the the positive psychology coach who’s helping me maintain motivation and discipline so I can finish this ^*&&!! PhD.

What’s currently in my toolchain?

Hardware, wearables and accessories

• My main laptop is an ASUS ROG Zephyrus G15 (no change since last report)
• Google Pixel 4a 5G running stock Android (no change since last report)
• Mobvoi Ticwatch Pro 2020 model (no change since last report)
• Fellowes book stand - again, very light and easily portable
• Logitech C920 web cam (no change since last report)
• Stadium USB microphone (no change since last report)
• Sennheiser Momentum 3 wireless bluetooth headphones (no change since last report)

Software

• Overleaf (no change since last report)
• VS Code with a range of plugins for writing code, thesis notes (no change since last report)
• Pandoc for document generation from MarkDown (no change since last report)
• Zotero for referencing (using Better BibTeX extension) (no change since last report)
• OneNote for Linux by @patrikx3 (no change since last report)
• Nightly edition of Firefox (no change since last report)
• Zoom (no change since last report)
• Microsoft Teams for Linux (no change since last report, but boy is it getting rubbish!)
• Gogh for Linux terminal preferences (no change since last report)
• Super productivity (instead of Task Warrior) (no change since last report)
• Cuckoo Timer for Pomodoro sessions (no change since last report)
• RescueTime for time tracking (no change since last report) - I should note that the new Timesheets functionality is super helpful.
• BeeMindr for commitment based goals (no change since last report)
• Okular as my preferred PDF reader (instead of Evince on Linux and Adobe Acrobat on Windows) (no change since last report)
• NocoDB for visual database work (no change since last report)
• ObservableHQ for data visualistion (no change since last report)

Techniques

• Pomodoro (no change since last report)
• Time blocking (used on and off)

Current gaps in my toolchain - what’s missing?

Daily coach and mentor

I’m finding some of Claude’s capabilities useful from a mentoring perspective, and I think I would genuinely pay for a daily coach and mentoring service - something more sophisticated that a task or action tracker, or emotion tracker, but something which has a long-term context and is able to help guide and motivate me toward my long-term objectives. I wonder if this is where services like ChatGPT or Claude are heading - essentially replacing middle management and team leaders.

Impact tracker

How do I know that the actions I am taking are having an impact? RescueTime tracks where I spend my time, and helps me see if I’ve gone down a Sapiens rabbit hole or binged a bit too much Expanse or For All Mankind (but they’re soooo good!). But how do I link my work and tasks to broader objectives and progress toward those objectives? Super Productivity has some of this capability, but is dependent on your decomposing broader goals into much finer-grained ones. And how might I link the work I’ve done to increase my physical activity and capability with longer-term fitness goals? There’s a space here amongst the Get Shit Done, One Minute Manager ecosystem for something that is better aligned to tracking impact.

No, I’m not accepting paid sponsorships or links

My State of the Toolchain posts get a bit of traction, and with that, I’ve started attracting people wanting me to link to their product or website, for a fee. That is, sponsored content - but of course they don’t want it to look like sponsored content. I usually respond harshly, dismissively and swearily to these requests.

If you want me to write about your product or service, build something awesome.

This has been a particularly contentious debate, centred primarily on whether the data used to train a model needs to be openly available for the resulting model to be considered open source.

On one side of the debate is a very pragmatic, practical perspective - the data used to create a model is often private, proprietary to an organisation, and the pre-processing steps difficult to reproduce. On the other is an idealist viewpoint - that for a model to claim to be open source, not only the code and model architecture need to be open, so to does the dataset on which it was trained.

The OSI has adopted a position halfway between the two sides:

“The new definition requires Open Source models to provide enough information about their training data so that a ‘skilled person can recreate a substantially equivalent system using the same or similar data”.

While my personal position is that open source AI models should be constituted from open data to earn the moniker - at the risk of open washing - there is a broader issue that is yet to enter the discourse.

In order to provide information about training data, we require dataset documentation.

Dataset documentation is information about a dataset that describes how that dataset was constituted, what it is constituted from and where it is intended to be used. One way of thinking about dataset documentation is like a nutrition label on a can - it tells you the ingredients, properties, and what percentage of its contents are fat, sugar, sodium and so on. That is, it helps you better understand what’s in the can.

Dataset documentation has gained increasing research attention in recent years - such as the work Timnit Gebru, Kate Crawford, Ben Hutchinson and others did on Datasheets for Datasets, and the work Emily M. Bender and Batya Friedman did on Data Statements for Natural Language Processing. My own research has focused on dataset documentation for speech data.

In the research space, academic organisations like the Association for Computational Linguistics now require dataset documentation to be provided with any dataset that is published with a research paper - a strong step forward. However, platforms that host such data, such as Hugging Face, while providing the structures to create dataset documentation, do not mandate it.

For example, HF has uploaded Mozilla’s Common Voice dataset to the platform - a dataset with which I am intimately familiar - however, it lacks any meaningful dataset documentation, making it harder for practitioners and researchers to make informed decisions about how speech models trained on this data will perform across axes such as genders, ages and accents.

That is, dataset documentation is the missing piece of the puzzle in the Open Source AI definition. Without effective dataset documentation practices, proformas, and platforms that support these, the Open Source AI definition risks paying lip service to reproducibility.

To move forward, I would like to see the Open Source Initiative become involved in efforts that help to standardise and improve practices and proformas around dataset documentation. Taking the position they have - that data does not need to be openly available - but must be adequately documented - commits the Open Source Initiative to advocating for better dataset documentation for AI training data.

Let’s work together to make sure that dataset documentation becomes just as important as data, model architectures or code for Open Source AI.

• Read more here about the Open Source AI definition via the OSI
• Read more about how the debate is not yet over via Steven Vaughan-Nichols for ZDNET

The context

Recently, I encouraged a family member to switch ISP from Internode to Aussie Broadband. There were a couple of drivers for the change (hah! pun!); my family member’s primary email was with Internode, but they recently divested of this service and it would soon cost extra. We’d also had some poor experiences with Internode, including line dropouts - which I had to escalate to a Service Manager to resolve. We also had a lot of trouble understanding the accent of the people on the Internode Service Desk. Conversely, I’d had an excellent experience with Aussie, who helped me connect my NBN service when it finally became available in my area. I spoke with a person who was physically in Morwell, super helpful, and who made everything easy. So, my family member switched to Aussie at my beckoning.

The churn process went smoothly. I popped around to my family member’s house, and re-configured the Internode-supplied modem to point to the VDSL service from Aussie. We got line sync, and voila, internet and wireless were working.

The problem became apparent went I went to re-configure the VoIP settings. They simply weren’t on the modem’s administration interface - anywhere.

The confusion

I’m not bragging, but I’m pretty good with most technical things. I run Linux. By choice. I am comfortable running Docker. On the command line. I can make scarfs that sense light and twinkle if it’s too dark, using Arduino. I run PiHole on my home network. I figured out dask. I thought this would just be a case of finding the right settings on the modem’s administrative interface.

I was, sadly, wrong.

Can I find a vanilla firmware?

But I could not find the Telephony / VoIP settings on the router - a TP-LINK Archver VR1600v. The modem had previously been supplied by Internode, and had some settings, like VoIP, pre-configured.

The first thing I tried was to go to the product’s home page on the TP-LINK website, to see if there was a generic firmware available to download. There wasn’t.

I considered trying to ssh into the modem and see if I could wget the relevant pages - but figured that was too much hassle.

The more I dug, the more I realised that the firmware had been deliberately hobbled by Internode - and that finding an alternative firmware was going to be an exercise in futility. There appeared to be several complaints about this issue on the Internode forums (who were bought out by TPG) (second forums link).

Can the manufacturer give me vanilla firmware?

I even lodged a support case with TP-LINK, on the off chance they could send me a generic firmware. “I can compile it from source, I’m technical!”, I pleaded. My attempt at shibboleet met with resounding defeat.

As you know, the model Archer VR1600v is a customized one for service providers. For such customized models, the firmware update is controlled by service providers and we do not have access to it. That’s why we are not able to find its firmware on our official website. 

And we suggest to contact TPG and ask them to unbind/revert the firmware to the vanilla version.

Since your family member changed the ISP, we recommend you consider a common model instead. Below are our regular DSL modem routers, models with the lower case “v” indicates they support VoIP.

I mean, points for trying to upsell me a new modem, but really the manufacturer is pushing me back to the service provider - Internode / TPG - to provide the vanilla firmware - which they have absolutely no incentive to do, because my family member switched ISPs.

UPDATE: Subsequent to posting this, @troybie contacted me to let me know of an NZ-based firmware floating around on Whirlpool (thx, @troybie!). To be honest, I don’t trust random firmwares that much, so I’m a bit hesitant to give it a try …

Can I roll my own firmware?

At this point, if you’re a networking or Linux flavoured techie, you’re probably asking yourself “why didn’t she flash it with OpenWRT”? To which I would respond “good question, and yeah I thought of that already”. There’s no OpenWRT firmware for this model, sadly. And while it was an attractive option for about 5 minutes, firmware isn’t my wheelhouse, and I don’t have time to upskill in rage-based firmware creation 🤬

UPDATE: Subsequent to posting this, @voltagex has let me know that trying to compile from TP-LINK’s GPL sources is a bust too, because they don’t work. Sorry for the nerd snipe there!

I was defeated.

The consternation

The issue here isn’t that I wasn’t able to flash a consumer-grade router with new firmware, so my family member could use the VoIP service they’re paying for. It’s a microcosm of a much broader problem with the software, hardware and service assemblages that form our socio-technical systems: their complexity is being abstracted away from us in ways that remove our ability to shape, re-purpose, and re-use them.

If you’re a programmer, you might think about abstraction in terms of data abstractions, where variables are specified as typed; integer, character, Boolean, based on their characteristics¹. If you are more philosophically inclined, you might think of abstraction as a process that identifies and generalises commonalities across concepts, or objects, or phenomena². That is, abstraction helps us deal with complexity³.

Abstraction can be useful. Abstraction means that if you can drive one car, you have a reasonable chance of being able to drive many others - because the user interface is similar - it shares many characteristics. Abstraction means that if you read one Annual Report, you have a fair chance of understanding other Annual Reports. Abstraction is what allows us to compare apples and oranges, and know that they are both fruit.

We use abstraction a lot in technology. Operating systems are abstractions - most folks push icons around a screen on a graphical user interface to copy, paste or rename a file. That’s an abstraction of lower-level commands, which we can still do on the command line, if we roll that way (yes, some of us roll that way). User guides are often abstractions - helping to walk people through the complexity of an application.

An ISP flashing a modem router with customised firmware with VoIP telephony settings preset, and unable to be modified is an abstraction. And it has several benefits - it makes the modem easier for the consumer to connect to the ISP’s network, and it reduces the support load on the ISP because they don’t have to walk the consumer through how to configure the modem router.

But when we abstract something, we trade complexity and nuance and difficulty for ease of use, simplicity - and reduced functionality. The cp command in Linux - to copy a file - has 32 options it can take. When I use the graphical user interface in my operating system to copy a file, I right click and select “Copy”. Abstracting firmware and hobbling its functionality yields simplicity at the cost of reduced functionality - it means I can’t change the telephony settings and make my family member’s VoIP work on a different ISP. It removes the choices I have as a consumer of technology

The conclusion

Abstraction is useful. It makes our lives easier. Simpler.

But it also reduces nuances. Complexity. And capability.

Sometimes, we need ways to reduce abstraction again.

To re-wild that which we’ve simplified.

And give ourselves more control and choice over how we use our technology.

Artwork: Wes Dickinson “modem glow” via Flickr CC-BY-NC 2.0

Facilitator Vanessa Toholka first prompted Meredith to explore how we got here - to a world of surveillance capitalism where data about us is used as a form of capital, allowing a concentration of power in big tech companies (Shoshana Zuboff and Jathan Sadowski were never mentioned by name, but their works on surveillance capitalism and datafication were palpable in the discussion). Here, she drew on neo-liberal history at the start of the tech boom in the 1990s, showing how permissible the regulatory environment was for organisations at the time - there were few constraints on how they could approach data capture. This led to an ecosystem where there were essentially two options for revenue generation: one for infrastructure - where revenues were generated by providing carriage of data, carriage of electricity or provision of storage; and one for advertising - where revenues were generated by micro-targeting cohorts and delivering ads to them. Even today, there is little room for alternative business models. Reflecting on this, I couldn’t help but be struck by Mozilla’s recent move into advertising, which seems deeply at odds with its ethos. All paths, it seems, lead to advertising.

Following on from this, Whittaker unpacked the etymology of AI, and how the term arose as a way to distance the work of John McCarthy - its coiner - from that of Norbert Wiener and his preferred term of cybernetics. This reminded me of the work of Thomas Rid’s book The Lost History of Cybernetics - which explicates the tension between east-coast and west-coast technological developments of the 1940s to the 1990s¹. She recalled the pivotal moment in 2012 when AlexNet was created - the fore-runner of ImageNet, which harnessed decades-old convolutional neural network technology - made possible by the availability of large-scale data², and noted how many AI practitioners, such as Yann LeCun, had gone to work for Big Tech - such as Meta or DeepMind.

The discussion turned to the imbrication of power and technology, and Whittaker drew on the work of authors such as David L. Crane and his book Making the Movement: How Activists Fought for Civil Rights with Buttons, Flyers, Pins, and Posters, and the work of Canadian activist Ursula Franklin - author of the Real World of Technology, in which she shows how science interacts with society and democracy - very much in the STS space.

One of the most resonant points Whittaker made here was around how privilege operates within activism. She is well-known for helping to organise - she explicitly eschewed the term “leader” - the Google Walkouts in 2018. She quoted the phrase “dig where you stand” - a social movement in the Nordics in the 1970s promoting public participation in labour history - and which has at its core a greater appreciation for the lived experience of the working class. Here, she advocated that we have to be aware of our privilege - and act in a way that protects people of less privilege. For example, there were many H1-B visa holders at Google in 2018. They couldn’t walk out, because they would have been risking their right to residency in the United States.

So “digging where you stand” is about appreciating the privilege and positionality you have in a context, and doing what you are able to to change or resist the status quo. Reflecting on this, as a technologist, I hold a great deal of privilege in the technologies I am able to use, because of my profession and education. I can run a PiHole and a uBlock and a Flipper Zero. And I can educate others about how to use these technologies to resist surveillance and data/capta capture.That is, there is a moral obligation upon technologists - even if companies such as Google choose to remove imperatives such as “do no evil” from their constitutions.

This was under-scored by a discussion on “surgical strikes” - in which drone technologies choose targets based on “data signatures” - patterns of behaviour that are associated with a profile of a particular target. These targets are stochastic, non-deterministic. And if you don’t have enough targets, you can just “lower the confidence score” to acquire more.

The audience, quite rightly, were aghast at this.

Where I thought this conversation might go - but didn’t - was into the area of encryption, particularly given the recent debate of a “Chat Control” law in the European Union, which would allow scanning of chat apps - like Signal, WhatsApp and others - ostensibly to detect CSAM.

Whittaker is deeply opposed to laws such as these which seek to weaken end to end encryption:

Either end-to-end encryption protects everyone, and enshrines security and privacy, or it’s broken for everyone. And breaking end-to-end encryption, particularly at such a geopolitically volatile time, is a disastrous proposition.

New Branding, Same Scanning: “Upload Moderation” Undermines End-to-End Encryption
A statement from Meredith Whittaker, Signal President, in the context of the EU debate
June 17th 2024

Regardless, the evening’s conversation was deeply engaging, provocative and well-informed.

Brava, Meredith and Vanessa.

What is an academic conference, and what makes it different to other types of conferences?

Here begins a brief typology of conferences.

• Hobbyist or community driven conferences - if you’ve ever been to an Everything Open or a DrupalSouth or a WordCamp or to something like a Perl or Rust conference, chances are it’s been volunteer organised by passionate members of a community. Sure, it comes with large costs like a venue, catering, conference dinners and so on - and needs decent sponsorship to run at-cost, but it’s run by hobbyists. I don’t say amateurs, because many of the community conferences I’ve been to have been better organised than commercial conferences. These sorts of conferences are often auspiced by a peak body or foundation, which provides a legal structure for things like insurance and financials. For example, Everything Open, DrupalSouth and WordCamp are all auspiced by Linux Australia. Profits from the conference - if there are any - generally go to supporting the community the conference is for (and by). Ticket prices for these conferences are generally more affordable, and vary with things like the ability of the organisers to attract sponsorship. Organisation of the conference tends to move around the community - it will be in one city in one year, then move to a different city the next, with a new team every year.
  
• Conferences with professional event management or audio visual or other professionally delivered services - sometimes a community driven conference will hire professional services for a portion of the event. This might be hiring a professional event manager, paying for professional audio visual services from the venue or a third party, and having catering supplied. The amount of professional services brought in will vary with the conference’s complexity and budget.
  
• Commercial conferences - here, the conference is a commercial venture, designed to generate a return on investment - whether that’s monetary or in the form of brand exposure and marketing leads. If you’ve been to something like PAX, or where the tickets were $AUD 2500 for two days, then you’re at a commercial conference. The sponsors generally buy speaking time, or “present” various areas of the conference. Companies who run large conferences will often have a dedicated conferencing team who run the conference - or suite of conferences - on an ongoing basis. Most of the Quest Events conferences - such as the Women in Technology Summit or the Generative AI summit fall into this category.

An academic conference generally follows a community conference format, with the following distinctions:

• The auspicing body is usually a learned society or academic association. Some of the societies and academic associations that auspice several conferences include the Association for Computing Machinery (ACM) - which runs conferences like FAccT and EEAMO - and the Institute of Electrical and Electronic Engineers (IEEE) - which runs several computing-related conferences.
  
• Like a community conference, the organising team generally rotates across institutions and locations.
  
• However, the people organising the conference are usually academics who practice in the field that the community is about. This is, they are hobbyist conference organisers. They may have a lot of experience, or may not. But conference organising is not their core competency (even though they may be very competent in it).
  
• Academic conferences often have additional requirements to community conferences to meet academic standards. These include:
  • Peer review of papers submitted to the CfP (often requiring 3-4 reviews per paper by people familiar with the discipline)
  • Production of a Conference Proceedings, using publishing tools like LaTeX, using platforms like OpenReview or EasyChair
  • Submission of proceedings to academic indices such as Scopus or to a proceedings anthology site (like ACL Anthology).

That is, academic conferences have all the requirements of non-academic conferences - venue booking, catering, social event management, schedule creation and tracking, speaker liaison, audio visual management (especially in an era of Zoom and hybrid), promotion of the event via media and public relations channels, delegate visa issues, accommodation and travel - and they have additional academic requirements like conference proceedings.

Why is an integrated academic conference service needed?

So, what’s the problem here?

Running conferences is part of a broader literature on academic workloads. I’m not going to litigate that material here, other than to say that over the past two decades, a more managerial approach to governing universities has resulted in the creation of workload models that try to capture the breadth of activities academics perform, and link them to KPIs used for academic promotion - such as the number and quality of research outputs and the load and quality of teaching¹.

However, from what I can tell, running conferences - as opposed to presenting at conferences - does not form part of mainstream academic workload models (but to be fair I haven’t conducted a thorough analysis - but it doesn’t show up in a cursory search of academic workload models).

For academics, running conferences is, in general, a non-promoteable activity. It doesn’t “count”. But without this activity, the conferences that academics need to present at in order to be promoted don’t happen. It’s paradoxical. Academics need conferences, but running conferences is not seen as a valuable activity for academics to undertake (because if it was, it would count toward promotion).

Running conferences takes academics away from their teaching and research activities. The activities that count towards KPIs and to promotion. Sure, aspects like reviewing submitted papers, or presenting research is directly aligned with academic KPIs, but running conferences is not.

So, there are two choices here. Either running conferences needs to count toward promotion, or running conferences - at least the non-academic aspects - needs to be moved to a non-academic function. Here, I’m arguing for the latter case - we need a dedicated, integrated service within universities for academic conferencing so academics can spend more time producing research outputs to be presented at conferences.

What would an integrated academic conferencing service look like?

Putting together a conference requires many disparate activities, and they “sit” in different places in universities.

• Room bookings: Some universities have conference centres or equivalent facilities, and space is managed differently at different institutions. But what is common is that room bookings are generally spread over three or more systems, with varying business rules around who can book what space when. This is an administrative task, that requires internal systems knowledge of the university. If the conference venue is off-campus, facilities bookings still need to be made
  
• Audio visual: A great conference requires great audio visual facilities - and the expertise to use them well. Even on the same university campus, different rooms have different equipment, and you need local knowledge to use them well. Moreover, AV support is needed during conferences - people now bring their own laptops. AV sometimes sits in the IT function, sometimes in a facilities function. And having AV people available in person is now a harder ask, as much AV equipment is now configured to be remotely monitored - IT service delivery is set up to only send people - “boots on the ground” - when a failure occurs, not to help set up equipment. But you need AV-savvy support during a conference, particularly if it’s running in hybrid mode.
  
• Event management: There are many activities involved in event management such as catering, security, VIP protocols if VIPs are in attendance, Welcome to Country ceremonies, car parking, logistics, registration and name tags - and they are primarily administrative and public relations functions. In universities, Event Management is often a central Marketing function, but is usually tasked with major university events like graduation and open days.
  
• Website: Each conference needs a new website every year, which contains information. Most conferences also require some form of registration, which might be provided by the Auspicing organisation, or via an external site like EventBrite or Humanitix. Each conference tends to use a different website setup - WordPress, Jekyll, Squarespace and so on. Within a university, the website content editing function is either centralised within Marketing, or devolved across organisational areas. However, universities tend to use enterprise CMSs, and conferences don’t. So it usually falls to a conference volunteer to wrangle the website (and its analytics, and integrations and so on).
  
• Publicity: Conferences need publicity to get people to submit to the CfP process, and to get people to register to attend. Many conferences will have an announcements mailing list or similar, and will have one or more social media properties. But someone needs to manage these - producing content not just for announcements, but surfacing content from previous conferences to grow awareness of the conference. This activity is even harder these days as the social media landscape has fragmented.
  
• Conference proceedings: Academic conference submissions are generally done through a platform, which generally uses LaTeX and a LaTeX template. The proceedings are then usually published, again through a platform. This often falls to a team member, when it’s largely an administrative duty - albeit one requiring some knowledge of the platforms, and LaTeX.
  
• Sponsorship: Garnering sponsorship for a conference is tough - and it’s harder to wrangle smaller amounts from many companies. It’s much easier to have fewer sponsors, of larger amounts. But this requires long-term relationships, and trust that sponsors will get value from their sponsorship - which is built up over time. In a university, generally the only people who deal with sponsorships are Advancement or Giving departments - and then generally it’s from a philanthropy or giving perspective.

Good academic conferences need all these functions working together in an integrated way.

How would this work in practice?

The way I see this working is that a full-time team is formed with the above skills, and the team provides services to conference leads - usually senior academics in faculties and schools. The conference lead acts as the client or in a project management sense, the Project Sponsor.

Conferences don’t run all year round - they are clustered in university holidays if the conference society or auspice is in Australia, but for auspicing organisations based in the United States, they tend to run in the American summer (June to August). Having a dedicated team means the conference prep could happen in the downtime before the actual conference happening. If there are not enough conferences being run in the university to keep a full time team engaged, then the team’s skills could be used to assist faculty and school events - which require many of the same skill sets - in particular things like audio visual.

There are going to be peak periods of operation for an integrated, dedicated team - and this is where I see casuals - often university students - having a key role to play. For example, media or journalism students could be engaged to do conference PR and social media, and project management students would be able to assist with conference organisation - there’s even an opportunity here for Work Integrated Learning.

There would need to be key criteria for the integrated team to take on a conference - because they have to be funded, and they have to be able to manage demand. Some of the criteria I see here would be the international profile of the conference, or perhaps its academic ranking. Having a FAccT or a CHI conference at your campus brings huge kudos to the university.

In terms of funding, I see a core team of 2-3 people - project management, audio visual and web/media - being able to prove the viability of the model. Back of the envelope, that’s about $AUD350k for 3-ish salaries plus on-costs. The challenge is being able to prove RoI for that $350k / year - how does the university get a return on that investment? KPIs that I see as useful here would be the increased presence on campus during “down” periods (which means that campus food outlets are more frequented), increased research collaborations, and increased opportunities for university students to attend, and be involved, in conferences.

But all of these are second or third order impacts of an integrated conference team. In Richard Rumelt’s book Good Strategy/Bad Strategy², he talks of hierarchies of objectives, and by setting a proximate objective, or guiding objective, lower level details will be taking care of. The proximate objective I see for an integrated conferencing service is “making the university the premier destination for academic conferences in Australia and the world”.

What are the risks, benefits and barriers over the current state of affairs?

Having a team with all of the above skills, full time, in a university would provide several benefits over ad-hoc teams forming for each conference.

• Firstly, it helps to make the staff time and effort spent on conferences much more transparent. Currently, a lot of conference work is distributed across schools and faculties. By bringing this work to a dedicated team, it makes the workload of conferencing more transparent.
  
• Secondly, by having a dedicated team, there is more scope for pooling resources for things like hybrid conferencing solutions. Zoom is great, but it’s not designed for hybrid conferences - when platforms like Venueless and Streamyard and others offer a better experience. Individually they’re expensive - and cumbersome to learn. By having a dedicated team that can deliver an excellent hybrid conferencing solution repeatedly, great solutions can be identified, tested - and replicated.
  
• From a content perspective, instead of having multiple staff create similar web content, a library of conference content for the university - travel, accommodation, sightseeing - could be produced that becomes “copy and paste”-able - and which also helps to showcase the university itself.
  
• A dedicated conferencing team would be better able to negotiate commercial arrangements with suppliers such as caterers and accommodation providers - because they could negotiate over a portfolio of conferences rather than a single conference.

There are also risks and barriers to this proposal:

• The business case is flakey - because the benefits are not immediately realisable (they take three or four steps on an investment logic map - they are indirect).
  
• Integrated teams that have skills from multiple areas - IT, facilities, Marketing - are always ripe for organisational empire building. These areas are not going to agree to having funding going to an integrated team, and will instead argue “those FTE roles should sit in my department and we will provide the service through our existing service management framework”. But what this means in reality for conference leads - often senior academics - is that they have to deal with three or four departments to get an outcome - that is, the academic has to do the co-ordinating work. The work they should be doing is the disciplinary work - the reviewing of submissions, setting a conference theme, facilitating research collaborations.

Parting thoughts

This isn’t a complete proposal - but it’s a start.

What are your thoughts? Is this service needed? Is it missing anything? How could it be made to work better?

Image: Reza Haffari keynoting #ALTA2023 in Melbourne, November 2023 via the conference’s Twitter channel @altanlp

I recently published a small #Tootfic - a very short piece of fiction writing - on Mastodon, and it garnered, unexpectedly, a large readership. So I wanted to unpack some of its themes in a longer-form post.

The Tootfic

Tay gently pushed the plastic door of the printer shut with an edifying “click”.

Servicing Dark Printers had been illegal for years now. They enjoyed the seditious thrill.

It had started as a subscription grab after the printer companies tried hobbling third party toner cartridges.

“Subscribe for a monthly fee and you’ll never run out of toner again.”

“Let us monitor your printer so you don’t have to.”

People saw it for what it was - vendor lock in - but they had no choice really, not after all the printer companies started doing it.

Then came generative AI.

Everyone wanted to scrape every word ever written on the internet, tokenize it and feed it to an #LLM. #Reddit sold out, then #Tumblr, even open source darling #WordPress - selling out their user base for filthy token lucre.

So people started hiding their words, their art, their thoughts, their expression, not behind disrespected robots.txt, but through obscurity.

Rejecting Website Boy’s “fewer algorithmic fanfares”, they forked into the Dark Fedi.

Unscrapeable, unscrutable, ungovernable.

But people had forgotten about the printers.

The printers had to be connected 24/7, for “monitoring”.

But you could tokenize postscript as easily as HTML.

And so every time a document was sent to a printer, it was harvested for tokens. Even secure documents. Documents not online.

Tay shut the metal door behind them, Dark Printer cossetted safely in its Faraday cage, and shuffled the hot stack of A4 paper it had borne.

It was a children’s story, about how words were sacred, and special, and how you had to earn the right to use them.

Unpacking Dark Printers

Central to this story is the concept of a “Dark Printer”, and I want to unpack how I arrived at this term.

Like all good dystopian science fiction, this piece has its roots in reality.

How printers work, and how printer companies want them to work

In the news this week, Hewlett-Packard unveiled an “all-in” printing subscription service, in which they lease the customer a printer, monitor the customer’s printing activities on the device, and send ink refills when needed. Many enterprise printing services work this way - allowing for remote monitoring, and reducing the “boots on the ground” needed to service the printing fleet.

But what caught my attention was this particular line:

“However, HP will also remotely monitor the type of documents (for example, a PDF or JPEG) printed, the devices and software used to initiate the print job, “peripheral devices,” and any other “metrics” that HP thinks are related to the subscription and decides to add to its remote monitoring.”

Scharon Harding, “HP wants you to pay up to $36/month to rent a printer that it monitors”,
Ars Technica, 1st March 2024

It’s not clear from this disclosure whether Hewlett-Packard would be monitoring the actual print job - just the software used to create it. When printing from the desktop to a desktop-connected printer, then generally, the job is spooled to the printer using software on the local machine. That is, the print job does not go to the cloud and then to the printer.

But in enterprise settings, you don’t print from your local machine straight to the printer. You send your print job, using the “Print” button, to a print server. The print server is usually a somewhere in the company’s network, and the job of the print server is to process the print job, and send it to the right printer - usually the one that you specified when you printed the print job. It’s the print server that provides functionality such as “release anywhere” printing - where you don’t need to specify the printer you want, you just go to the nearest printer and “release” the job.

The print server mediates the interaction between the user and their print job.

Mediation is the act of negotiating an outcome between two parties. Sometimes, mediation is requested, or desired, by the two parties - a marriage counsellor helping two spouses navigate their way through conflict - or an arbitration expert assisting two business people to reach an agreeable settlement.

Technological mediation refers to the how technology affords¹, constrains and shapes human action and human experience².

Companies often seek to to mediate our interactions with technology, in an effort to shape our behaviours, often to suit the purpose of the company. Think about default settings in apps or platforms - who you share content with, what the app is allowed to do, how it operates³. Are those defaults designed for you or for the company?

What printer companies are doing by moving to a subscription model is seeking to mediate your interactions with the printer. They want to constrain what you’re able to do with the technology - to derive a profit. This includes “blocking” third-party toner cartridges through firmware updates.

It doesn’t take much imagination to pull on this thread and consider what other actions printer companies might constrain. I wondered, for example, what might happen if a company that sells cloud subscription-based graphics software bought out a printer company? Would you only be able to print from that graphics software on your printer? This scenario is reminiscent of Piers Anthony’s short story The Toaster from 1985, in which toasters constrain what bread can be toasted in them, and they have to be jailbroken to use third party ink, I mean, bread (thanks Morty for the reference).

The race to ingest all the world’s words

Rather than take that path, I wanted to intersect with another key theme that is as yet under-explored in dystopian science fiction (with, perhaps, the exception of Neal Stephenson’s Anathem and its treatment of bullshit generation).

The rise of generative AI for words - ChatGPT and Mistral, images - DALL-E 2, Image Creator, VALL-E for voices - is old news in general. But what is only starting to surface is the unquenchable thirst LLMs have for more input data - more words, more images, and more voices - that can be tokenized to train them.

I don’t have good sources for this, but I suspect that LLM companies have already ingested all the words available on the public internet. This would explain the desire to build “data partnerships” with companies - where the LLM maker gets access to private data for LLM training, and the company is able to fine-tune the LLM using their private data.

It’s plausible that printer companies would want in on this action, right?

Dark X

And a third thread I wanted to introduce was the concept of “dark”.

We have the “dark web” - unsearchable web servers accessible only using tools like ToR. And we have “dark kitchens” - delivery-only restaurant operations that are hidden from plain sight, part of a broader movement of “dark stores”. We have “dark patterns” - user interfaces designed to malevolently steer users to perform actions they may not have intended (see mediation).

And I wondered what these patterns would mean for printers in a world where every word is scrapeable and saleable for tokenisation and LLM ingestino.

Printers that are not visible, not connected to the web. Printers that are not monitored. Printers that are unable to mediate.

Dark Printers.

Image credit: Christiaan Colen HP 7310 Printer, CC-BY-SA via Flickr

However, dask required a steep learning curve - it’s been decades since I covered parallel or concurrent programming in my undergrad - and I’m noting my learnings here for anyone else following a similar learning path.

The problem

The specific problem I was trying to solve with dask was to append a value to a column in a pandas dataframe, where another column value matched a value from a tsv file. This sounds really straightforward doesn’t it?

for rows in dataframe: 
    for rows in tsv file: 
        if match: 
            append value to column
            break # we don’t need to keep searching if matched

And it would be.

Except for the problem of scale.

In my case, the pandas dataframe was over 1 million rows long, and the tsv file was over 2 million rows long. This meant that for every row in the dataframe, Python had to search up to 2 million rows in the tsv - meaning an upper bound of 1 million x 2 million = 2 trillion (2 x 10¹²) operations to match all the rows in the dataframe (and a lower bound, assuming that each row in the dataframe matched the first row in the tsv file, which is, admittedly, unrealistic, of 1 million or 1 x 10⁶ operations). In Big O notation, used to show the computational complexity of algorithms, this approach would have linear complexity - - that is, the computation scales linearly with an increase in either the dataframe or tsv file size (I think - algorithms are not my forte. I’m sure if I’m wrong I will be corrected 😉 ).

I initially set this to run on jupyter, thinking, naïvely, that it would automagically find a way to parallelise this operation over my laptop’s 16 cores. I was wrong, and jupyter, using the tqdmlibrary, estimated that this operation would take 21 days to complete. I confirmed this by running htop, which showed that only one of my CPU cores was heavily loaded.

Selecting dask

I first considered some optimisations to reduce the computation required.

Could I remove the value in the tsv file once a match was found, thus reducing the search space of subsequent operations? This would have been possible if I was certain that there was one and only match for each row in the dataframe with a row in the tsv file, but that wasn’t the case here.

Could I use Python’s inbuilt multiprocessing library to help?

I tried this, and my jupyter notebook ran for several hours , but I had done something wrong with the start() and join() calls, and manual verification of the matching failed.

So, I turned to dask, primarily because it appeared to be specifically addressing the problem of scaling data science-based workloads in Python, and I had also seen Genevieve Buckley’s talk using dask for image processing at PyConline AU 2020, and thought it might be the right tool for the job.

First steps with dask

But I hadn’t done concurrency or parallelisation work since undergrad, and needed to start from the basics. I tried the dask tutorial, which I had to run in binder because I couldn’t find the right set of dependencies to run it under pip on my local machine (no, I’m not running conda and pip together, sorry).

binder, to be blunt, sucked. It refreshed my notebook every waves hands 15 minutes, and I lost the work that I had done. So, a frustrating start. But I was grateful that there was at least a tutorial available.

This gave me at least some basic concepts of dask:

• The Client - which communicates with the Scheduler, whose job it is to schedule tasks across the workers, processes, threads and processors.
• The concept of collections - data structures such as Arrays, Dataframes and Bags, which mimic data structures familiar to those who use pandas or numpy.
• Dask dataframes - which parallelise pandas dataframes using partitioning
• The concept of lazy evaluation, in which computation is delayed until needed, which provides efficiencies in task sharing. The meta keyword, when passed to a function that undertakes some sort of computation, tells daskwhat the resulting output data structure should be. daskhas some handy built-ins for generating values for meta. From what I could understand, metaassistsdask in optimisation, because the resulting data structure is known a priori.
• The concept of intermediate results, for example entire dataframes that are calculated before being reduced in some way, and how dask handles them.
• Using the persist() function to store commonly accessed data in memory, as long as that object was small enough to fit into available memory.

That was enough to get started on the problem.

Setting up my environment to run dask

Installing daskwithin Python was reasonably straightforward. It essentially requires installing a Python package, and a range of dependencies. The only one that had to be installed as a binary was graphviz. Because I was using dask with jupyter, I installed the dask-labextension package into my jupyter virtual environment, which provided observability into dask’s operations, such as CPU and memory utilisation, and task execution.

Writing my dataframe iteration as a function

One of design features I really appreciated while learningdaskwas that much of the functionality of pandas dataframes was replicated: that is, I could create a dask dataframe, and for the most part, it would have similar behaviour to that of a pandasdataframe. daskabstracted away the complexity underneath.

However, in order to take full advantage of dask’s capabilities, I needed to write code in a particular way. Iterating over a dataframe in pandas, using something like iterrows(), is an inherently expensive operation, and the same is true in dask.

However, dask provides a way to map a function across partitions of a dataframe, so that the iterations can be processed in parallel.

out_ddf = ddf.map_partitions(myFunction, 
                             myParameter,  
                             meta=columns_meta)

This seemed straightforward at first, but the devil was in the detail.

Finding optimal settings is the hardest part of dask

Partitioning the dask dataframe

The key thing I struggled with was understanding how to optimise settings like the number of partitions of the dask dataframe, the number of workers or threads the Client should have and so on. There are several tradeoffs in these decisions: the more partitions your dask dataframe has, the more overhead there is in joining the results of each of the partitions (which are computed in parallel), but the more it takes advantages of multiple cores, if available.

The daskdocumentation has some information on best practices for partitioning dask dataframes, and there was also a useful article on Stack Overflow on the topic by Matthew Rocklin, a core developer of dask, and who answers a lot of the Stack Overflow question on dask (huge thank you, Matthew!).

In essence, daskdefaults to partitioning dataframes so that each partition is around 100Mb in size. What I found in my experiments however, was that each partition engaged a single CPU core. That is, if the dataframe was partitioned into eight partitions by default, then only eight cores were engaged. That meant that I couldn’t have a higher number of partitions than my number of cores. I eventually settled on using 100 partitions, where I had 512 cores (256 cores dual-threaded) available.

Optimal settings for the Client

I had similar struggles in finding the optimal settings for Client, too. In dask, the Client is the main interface with the Scheduler. The Scheduler’s job is to optimise execution of the workload, such as by determining the dependency graph, in which the dependency of computation operations is calculated, decomposing operations into smaller operations, and co-ordinating workers that are performing computation, and reporting results back to the Client.

At this point, I decided to read through Holden Karau and Mika Kimmins’ book Scaling Python with Dask, to glean some more pointers (no pun intended).

This explained in more detail the relationship between the Client and Scheduler, but I still remain confused about different parts of the dask ecosystem, such as distributed - the library that allows for lazy evaluation.

In particular, this book provided more information on the optimal choice of Client settings, including balancing the number of workers and threads depending on the type of compute workload you were processing. For my workload - iterating over a large dataframe - I found that a small number of workers (24) each having a large number of threads (64) worked well, as long as I assigned sufficient RAM to each worker (64GB).

In previous iterations, I had tried increasing the number of workers well over 100, but rapidly ran into the situation of workers dying unexpectedly. When digging into this, I found that it was always the “communications” worker that was dying, and I suspect what was happening was that there were too many workers for the communications worker to keep track of inter-worker communications, causing it to die. Again, there was a trade-off here: more workers meant more parallelisation of available compute infrastructure, but it also increased the number of worker inter-communication processes.

Nested delayed objects

Another “gotcha” for me was nested delayed objects.

Above, I had rewritten my function using map_partitions():

out_ddf = ddf.map_partitions(myFunction, 
                             myParameter,  
                             meta=columns_meta)

I then needed to perform lazy evaluation on this, using;

result_df = out_ddf.compute()

However, when I inspected the value of result_df, it was a dataframe, but it contained 100 delayed objects, rather than the processed dataframes I was hoping for. This is because the mapping function had nested delayed objects - so I had to run another level of lazy evaluation:

completed_df = dask.compute(*result_df)

This did result in the evaluated dataframes I needed - and then I simply combined the 100 dataframes together with a merge operation.

merged_df = pd.concat(completed_df, ignore_index=True)

Next steps with dask

There’s a lot that I still need to learn with dask - such as different data structures like Arrays and Bags, which I didn’t use in this task. But I’m still pretty pleased at being able to figure out enough dask to compute something that was going to take 21 days on a single CPU in around 12 hours on a CPU cluster!

Written for a business audience, this book has two distinct sections. The first provides a gentle, integrated primer on voice technologies, such as automatic speech recognition (ASR), speech to text (STT), text to speech (TTS) or voice cloning, and natural language processing (NLP), and links these to the human needs fulfilled by voice technology. The second is essentially an extended pitch deck. Unabashedly techno-optimist in outlook, it seeks to grow the market for voice technologies by encouraging the reader to examine their own organisation’s operations for voice technology use cases, and provides a detailed guide to the user research and interface design steps needed to implement a voice technology program.

This is unsurprising, given Tobias Dengel is the CEO of WillowTree, an AI and digital product consulting company recently acquired by TELUS international for $USD 1.2 billion - which focuses on gathering training data for AI applications. His expertise in human-computer interaction (HCI) and user-centred design (UCD) is evident in the first half of the book, where voice technologies are continually grounded in user tasks and experiences. In the second, his experience is shown in the methods advocated for exploring voice use cases, with a focus on HCD methods such as journey mapping. Co-author Karl Weber is an editor; his collaboration with Dengel makes the text readily approachable and succinct; terms unfamiliar to the lay reader are well described, and the use of acronyms is minimal.

The book draws heavily on examples from industry to highlight key claims, however some of these are now dated. Stanford Open Voice Assistant Lab (OVAL)’s Almond assistant was re-named Genie in 2021, however has not had any active development for over two years, and the research group has pivoted to working primarily in the large language model (LLM) space. The Open Voice Network’s initiatives on trustworthy voice assistants have now been folded into the umbrella of the Linux Foundation. This is perhaps unavoidable in such a fast-moving space.

Part One - Aligning the use of voice technology to the human need for communication

Each chapter in the first half of the book details a particular human need that is met by voice technology.

The Prologue paints a picture of the transformative power of voice tech, showing how it was used to help those physically impaired to be able to communicate again - using speech - the most natural form of communication.

The Introduction makes a bolder claim - that voice is a technological revolution - akin to the internet or to the mobile phone: nascent, latent, reaching a tipping point of “ubiquity and popularity” that we should all be prepared for lest it catch us unawares. While acknowledging that voice tech is currently limited in application, and harbours a panoply of challenges, the authors hand-wave these away, pointing to the rapid advances being made across the vibrant voice tech ecosystem - inhabited by companies such as ReadSpeaker, SoundHound, Cerence and others. The sizeable investments made in voice are given as evidence for the technological revolution, but differentiated from over-hyped failures such as blockchain and the metaverse in that voice “fulfills basic human needs”, which are articulated in subsequent chapters.

Speed makes the case for “even marginal improvements in speed/efficiency” when designing user interfaces, highlighting examples such as search engines and online shopping websites to reinforce the point that speaking to machines is often quicker than typing to them. It imagines a world where the keyboard is eschewed in favour of the the microphone as the primary mode of data input, because this is faster - and time is money. The physical toll of such a change - can you imagine speaking for the same amount of time you type? - is left unexamined. I wonder what Mica Endsley or other human factors scholars would make of this claim.

The next chapter demonstrates how voice technology meets the need of Safety - by being available to assist when the user is physically incapacitated. There is a claim made in this chapter that was particularly contentious: that having a voice assistant in the cockpit would “prevent crashes and save lives”. While plane <-> tower communication is definitely a contributory factor to many incidents, there is no discussion here of the complexity introduced by voice assistants. Imagine, for example, the utterance engine one out! being mis-transcribed as engine won naught!. Sure, the language model can be weighted for cockpit utterances, but mis-transcription is still rife, even in state of the art systems (Whisper, for example, has a 9.3% Word Error Rate as tested on Common Voice 15).

Knowledge makes the case for voice technology as an interface to the world’s information. Rather than having data at your finger tips, it’s now available on the tip of your tongue - overcoming the limitations of screen real estate. Dengel and Weber also make the case here for voice where users are not computer literate: you don’t have to know how to use a computer to ask a question of a voice assistant. What is not well explained here is that access to knowledge is mediated through millions of APIs - and to curate or synthesise them requires additional capabilities. The potential for commercialisation to skew results in a particular way (such as booking sites preferencing those providers that pay them the most) is left unaddressed. This chapter also touches on voice technology as one of many anticipatory systems - having predictive capabilities through audio feature detection to infer an event is about to happen, and respond. What isn’t covered is the downside of this form of machine surveillance, covered well by researchers Joel Stern, Sean Dockray and James Parker in their Machine Listening: Exposed collaboration.

In the chapter on Inclusion, the authors make the case for voice technology building “a more inclusive society”, pointing to advancements in screen readers, speech to text and smart hearing aids as mechanisms that help in “…liberating and empowering individuals who have too long been excluded from mainstream society…”. The challenge of machine translation for the world’s 7100 spoken languages is also addressed, and inequities in the availability of tooling for under-resourced languages and the existing Anglo-centrism of the tech sector, quite rightly, highlighted. Kathleen Siminyu’s work with Common Voice’s East Africa project, which is providing speech data and tools for the Kiswahili project, gets a mention, which delighted me, however when chatting with her, she was unaware of being featured. Absent was any argument for addressing the lack of investment in low-yield languages - languages whose speakers are not “profitable”. This is likely to remain the purview of NGOs and governments for the foreseeable future, lamentably.

Engagement makes the case for voice technology making life “more creative, entertaining and enjoyable”, using radio and television as previously emerging technologies that were fun to use, which drove adoption. Dengel and Weber speculate about what might happen to voice actors in a time of synthesised voices, seeing both the economic reality of the cost of live narration, and, counter-intuitively, the increasing value of human voices in a soundscape saturated by synthetic speech. They go on to link voice tech to the metaverse and to virtual reality, showing how it is a necessary building block in “multi-modal” experiences. Again, there was no concomitant discussion of the ethics of synthetic speech - and importantly, how “synthetification” - the growing movement to synthetic media - shapes power relations, labour relations and who profits.

The chapter on Transformation ties voice technology to “fundamental changes to business models”, through mechanisms such as voice identification through biometrics, and the aggregation of services to provide a streamlined, personal offering. It covers the move from click-through rate (CTR) in screen advertising to say-through rate (STR) for voice-enabled advertising; again however, it does not explore the ethical or societal issues such changes might bring. I’m reminded here of Professor Joe Turow’s excellent The Voice Catchers - and how voice is being used as a mechanism to target advertising. The chapter goes one step further, exploring the use of vocal biomarkers in health - but again, without the attendant discussion of unintended consequences. Who stands to benefit if a disease can be diagnosed simply through speaking?

Part Two - A program of work for implementing voice technology use cases within the enterprise

Part Two of The Sound of the Future moves from explicating use cases for voice technology to encouraging the reader to implement them, with attendant advice on strategies for doing so.

The chapter on Falling Barriers traces the recent history of voice assistants like Siri and Alexa, positing that what people really want is something more akin to an “all-purpose valet”. This leads into a discussion on technology breakthroughs, and the factors which incentivise them, and uses the COVID-19 pandemic as a case in point - where hands-free, remote interaction provided by voice-enabled devices helped practitioners avoid infection. Here, I would have enjoyed more grounding on the various innovation theories (I cover these very briefly in my post about the inflection point we’re seeing in voice assistants at the moment), however this book is clearly aimed at a business, rather than academic, audience. The chapter goes on to outline the key layers of the voice technology stack, such as automatic speech recognition (ASR), natural language processing (NLP) and conversational AI, providing a precis of the current state of the art of each, and remaining barriers. The paradigm of “multi-modal interaction” is then introduced, situating voice technologies alongside haptics and visual interfaces as a constellation of interfaces that collectively are shifting how we sense and respond to our cyber-physical world. User trust in voice technologies is then introduced as another barrier which must be overcome to ease widespread adoption, in particular citing the Trustmark Initiative from the Open Voice Network as a signal that this barrier is falling. The chapter concludes with an overview of how Dengel sees trajectories of development in voice technology, from automation to business process redesign, to transformation of business models.

Making voice an integral part of your existing business systems encourages the reader to “seize the opportunities” voice technologies present, by first identifying places where voice technology could be integrated into existing business systems. The authors provide a helpful list of six principles for assessing whether an interaction is well suited to voice integration, and go on to use examples from industry to highlight how these principles are applied.

In the Training voice tools to understand your world chapter, the authors cover a problem that has long faced voice technology practitioners - the domain specific nature of spoken language. The utterance (spoken phrase) “twelve fifty” has very different meanings in different contexts - it could mean twelve pounds fifty, 12.50pm, 1250g and so on. The advice here is for organisations to identify the “friction points” their customers face, using tools such as journey mapping to better understand those contexts. The chapter goes on to advocate for prototyping of voice technology tools, using UX methods to elicit feedback to guide iterative development, and ensure that the intent of the user - the task the user wants to perform - is matched by the system. The concepts of error flow handling and conversational repair mechanisms are covered here too - essentially serving as a primer on voice user interface design.

Designing and redesigning the multimodal user experience makes the case for voice technologies as part of an omni-channel digital user experience, highlighting voice’s place in an overall brand experience. It discusses how voice can be used to augment and reinforce other digital channels, such as text-based chatbots or graphical user interfaces. Thankfully, there is little hype about the metaverse - which - given its current white elephant status in industry - would detract from the argument that voice technology on its own is transformative - the argument here is that its transformative power emerges in concert with other technology. The chapter includes advice on how to plan an iterative voice UX (VUX) experience design process, and, also pleasingly, highlights the need for inter-disciplinary teams and executive support.

The concluding paragraphs reiterate the argument that “successful new technology is about meeting basic human needs”, and that to be successful, companies must adopt voice - or face defeat in the marketplace.

The verdict

This book is helpful for businesses who are making their first forays into voice assistants, voice user experience (VUX) or conversational AI, in particular those coming to it from a product management or business analysis background. The use cases for voice are expansively surveyed, and applicable to many industries. However, the technical detail is too light for those needing a deeper guide to the pitfalls of voice technologies, such as accent bias in speech recognition, ambiguous named entity recognition in natural language processing or the privacy dangers of voice cloning.

Moreover, while Dengel and Weber correctly identify that many threads of innovation underpin the current state of the art in voice technology - hardware improvements, advances in deep learning and neural networks, and the availability of more speech data upon which machine learning models may be trained, they gloss over the many challenges in the space - the trust people need to have in assistants, the poor performance of voice technology for accented or disordered speech, the privacy and ethical challenges of requiring user data to be effective, and above all, the question of who profits from speech data gathered from individual people.

Beckoning to previous technological path dependencies, they hold that

“… this is the story of any new technical wave. It takes years for entrepreneurs, designers, and engineers to shift their thinking to take full advantage of any new technological paradigm.”

Introduction

When I think of the precursors to today’s voice and speech technologies - the Audrey and the Shoebox, the Harpy, to the Tangora, to Dragon Naturally Speaking, even as far back as Christian Gottlieb Kratzenstein’s work on synthesising human speech with the “vowel organ”, I can’t help but wonder - are voice technologies really a new technical wave? And in taking full advantage of this new technological paradigm, who is it that is taken advantage of?

If voice is the sound of the future, then we must have other conversations about what that future sounds like - and whose voices are heard.

What’s changed since the last report?

Although I’m still finishing my PhD at the Australian National University’s School of Cybernetics, there’s been a shift this year to doing a lot more work in python, particularly using pandas, MatPlotLib, and very recently, dask - that is, a lot more quantitative data work, and a lot more work with LaTeX, writing research papers.

Software development, data analysis and writing

My work this year has heavily revolved around data analysis (for example, this paper, which uses ObservableHQ and a bunch of pandas to analyse accents in the Common Voice dataset). With Atom being sunsetted, this necessitated a change to my primary software development and data analysis setup, which I also used for writing in Markdown.

Moving to VS code, albeit begrudgingly

Because I write in a few different languages - Markdown and LaTeX for writing and documentation, python for software development, javascript for visualisation, and work with a lot of file formats like tsv and csv and JSON, I needed an IDE that could handle all of these well. Although it pains me to say it, because it’s part of the dominant Microsoft + GitHub + Copilot + OpenAI + LinkedIn + Nuance, VS code turned out to be the ideal solution (when combined with a large set of plugins). VS code has been working very well, and the sync feature in particular - where I can sync settings between multiple installations on multiple operating systems - has been super useful.

Moving to Monaspace fonts

I spend a lot of time in VS Code, writing material, and so my font choice is also critical for productivity. Generally, I use a monospace font - and Ubuntu Mono has for a long time been my go-to. However, after coming across it on Hacker News, I’ve made the switch to the Monaspace family, specifically Monaspace Argon, because of its flexibility. The idea of a flexible font family that can be adjusted with CSS is really appealing, and it’s a beautiful example of font design, reminiscent of Donald E. Knuth’s work on Computer Modern, still widely used in the LaTeX world.

Embracing Overleaf

When I started writing a lot of LaTeX for research work, I baulked at using Overleaf because of the premium pricing. My university now has an enterprise license, so I don’t have to wear the cost of it, and I’ve finally embraced it. If I’m on a slow internet connection (which is rare these days, my Aussie Broadband FTTN service has been rock solid and I get a consistent 55Mbps + down), then I’m able to sync it with a git repo - it supports GitHub natively, and any other git repo via a generic git interface - and workoffline. In the last six months, Overleaf has released some WYSIWYG features which are very useful for drafting, and organisations like the ACM have increased their integration with the platform, making it far easier to submit research papers to journals and conferences. Overleaf is here to say.

Search and information discovery

As generative AI-spawned content begins to fill the internet with hastily-created, ad-stuffed, keyword-matching cruft, our ability to find relevant, factual and detailed content becomes compromised. Search for common python programming-related terms, and the top results will be ad-plastered, pop-up bearing “farticles” (shitty articles) designed to rank on search engines, not deliver answers to questions. And existing search engines like Google or Duck Duck Go haven’t yet caught up. Bing auto-generates incorrect responses to questions - for example if you ask “how can I contact Kathy Reid?”, it provides you with the information for Kathy Reid who is a teacher in Grand Island, New York.

So I wanted a search engine that could sift through the cruft, where high quality websites would be highly-ranked, and where I could exclude farticle farms. Enter Kagi - which variously translates into “now”, “day”, “morning” or “ago” - all with a temporal connotation - in Malay languages.

Referred by a friend, I tried out Kagi for a month - at a cost of around $AUD 8 - and while it was good - I was able to readily filter out low-quality results - I exhausted the number of API calls for my subscription level (300) very quickly - within about two weeks - and made the decision not to increase my subscription ($USD 10 or around $AUD 17 per month). I was impressed by Kagi not having any tracking (less surveillance capitalism is a good thing), but I couldn’t sustain this price point just for a search engine, particularly given the weak Aussie dollar.

A new laptop (also, begrudingly)

I had been using an ASUS Zephyrus ROG G15 since late 2020, and it was a fantastic machine - 16GB of RAM, decent GPU (3060 from memory) and Ryzen CPUs. It was a dream to use. And then, it failed. With a whimper. One night, while I was sitting in bed, the screen just started to show vertical lines, then went blank. I can still use it via an HDMI connection, but investigations done by ASUS showed that there was both a screen fault (replaceable for a few hundred dollars, worth it) and a motherboard error (replaceable for $AUD 2000, not worth it). And so it was time for a new machine.

To be clear, I use my laptop around 12-15 hours a day. It’s the tool of my trade - how I derive an income, an entertainment centre, and something I need to enjoy using. I was very happy with the Zephyrus, but wanted a little more RAM, and ended up getting a run-out G15 with 32GB of RAM, an NVIDIA GeForce 3080 RTX and 16 Ryzen 9 6900HS cores for under $AUD 4k. To be clear, this was a major purchase for me, and my laptop is now worth more than my car. But I drive my 20-year-old car on errands twice a week, not for 12-15 hours a day.

This machine is a dream to use. I dual-boot Windows and Ubuntu 22.04, and it can cut through pandas workloads like a hot knife through butter. Windows 11 is a vast improvement on Windows 10 (once you turn off all the tracking features). The ASUS Linux project doesn’t quite give me equivalent functionality (especially keyboard backlighting) on Linux, but it’s reasonable.

Let’s hope it lasts more than 2 years before hardware failing.

Planning and productivity

Passion Planner ditched in favour of Leuchtturm1917 A4 planner

I’ve been an avid Passion Planner user since 2019 - so five consecutive years - and love the annual week-to-a-view planner. It’s a great product. But, and it’s a big but, the shipping from the United States has become exorbitant - and I was looking at $AUD 40 to ship one A4 planner this year. I really hope they find a wholesale supplier in Australia soon, but in the meantime I shopped around and settled on the Leuchtturm1917 A4 week to a page planner, which retains for $AUD 68, about the same price as the Passion Planner, minus the shipping.

I can’t comment on how well it’s working on out, but you can expect to see an update in next year’s State of the Toolchain.

Changing my social media footprint

I’ve made a concerted effort this year to reduce my presence on Twitter and Instagram. Since Elonification, Twitter has descended into a cess-pit of racist, misogynist drivel. So I’ve locked my account, but have kept it so that the name cannot be hijacked by others. Instead, I’ve found my people on Mastodon, with big ups to Shlee for all they do for aus.social. Mastodon is much more academic-oriented that the Blue Hell Site, and it’s where I spend most of my time.

The Mastodon server I host for academic and research group accounts (that tend not to have a natural home elsewhere) - https://fediverse.au - is ticking along. I originally thought I’d be sunsetting this by the end of 2024, but universities haven’t been quick on the uptake to host their own Mastodon accounts, so I suspect it will be around for a bit longer.

Bluesky seems to be a slow burn at the moment, but I’ve got an account there basically to make sure the name can’t be taken.

I’ve ditched Instagram after they requested I upload a selfie with my name on it to verify my identity. Get forked. Instead, I’ve joined Pixelfed, and love it there (yep, another Fediverse-compatible service hosted by Shlee. As I said, Big Ups).

Goodreads has been bought out for Amazon for a while now, and the closest Fediverse-compatible alternative is shaping up to be Bookwyrm - an in particular its import function, which allows you to bring in content from Goodreads. Big ups here to Hugh Rundle for all the dev work he does on Bookwyrm.

What’s currently in my toolchain?

Hardware, wearables and accessories

• My main laptop is an ASUS ROG Zephyrus G15 (changed since last report)
• Google Pixel 4a 5G running stock Android (no change since last report)
• Mobvoi Ticwatch Pro 2020 model (no change since last report)
• Fellowes book stand - again, very light and easily portable
• Logitech C920 web cam (no change since last report)
• Stadium USB microphone (no change since last report)
• Sennheiser Momentum 3 wireless bluetooth headphones (changed since last report)

Software

• VS Code with a range of plugins for writing code, thesis notes (changed since last report)
• Pandoc for document generation from MarkDown (no change since last report)
• Zotero for referencing (using Better BibTeX extension) (no change since last report)
• OneNote for Linux by @patrikx3 (no change since last report)
• Nightly edition of Firefox (no change since last report)
• Zoom (no change since last report)
• Microsoft Teams for Linux (no change since last report, but boy is it getting rubbish!)
• Gogh for Linux terminal preferences (no change since last report)
• Super productivity (instead of Task Warrior) (no change since last report)
• Cuckoo Timer for Pomodoro sessions (no change since last report)
• RescueTime for time tracking (no change since last report) - I should note that the new Timesheets functionality is super helpful.
• BeeMindr for commitment based goals (no change since last report)
• Mycroft as my Linux-based voice assistant (no change since last report)
• Okular as my preferred PDF reader (instead of Evince on Linux and Adobe Acrobat on Windows) (changed since last report)
• NocoDB for visual database work (no change since last report)
• ObservableHQ for data visualistion (no change since last report)

Techniques

• Pomodoro (no change since last report)
• Passion Planner for planning (no change since last report)
• Time blocking (used on and off, but a lot more recently)

Current gaps in my toolchain - what’s missing?

Visual git and git lfs

This one is more a case of “it’s not missing any more because visual git tools have become more integrated into other tools”. In previous posts, I opined about the lack of low-cost visual git tools (GitKraken was high on my list but at a yearly subscription of $USD 48, or about $AUD 80 with the rubbish exchange rate, it was still too much), but this need has ebbed as git has become more integrated in other tooling. VS Code comes with a bunch of git integrations, which suit 80% of my use cases.

My needs here have also changed because I am using git lfs a lot more, as I work with large data files (git lfs is required if you are committing files larger than 100Mb to most platforms like GitHub or GitLab). It takes a bit of getting used to (it stores pointers rather than objects, and so pruning and the like require a different set of commands), and many of the visual editors don’t provide built in support for git lfs.

So it’s back to the command line for me.

Second brain and organise my life tools - is this a job for generative AI?

After giving StartMe a shot, I found that it just didn’t have enough integrations to be useful, and have stopped using it. I really want something which is the equivalent of my Office 365 home page - with my recent files, websites, recurring tasks - except not limited to the Microsoft universe! I have been half-tempted to write something myself - it would be a cross between IFTTT, Huginn, Dropbox, StartMe and a voice assistant - except this would Be A Very Bad Idea and I stopped myself.

What I want is an integrated personal assistant, that has access to all the third party services I use, and has a discovery or prediction layer that runs on top of all of them - fore-grounding the files or tasks that need immediate attention, back-grounding those less relevant, and predicting, anticipating what I need to do before I need to do it. Privacy and security implications aside (imagine what data brokers would pay for that attention layer in an attention economy), this sounds like a job for generative AI tools.

Dropbox seems to have made a very ham-fisted entre into this genre, by adding tools into its desktop product that integrate with OpenAI, and then turning them on by default. No Dropbox, not like that!

I still think that this use case is a job for generative AI, but its implementation needs to be guided by user agency, user privacy and trust - a similar challenge ahead of voice assistant developers. Moreover, it raises questions of diversity and individual choice. Do I work differently to other people? What about high performers? Will I be “nudged” to structure my work to emulate high performers? And will I be penalised somehow if I do not? This is why I’m so dubious about tools like Microsoft’s usage analytics - they give user-level detail about tool usage and adoption, which are not necessarily good proxies for “productivity”. How might generative AI help me work better, without selling me out?

What’s driving my choice of tools?

You’re probably noticing some key themes about my choice of tools.

• Integration: I need tools to work across as many of my operating systems (Linux, Windows, Android) as seamlessly as possible.
• Privacy: I eschew products that aren’t transparent about what data is collected, or how that data is handled, particularly given the rise in generative AI. I’m now planning to migrate from Dropbox to NextCloud, because Dropbox has broken my trust.
• Lack of lock-in: I want products that I can swap in and out - commodotisation is generally a good thing for consumers. I don’t want to buy a whole bunch of new chargers for a new hardware purchase. I don’t want to be beholden to walled-garden, lock-in ecosystems. This means features like import and export, and support for standard file formats like csv and json.
• Data sovereignty: The concept of data sovereignty is traditionally associated with the collection of data from under-served groups - such as Indigenous groups, with respect to the collection of spoken language data. Here, I use the term to refer to the ownership of data collected about me by the services I use - closely tied to the concept of surveillance capitalism - and who benefits from that data. Can I see how that data is used or how that data is used to create products? How is data about me monetised?

No, I’m not accepting paid sponsorships or links

My State of the Toolchain posts get a bit of traction, and with that, I’ve started attracting people wanting me to link to their product or website, for a fee. That is, sponsored content - but of course they don’t want it to look like sponsored content. I usually respond harshly, dismissively and swearily to these requests. If you want me to write about your product or service, build something awesome.

Introduction, overview and general observations

About ALTA and the ALTA workshop

The ALTA conference is the annual workshop of the Australian Language Technology Association. Proceedings of the workshop are published in the Association for Computational Linguistics (ACL) Anthology.

According to their website:

“The purpose of the Australasian Language Technology Association is to promote language technology research and development in Australia and New Zealand. ALTA will organise regular events for the exchange of research results and for academic and industrial training, and will co-ordinate activities with other professional societies. ALTA is a founding regional organisation of the Asian Federation of Natural Language Processing.”

This was the first time I had attended ALTA, and I submitted here based on it being the key linguistics conference in Australia. There are several data science conferences, but my inter-disciplinary work crosses data, speech recognition, fairness, bias and social science: it’s hard to know where to submit. My work was definitely a little of out of place here; there were only two other speech-related papers accepted - one on cepstral features in signal processing by A/Prof Shunichi Ishihara which was veryinteresting, and one by Yishan Huang on encoding prefixes and suffixes in Southern Min - also interesting. Overall, most of the papers here had a very quantitative and computational focus, unlike say conferences like HCI or CSCW which generally present more qualitative work.

As a first-timer to the Workshop, I found the community welcoming and helpful; lots of people said hello and introduced themselves.

Acceptance rate and

academic impact factor

At the Workshop opening, the Program Chair, Jey Han Lau, advised that the acceptance rate of the conference was 68%; 25 papers were submitted and 17 were accepted. I was surprised that so few papers were submitted to this conference given that it’s the main linguistics conference in Australia, but that became clearer during the ALTA AGM when Massimo Piccardi advised that the conference’s impact rating had been demoted from “Australasian B” to “Australasian C” by CORE - the Computing Research and Education of Australasia. I had never heard of this group before, but apparently the Australian Research Council uses their rankings to assess grant applications. The primary factor contributing to the demotion was that the conference was not indexed by SCOPUS, but the ALTA Committee has addressed this and the conference is now indexed.

Looking at this from a dynamics perspective, it’s clear that rankings play a part in terms of which venues academics working on NLP submit to. This makes sense; academic time is limited, so it’s rational to submit to output targets that are highly ranked in terms of “impact factor”. I wonder, however, whether this means the ALTA workshop misses out on interesting or novel work because it’s considered low-ranked - that is, whether authors “withhold” that work for higher-ranked international conferences like EMNLP?

Venue and hybrid nature

The venue was Melbourne Connect, a purpose-built collaboration space intended to help
academia and industry forge closer connections. The venue positions itself as a “collaboration ecosystem” and the space reflects this goal; there are lots of comfortable seats, presentation spaces, smaller collaboration rooms and lecture-style facilities. Located on Swanston Street between RMIT and University of Melbourne, it was easy to access. The venue had Eduroam WiFi via the University of Melbourne, which was very useful. It was an excellent venue choice for the workshop.

The conference was “online-first” in that most of the presenters and audience was in the physical space, however Zoom was used to provide a hybrid capability for presenters who were unable to be there in person. This was generally well managed, but was often distracting for the audience as each speaker had to connect to Zoom when they were at the lectern. I feel like there has to be an easier way to create a hybrid experience.

What will I do

differently after this conference?

• All things pytorch and python. All of the modelling work done here was done using python and more importantly, pytorch. There was a lot ofscikit-learn being used (I was expecting a lot morespaCy, but folks are using scikit-learn). I am not as fluent in pytorch as I would like to be. Alternatives such as keras and TensorFlowappear to have gone by the wayside.
• Think more deeply about annotation of speech data, such as transcription, because this has implications for how NLP research is conducted on text corpora.

Key themes and disciplinary directions

Large language models, and transformers in particular, dominate the field but there is a turn to “grounding” models using Retrieval Augmented Generation (RAG)

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,
Gomez, A.N., Kaiser, Ł. and Polosukhin, I., 2017. Attention is all you
need. Advances in neural information processing systems, 30.

Devlin, J., Chang, M.W., Lee, K. and Toutanova, K., 2018. Bert:
Pre-training of deep bidirectional transformers for language
understanding. arXiv preprint arXiv:1810.04805.

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O.,
Lewis, M., Zettlemoyer, L. and Stoyanov, V., 2019. Roberta: A robustly
optimized bert pretraining approach. arXiv preprint
arXiv:1907.11692.

Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal,
N., Küttler, H., Lewis, M., Yih, W.T., Rocktäschel, T. and Riedel, S.,
2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.
Advances in Neural Information Processing Systems, 33

Since their advent in 2017, attention-based transformer models have re-shaped most branches of machine learning, and NLP is no exception. Many of the papers presented work which compared traditional statistical models such as alongside transformer-based NLP models such as BERT and RoBERTa. With only some minor exceptions, the transformer-based models outperformed the statistical models, especially when they were fine-tuned for a task.

Transformer models are here to stay.

Moreover, several authors used LLMs such as GPT variants to provide synthetic data for their studies - primarily to augment the sparse data sources they were working with. The competition task for the conference (which I did not participate in) also used LLMs - with the challenge being to distinguish between human-generated and LLM-generated text. There was no discussion of the biases or the material encoded by LLMs - such as them responding with more American-inflected text (e.g. “faucet” instead of “tap” or “diaper” instead of “nappy”), and what implications this has for using LLMs to generate synthetic data.

However, there appears to be a key turn in the field to “grounding” LLMs onto other forms of knowledge representation to improve their explainability and to reduce their tendency to hallucinate factually incorrect responses to prompts. The key tools being used to “ground” LLMs are knowledge graphs. They add a mechanism that allows the LLM to “reason” over the knowledge graph.

This is also known as Retrieval Augmented Generation (RAG) and this is a good blog post on it from NVIDIA.

One of the gaps I see in the LLM space is an Australian-specific LLM and knowledge graph - that can handle Australian-specific words (named entities such as place names - Geelong, Yackandandah, Canberra - which are Indigenous-derived) and maps Australian-specific knowledge - things like not spreading Vegemite too thickly on toast. As we think about metaverses, and about how content is synthetically generated (imagery, video, speech, text, dialogues) within the metaverse, having graphs to ground knowledge for “authentic” or “realistic” experiences is going to be important, particularly if the metaverse is used for training, which is a key use case.

Ensemble models: Using multiple models for better prediction

Many of the papers adopted an “ensemble” approach when building models. Essentially, ensemble learning uses multiple models, combined with some form of voting mechanism, to improve the prediction task. I hadn’t come across these in any depth previously (they’re not generally used in speech recognition) and I found this paper a very useful primer:

Sagi, O. and Rokach, L., 2018. Ensemble learning: A survey. Wiley
Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4),
p.e1249.

Everybody wants to do the model work, not the data work

Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P.
and Aroyo, L.M., 2021, May. “Everyone wants to do the model work, not
the data work”: Data Cascades in High-Stakes AI. In proceedings of the
2021 CHI Conference on Human Factors in Computing Systems
(pp. 1-15).

Nithya Sambasivan and others, in their 2021 paper, highlighted the role that data quality plays in high-stakes environments in which AI technologies are used, and how data quality can cause compounding harms for AI models. The title of this paper - “everybody wants to do the model work, not the data work” - is something I’m constantly reminded of in speech and language technologies. Several speakers highlighted the challenge of being able to obtain enough data from which to build models, which is a particular barrier to NLP in the health field - which, rightly, places strict control on patient medical records. Indeed, one speaker remarked that their model work was “really a very small part of the project” - which had involved 283 patient records being coded by two clinical practitioners to determine a taxonomy that was subsequently used for modelling. In some conferences, such as NeurIPS, there are now dedicated tracks for dataset development, which seek to recognise the impact, and elevate the prestige, of data work.

I wonder if ALTA needs a similar track, dedicated to building NLP datasets?

The need for the intersectionality of NLP and other domains

Many of the talks at the workshop focused on ‘clinical NLP’ - the application of NLP techniques to medical and healthcare settings, primarily because there is a large team at University of Melbourne working on this. The field is recognising the need to intersect with other domains due to the specialist language and language patterns that are specific to domains; this presents more opportunities for inter-disciplinary work.

There is little attention being paid to the broader market implications of LLMs and the dependencies being created

In the keynote by Terrance Szymanski, the Principal Data Scientist at SEEK, he outlined how the LLM market is working - with companies that are producers of LLMs (such as OpenAI and Anthropic), and companies that are consumers of NLP products - essentially most of the companies that don’t do a lot of in-house NLP model production. There are broader implications here that weren’t touched on - such as all of the major LLM providers being American. Sure, there are Australian companies building on top of products like GPT, but we don’t have people building LLM models in Australia. This raises questions of increasing dependence on cloud platform providers - LLMs are another form of cloud product - and a gap in our national capability.

Is this a market gap or opportunity to increase our national capability?

There is also the similar argument here of increasing dependence by academia on corporate-owned LLMs. Professor Heng Ji in her keynote called out the need for an academic-owned LLM to reduce this dependence. There are already efforts here by groups like Eleuther AI who have lots of open source LLMs, but AFAIK there is no stewardship of similar efforts for academia. This is in contrast to open models for NLP like the BERT family, which are freely available.

See also this take about how open source is the winner from the OpenAI fallout.

The need for more work on Australian English and Australian Indigenous languages

I was surprised that none of the papers focused on Australian English or Australian Indigenous languages, especially given the efforts happening around UNESCO’s Decade of Indigenous Languages. This may be because the biennial PULiiMA Indigenous Languages and Technology Conference operates in this space, and is supported by groups working on Indigenous languages like First Languages Australia and AIATSIS. Similarly, while authors frequently described the problem of not having enough data to work with, particularly medical and health data, there were no discussions of data sovereignty, of decolonising language technology or of linguistic justice.

In conversations I’ve had with institutions who are tentatively adopting machine learning-enabled technologies to aid with tasks like preservation of audio-visual material, and indexing of speech collections, the predominant models in this space tend to have poor results on Indigenous-derived words - for example, mis-transcribing “Eora Nation” as “urination”; both incorrect and highly offensive. There is still much to do in this space.

Talk highlights - Day 1 - Wednesday 28th November

Introduction to clinical NLP

Dr Mike Conway, Senior Lecturer in Digital Health

Dr Conway introduced the topic of clinical NLP, which is the application of NLP to medical and health settings. He explained that there isn’t enough intersection between health and computer science. There is huge growth in health related data, but it’s not usable - that is, it’s hard to extract information from it.

Clinical text has specific properties that make it difficult to work with using NLP tools. Some data is structured - like clinical notes, ICD-10 codes, test orders and test results, but some data is unstructured, for example a patient’s preferred name, or whether the patient has social support - e.g. “arrived with wife and daughter”; that’s an inference that can be made from the unstructured data. Many tasks in health are inefficient and NLP has many possible applications - such as real time outbreak and disease surveillance.

Clinical NLP is challenging for many linguistic reasons:

• derivation: forming a new word using a suffix or prefix such as “unrelated” from “related”
• inflection: a related word - “patients” from “patient”
• synonyms: many diseases can be expressed using similar terms, such as “adrenal insufficiency” for “Addison’s disease”
• polysemy: where one word has many meanings depending on the context - “discharge from a wound” is different to “discharging a patient”
• acronyms: vary between speciality and by geography (e.g. US abbreviations are different to Australian abbreviations)
• spelling errors: are often present in clinical notes - up to 10% of terms are mis-spelled
• negation: clinicians’ diagnostic process is often to rule things out, so negation is important. Negation can be implicit or explicit.
• uncertainty: diagnosticians are often equivocal about their diagnoses - e.g. “tentative” or “probably”. Uncertainty exists on a spectrum between certainly positive to certainly negative so it’s not a binary classification problem.
• temporality: did the patient previously have a condition or are they currently presenting with symptoms or the condition?
• discourse structure: clinical notes often follow an anatomical location of the body, but some sections have more weight
• co-reference: when is a nodule a tumour?

Clinical data is difficult to come by - “clinical text is precious”. There are limited public datasets. Data work typically done by people embedded in health. Commercial datasets such as MIMIC are
available but they are expensive.

This mirrors my experience with speech data - high quality, high volume data is hard to come by.

Clinical NLP in practice

Dr Vlada Rosova, Research Fellow, Applied ML

Dr Rosova provided a walkthrough of her postdoc project, which is to detect invasive fungal infections in immuno-compromised patients with blood cancers based on histopathology notes. Fungal infections in this cohort can be deadly, and her work alerts clinicians to the possibility
of fungal infection.

She walked through the data preparation step, in which 283 reports were coded by clinicians using the BRAT rapid annotation tool. I hadn’t come across this tool before (we use Elan in speech recognition), and I noted that it was open source, but its last release was 2012. Some of the important features in the text were explicit negation “no presence of fungal spores”, named entities “Aspergillus” and stains used for specimen analysis “Grocott”. From this text, several concept categories were identified which were mutually exclusive.

Then, a range of tools were used to train models, and then assessed for their accuracy in this binary classification (“is fungus present?”) task. The dictionary tools have poor performance if the training set is too small, and the lexical diversity is high, but they are easy to interpret. The next model was a conditional random fields model, which allows contextual features to be incorporated, which helps with the challenge of polysemy. The types of features that were used in the CRF model were start and end character positions, capitalisation patterns, morphological patterns and parts of speech (POS) tags. The POS tags were not that helpful as a feature, which I found very surprising. CRF is a good predictor if the data items have a consistent linguistic structure, but poor if the attributes used are not predictive.

Dr Rosova used this image from Wikipedia to explain the hierarchical levels of meaning in
language:

• Pragmatics - meaning in context
• Semantics - literal meaning
• Syntax - phrases and sentences
• Morphology - words
• Phonology - phonemes
• Phonetics - speech sounds

NLP operates at a morphology level, but what they are trying to provide is pragmatic advice. So there’s a shift in the level of how language operates.

This links to concepts such as datafication and vectorisation and the work of Xiaochang Li, who talks a lot about voice and speech being turned into data, and how that changes how we view language.

Li, X., 2023. “There’s no data like more data” automatic speech recognition and the making of algorithmic culture. Osiris 38, 165–182.

Next, transformer models were used, which handles semantic context much better because of the use of attention in transformer models. There are many pre-trained transformer models for the clinical setting, including BioBERT, PubMedBERT, ClinicalBERT and DischargeSummaryBERT. The key question for the practitioner is “which flavour of model best fits your use case”. Using the transformer approach provided the best accuracy because it was the most flexible.

The other important takeaway here was the need for data preparation to be methodical - you have to prepare you data, e.g. tokenisation, in exactly the same way as the transformer model you are using.

Side note: I really liked the way that Dr Rosova used the slide functionality in Jupyter notebook to turn her Jupyter notebook into slides. I dug into this after her talk, and figured out how to do it, and it would be a very useful way to present a tutorial. Jupyter Lab exports the notebook to HTML, which is rendered through reveal.js.

Talk highlights - Day 2 - Main conference - Thursday 30th November

Keynote - Professor Reza Haffari - Towards Effective NLP Systems: Cultural

Norms, Explainability, and Reasoning Enhancement

Professor Reza Haffari, Monash University

Prof Haffari outlined some challenges that are still open for
NLP:

• To be capable of reasoning and communicating with humans in
  context
• Recognising and reasoning with cultural norms
• Reasoning with reduced hallucination

In particular, his research focuses on NLP for cross-cultural settings, where we want to prevent violations of politeness and social norms. His research lab has been working on dialogue assistance for cross-cultural settings that provides three key functions:

• detection of norm violation
• an explanation of the violation
• remediations

The overarching goal is to assist in recognising cultural misunderstandings for improved conversations. This presents several engineering problems - it has to be memory and compute efficient, and it has to have low cognitive load for the user. This work provides a social, cultural and emotional analysis on top of speech recognition or speech synthesis.

“Social-cultural norms are guiding principles for personal conduct in social interactions. They promote healthy relationships, effective communication and social integration” - I thought this was an excellent definition of social-cultural norms.

A lack of annotated data impedes the development of systems for socially-aware dialogues. Evaluation benchmarks are needed in this space. Again, we see the need for more annotated data, and how a lack of data impedes research goals. Annotation is complex and high quality benchmarks are crucial - creating datasets can be rewarding but takes high effort. Part of Prof Haffari’s team’s work has been to create the SocialDial benchmark dataset which contains 1536 multi-turn dialogues created by humans, augmented by 4870 dialogues generated by ChatGPT. The authors use a DialogueFrame approach which captures variables describing the context of the dialogue, including social norms and rules of interest, the distance / relationship of participants, and the topic and location.

Zhan, H., Li, Z., Wang, Y., Luo, L., Feng, T., Kang, X., Hua, Y., Qu, L., Soon, L.K., Sharma, S. and Zukerman, I., 2023. SocialDial: A Benchmark for Socially-Aware Dialogue Systems. arXiv preprint arXiv:2304.12026.

It draws on work by Hovy & Yang around the social factors that are important in language.

Hovy, D. and Yang, D., 2021, June. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 588-602).

Prof Haffari outlined the data processing pipeline, which involved human annotation by trained annotators, which incurred a high costs. The synthetic data was cheaper and scalable, but very noisy. Ratings distributions were used for different social and norm factors to demonstrate inter-annotator agreement. The synthetic data generated by the LLM had generally high agreement.

This was used to train a model for detection of utterance level norm validation (an utterance in speech recognition is a spoken phrase).

Moghimifar, F., Qu, S., Wu, T., Li, Y.F. and Haffari, G., 2023.
NormMark: A Weakly Supervised Markov Model for Socio-cultural Norm
Discovery. arXiv preprint arXiv:2305.16598.

norms and respect were used as the two vectors - so whether the norm was violated, and the level of violation such as “minorly disrespectful” or “egregiously disrespectful”.

Here, norms were used for reasoning about a situation. The team was bottle-necked by the need for a rule base. The rule base can be extracted using LLMs. Socio-cultural norm statements were created for prompting the LLM, and then the generated phrases were used to help refine the reasoning of the LLM. The team studies the role of training data for studying LLMs, in areas such as the leakage of training data to the test set, and the effect of frequency on the learned representations and emerging behaviour. The Koala index shows the k-gram overlap in the test and training set.

In NLP, k-gram is a sequence of k words, so a bi-gram is a two-word sequence like “adorable corgi” while a trigram is a three-word sequence such as “very adorable corgi”. If k-gram overlap between the training set and test set is high, then it may indicate leakage of training data to the test set.

I also had a though here about k-gram overlap: could it help us identify training data for models where the training data is opaque, such as Whisper? Not sure how to apply it …

Vu, T.T., He, X., Haffari, G. and Shareghi, E., 2023. Koala: An Index for Quantifying Overlaps with Pre-training Corpora. arXiv preprint arXiv:2303.14770.

Next, the team used knowledge graphs so that the LLM could do knowledge verification. This was evaluated using several datasets such as Google-RE question answer set and the T-REx dataset.

LLMs lack up to date knowledge and experience hallucination during reasoning. Knowledge graphs contain a reliable source of knowledge. Theythen followed a planning-retrieval-reasoning process, where the question qas converted to a sequence of knowledge graph relations, so it was “grounded” in the knowledge graph. It could be grounded on multiple paths in the graph. The reasoning combined the results to provide explainable, interpretable explanations of rule violation and possible remediations (such as a suggested alternate phrase).

Future directions of the work include more multicultural / more multi-lingual approaches (this work was just Chinese and English), more work with sub-cultures and fine-grained context, incorporating visual or emotional behaviour. Encoding for age or gender norms is future work.

A neuro-symbolic approach capitalises on the strength of each approach (stochastic & deterministic) to improve learning.

Keynote - Professor Heng Ji, SmartBook: An AI Prophetess for Disaster Reporting

and Forecasting

Professor Heng Ji, University of Illinois at Urbana

Professor Ji introduced her talk by quoting from Dickens - it is the best of times and worst of times, it is the time of wisdom and the time of foolishness. She linked this to the rise of LLMs - which produce text sequences word by word by computing sequences. They are one size fits all models - applied to hundreds of tasks. The models have high costs, and both close some research avenues and open up new ones. LLMs have not caught up with the state of the art (SOTA) in more complex tasks yet.

The specific project Professor Ji was speaking to was situation report generation - understanding complex, unfolding events to extract the common truth, across heterogenous data, to guide action planning, such as disaster relief or humanitarian response. Currently, production of this type of reporting is manual and slow, and the information rapidly becomes outdated as the conflict or crisis situation evolves.

The concept of the SmartBook software is to provide information summaries that are grounded on to
known truthful data sources. GPT-4 cannot give an evidence basis for the content it generates. Each chapter of the SmartBook makes a set of claims that are grounded on to an evidence basis, to allow the analyst to assess the veracity of the claim (for example, if a source’s integrity changes over time). The claims are represented using a structure called a “claim frame”, which I had not come across before. SmartBook provides a timeline of events by clustering news articles. This is challenging because it’s open domain information extraction.

Sun, C., Li, J., Fung, Y.R., Chan, H.P., Abdelzaher, T., Zhai, C. and Ji, H., 2023. Decoding the silent majority: Inducing belief augmented social graph with large language model for response forecasting. arXiv preprint arXiv:2310.13297.

Professor Ji used the concept of tropes from storytelling to illustrate how many stories fit into a common narrative pattern, and used this to show how the StoryBook is structured; different types of situations have different narrative patterns, with key events that have a temporal relationship - events happen in a sequence. The StoryBook makes use of a schema, and this harks back to “old AI approaches” - such as the representation of causal events using scripts per Schank and Abelson:

Schank, R.C. and Abelson, R.P., 1975. Scripts, plans and goals. In
Proceedings of the 4th International Joint Conference on Artificial
Intelligence. IJCAI (Vol. 1).

Here, Schank and Abelson define a script as “a predetermined, stereotyped sequence of actions” - that can be learned to represent a set of causally-related events.

A key critique I have of this approach is “how do you know which stereotype applies to the situation you’re analysing?”. For example, the invasion of Ukraine by Russia followed a different playbook to the
guerilla warfare tactics employed by the Viet Cong in North Vietnam, and the playbook for responding to the Haitian earthquake crisis is different again. How do you know what type of problem or situation
you’re dealing with? How we frame problems often drives how we approach intervening in those problems, so if you use an inappropriate stereotype or script, you could be intervening in harmful ways. I see links here to the Cynefin model used to support decision making - that is, you have to recognise the type of problem you’re dealing with. I think what they were doing here was using the LLM to help generate the schema. If that’s the case, then how do you “flip the narrative” or “do something differently” to what has been done before? I think this approach has the potential to reinforce interventions that may not have worked before …

I also found it really interesting that we’re pulling from 1970s material - so much of this is still relevant for knowledge representation.

To identify the schema for a SmartBook, the team applied open domain hierarchical schema induction - there is implicit knowledge hidden in LLMs. By using retrieval-augmented prompting, they were able to generate schemas from the LLM. The schema made use of Allen’s base relations - used for reasoning over temporal logic:

Allen, J.F. and Ferguson, G., 1994. Actions and events in interval
temporal logic. Journal of logic and computation, 4(5), pp.531-579.

Professor Ji gave the example of creating an “Epidemic Schema” for disease outbreak, where the key events of the schema - identification, treatment, death, recovery, could be represented as a knowledge graph. The schema was built using a method called enhanced information extraction. Traditional information extraction is token-based, but enhanced information extraction uses the schema (which is a systems representation) to ground the information extraction structurally - the information extraction can reason over the schema to validate the claims its making.

LLMs can mess up the causal relationships when they do generation - they get the entities correct, but they frequently get the relationships between entities incorrect. Professor Ji gave the example of a picture with a soldier holding a baby, ostensibly rescuing it, being followed by a service dog. The LLM infers “soldier feeding a baby to a dog”. The entities are correctly recognised, but their causal relationships are not. The schema helps the LLM to correctly infer causal relationships.

Side note: I am reminded here of a case study in data visualisation from the American conflict in Afghanistan where a data visualisation - called the “hairball graphic” - was used to represent
the conflict. It was developed by General McChrystal, and it showed the complexity of the conflict - and the challenge in intervening link to NYT article. At what level of complexity does the schema stop? Where do you draw schema boundaries?

In her conclusion, Professor Ji held that the field of NLP is still very exciting - LLMs offer as many new research directions as those they close off. She is particularly interested in working with researchers from other disciplines. She noted that there is a need for more global computing resource infrastructures to develop academia-owned, really open source LLMs (preach!), where you have 100% access to the data and the code for training, otherwise academia will be even more dependent on LLM producer companies in order to conduct research.

My key critique of this work (and I’m conscious that I’m not qualified to critique a professor) is that it did not address the “building castles on sand” argument. That is, we’re building decision making tools, and even if their factual basis is enhanced using retrieval augmented generation capabilities - grounding on to knowledge models - there is significant potential for them to be poisoned, particularly in an information warfare scenario. There wasn’t anything in this talk about disinformation or about how to make the system robust to content poisoning from threat actors, who would be motivated to shape the data that was being summarised.

Talk highlights - Day 3 - Main conference - Friday 1st December

Panel Discussion [Panellists: Long Duong, Reza Haffari, Ed Hovy and Karin Verspoor]

This panel was moderated by Jey Han Lau, with senior people from the community responding to questions posed by Jey Han. I have synthesized the panel responses.

What are the biggest technical problems of LLMs?

• Semantics. LLMs are next word prediction machines. They don’t have meaning - semantics - and can’t reason.
• High stakes consequences and accountability. This is important for high stakes consequences - where actors are making decisions or taking actions based on LLM outputs. There’s a difference between explainability, verifiability and interpretability. However, they are getting better - pretty soon you will get to reasoning and explanations, which gives you interpretability. This are in principle solvable with more layers. Who’s in the wrong if an LLm gives false information and it’s acted upon?
• Cost vs benefit. They have a very high financial and environmental cost. Do you need GPT-5? What cost does the new capability require? There is also a cost of maintenance - updating knowledge and preventing unintended consequences.
• Capability vs scalability. Capability doesn’t necessarily depend on scalability. What allows you to do better reasoning without scaling (_read: retrieval augmented generation_). We can’t scale the current paradigm, we need a different paradigm. How do we change the paradigm?

How should academics position themselves in NLP? Especially cf. the resources of Big Tech?

• Novelty - the ACL community is always looking for new tasks, new data (_oh look, data again!_), new fields. We need to compete in novel areas, such as low resource languages (_actually models like No Language Left Behind are in this space now too_).
• Collaboration rather than competition between industry and academia - even if you treat models as a black box, there’s a lot you can do. Collaboration is also important for openness. The models from OpenAI are not open, but models like LLAMA are available. How do I build a model myself? How do you embed the different layers into the model for different forms of knowledge representation?
• Benchmarking and evaluation - there’s a lot of work still to be done in evaluation and benchmarking.
• Simple solutions - often we don’t need big models when a simple model will do.
• Shifting representations - Every time there’s a shift in representation - e.g. to vectorising everything, then we had Hidden Markov Models, they went from speech into information retrieval. Now everyone is speaking about _embeddings_ as a form of representation. We talk the same language, which allows us to do cross model integration (_oh, and to collaborate across disciplines, maybe?_)
• Smaller, more personalised models - but this raises privacy implications. How much do you want a model to know about you. They are becoming embedded in daily life.
• What do you become - lots of discussion here about “putting it all together” - the systems thinking, the orchestration. NLP is becoming a commodity - it’s becoming more generalised. We are giving machines the power - so things like translation are now the domain of machines rather than humans.
• What is the role of humans in the loop and the consequence of having rewards in LLM - How do you reward a thing? (_this was an epistemic discussion_) To reward something, it must have an internal goal system, a teleology. An LLM today has none of that. If you give an LLM a goal structure - if you achieve your goal, you will have _more_ of something - that’s rewardable. It’s scary - it’s one more step toward human level cognition. In reinforcement learning, there have been reward functions for a long time - we think of models as `agents`. You have to read through the hype. You have to make goals _verifiable_. We need to perform assurance on LLMs - simulation, formal verification.

Keynote - Dr Terrence Szymanski - Commoditisation of NLP in industry

Dr Terrence Szymanski, Principal Data Scientist at Seek

This was the highlight of the schedule in my opinion - fantastic speaker.

There are 2 million developers building with OpenAI’s tools like ChatGPT - they are not traditional NLP people. The number of ACL submissions has increased four-fold in recent years. NLP has become commodotised - there are new research developments, more use cases for applied NLP, increased interest and a better ecosystem of NLP tools.

Commodotisation means things are widely available; they are mass market produced. They are fungible, interchangeable. Even the commodoty pricing structures between cloud providers are similar. Similar patterns of commodotisation were seen in internet adoption, machine learning. IT is commodotised, it is no longer as strategically important - it is considered table stakes. AI is becoming table stakes, it is becoming infrastructure that will be taken for granted.

There are essentially two groups in the market - NLP producers like OpenAI and Anthropic who make the NLP models available for consumption, and consumers who use the available NLP models, like GPT. Each presents different opportunities. 75-80% of Australian companies haven’t adopted AI as yet.

Commodotisation is _not_ democratisation. People are trying to lock up the models behind paywalls, and use the threat of regulation of open source to dissuade open source models.

There are two options for companies looking to adopt NLP tools - build it or buy it. The tools available to build your own have expanded - many of which started as research tools and have now been released as open source software. On the buy side, tools have evolved from proprietary tools to software as a service.

T-shaped individuals are needed for cross-functional teams - product managers, data scientists, engineers. What happens if the data scientists go away because you don’t need them to train models anymore? There will always be jobs for good data scientists, because we do a lot more than train models. Jobs will change. But over-simplifying things - for example with low-code and no-code solutions, comes with risks too. The appeal of commodity LLMs is their ease of use - but there are unintended consequences - such as the doinks incident where the Australia Post chatbot gave an unexpected response.

Dr Szymanski covered several use cases of NLP used at SEEK, such as

• Search recommendation and matching
• Job search that needs to disambiguate location from job title - e.g. “Research Officer” and Research and Officer are both suburbs of Melbourne
• Related searches
• Matching and scoring jobs to candidates
• Profile search with an antiquated role to category taxonomy
• Category suggestions for job posters
• Content moderation - e.g. a job ad that contains “Must be female less than 30 years old” - possible legislative breach
• Job ad summarisation. This is very dependent on the domain the job is in - so you need to be across your data. Automated tools may not summarise correctly.
• Skill suggestions - need named entity recognition to know that “Python” and “Pandas” are similar skillsets.
• Cookie-cutter NLP approaches won’t work well - it needs to be tailored to the specific use case.

In terms of how NLP people can respond, you need to be aware that over time, the build skillset - building models - will be less in demand. Classic NLP methods are still useful, because they complement the LLM / deep learning approaches.

LLMs are bleeding into IT. If LLMs read all of GitHub, then they can generate code. For databases or for information security they can generate code for detecting anomalies. Is that the future of programming - via prompt? It won’t replace analysts and engineers. You might see fine-tuned LLMs trained on internal code repositories so that it adopts the organisation’s internal style.

In terms of making the buy versus build decision, you need to be clear about what you’re getting out of the model. You might be better of with an engineer to stitch together some pre-trained models first - that is, use the commodity solutions first to see _where_ you need to build.

Keynote - Professor Heng Ji - AI Scientist - Knowledge Empowered Joint Large and Molecule Modelling to accelerate drug discovery

Scientists are looking for molecules in a large space - 166 billion small molecules and 970 mllion are drug like. Cannot do a manual search of this space. It costs $USD 1.3 billion to bring a new drug to market. Need to democratise drug discovery.

At the moment, AI for science doesn’t include chemists and doctors - they are doing benchmark work rather than doing discovery work. There are over 500,000 papers published in PubMed each year - it is impossible for a human to read them all. There is a lot of scope for precision medicine in AI. How can we use knowledge to help doctors?

Using the example of kinase inhibitors, which can be turned into drugs to stop the spread of cancer, Professor Ji explained how the structure of the molecule needs to be understood to predict the toxicity of the compound. New molecules can be assembled overnight, but the barrier is in _finding_ the right molecules first. Many kinase inhibitors cannot pass the blood-brain barrier, and are therefore less effective. Using already approved moelecules, can you identify new molecules with desirable properties?

The key challenges in this work were about representing the chemical structures. Much of the knowledge around these structures is tacit, not available in a written form. They used a flattened graph structure for a graph neural network. The molecule structure does not encode properties. The model they trained to identify properties was very close to ground truth. The model was augmented with a knowledge parser. The long context of the molecules was a challenge - authors write long sentences. Using LLMs for this task was challenges - if you have too many tokens under attention you have an increase in attention entropy (which reduces the accuracy of the word embeddings).

In general, there is a need for more multi-disciplinary work here, and Professor Ji would like to move towards an AI-automated science lab. The challenges here are the availability of high quality data (data again!) and high quality papers to train from.

Universities have been challenged, since the inception of Facebook, Twitter and other platforms in the late 2000s to mid-2010s, to understand how to adopt, integrate and govern their intersection with institutional activities - such as learning, promotion, student engagement and research impact¹. An early paper in this space by then-Deakin academic, Stuart Palmer, used innovative data visualisation techniques to demonstrate how different types of institutions engaged their stakeholders via Twitter, showing varying patterns of use ².

Over the intervening decade we have witnessed increasing maturity and sophistication in how social media is wielded by universities³. Social media is now tightly integrated with analytics platforms that track student recruitment and retention, interwoven with departmental KPIs for objectives such as “impact” and “engagement”, and even used as an argument to advocate for academic promotion (for example, Professor Inger Mewburn at the Australian National University has written about how her social media presence as the Thesis Whisperer helped her career progress).

While I can’t find a lot of empirical work in this space, the work that does exist suggests that the choice of which social media platforms upon which to build a presence is driven largely through its ability to attract and recruit new students ⁴; particularly lucrative international students - given that Commonwealth Supported Places, in general, don’t generate financial surpluses for universities (for example, see the Universities Australia Higher Education: Facts and Figures, 2020, p.14. for a run-down of how the value of a Commonwealth Supported Place for a university has changed over time). This is why universities have accounts on social media platforms popular in countries whose students frequently choose Australia as a tertiary education destination - many universities have a Weibo account (for example, here’s Deakin’s official Weibo account). It is also why universities have “followed their students” onto platforms such as TikTok and Snapchat, disregarding potential security implications (e.g. the University of South Australia).

So why aren’t Australian universities moving to Mastodon?

In late 2022, with the Elonificaton on Twitter, many former users of the platform made the move to Mastodon. I’ve written previously about the characteristics of Mastodon - and in particular, its decentralised nature. Mastodon is not a single platform. It’s a federation of servers, each with its own particular rules, and criteria for who can register an account on a server. Amid the mass migration to Mastodon from Twitter - dubbed the #TwitterMigration - many “brand” accounts - accounts representing an organisation or company - found themselves without a home, as server administrators restricted who could create an account, and, understandably, baulked at providing free infrastructure for the purposes of brand advancement.

This was a key motivation for me to create the Mastodon server, Fediverse.AU, with the explicit purpose of providing a home for Australian university and research organisations without a home in the Fediverse. I haven’t heavily promoted this server, and that’s not the purpose of this post. It’s not listed in the official Mastodon directory of servers, primarily because it doesn’t meet one of the criteria of the Mastodon server covenant; I haven’t set up emergency access for another person (yet). But I had benefited significantly from having a presence on Twitter - connecting with others, sharing research and leads, and amplifying content useful to me and to others with similar interests. And I knew the value of networks; that the connectivity of a social network yields benefits beyond its components - emergent properties ⁵.

At the time of writing, very few Australian universities have an official presence on Twitter rival, Mastodon. There are exceptions; the ANU has “reserved” equivalent names for many of its social accounts on the Mastodon server mastodon.social(this is a poor choice of server - the reasons why are beyond the remit of this post - essentially, that host is managed out of Germany by Mastodon company founder and key software contributor Eugen Rochko, subject to German law) but as of writing, only the Research at ANU account was actively posting. Some smaller sub-organisations have an account on Fediverse.AU - I feel it’s wrong to promote them here, but I have accounts for groups out of UNSW, University of Melbourne, ANU, and a cross-institutional ARC centre.

So why haven’t more university and research groups flocked to Mastodon?

Without empirical work, I can’t answer with any certainty - so what follows is speculation.

The future students aren’t there

Above, I wrote that social media channel choice had been guided by recruitment and engagement efforts targeted to students. In my interactions on Mastodon, I’ve observed that there are many academics, and aspiring academics (postgrads, postdocs etc). But there are fewer undergraduate, or pre-undergraduate folx. So Mastodon is a poor choice to target brand awareness and marketing efforts, which are intended for student recruitment.

The future students, simply put, aren’t there. And university marketing and social media activities are oriented to student recruitment.

Understanding of Mastodon federation and server infrastructure

I also believe that universities don’t have a strong understanding of how Mastodon as a platform works. There are two key choices in where to have an organisational account; the organisation can choose to have an account on an existing server, or to deploy their own server infrastructure. mastodon.social is the default choice , the “easy” choice when searching for Mastodon servers to join - the result of choice architecture employed by Eugen Rochko (known as @gargron on the platform).

To be fair, I can see why using an existing server is an attractive choice. Setting up a Mastodon server, while generally straightforward, and while offering the organisation the highest level of control over the rules of the server and any integrations it may have, still requires considerable manual configuration. Over the years, many Australian university IT organisations have outsourced their functions to vendors. They no longer host email. They no longer host many of their SaaS applications (ever searched for a job at university lately? All of them use PageUp, hosted externally and branded to look like the university, using a combination of F5 and reverse proxying). So why would they choose to host their own Mastodon server? Universities that are interested in this option will be wanting their preferred vendors to provide a turnkey solution. They don’t want the responsibility of hosting - and patching - their own infrastructure (the TootRoot vulnerability is a case in point).

As an additional disincentive, the documentation for administering a Mastodon server is better than it used to be, but it’s still not great.

There is a major danger I see here in this lack of understanding - and that’s in the domain names which are emerging as a de facto standard for Mastodon. In general, organisations are adopting a something.social or a social.organisation.org pattern for their Mastodon presence. Many scholars are on servers named scholar.social, hci.social and sigmoid.social. As the .social domain becomes more closely associated with Mastodon servers, it’s also become prime domain name property - the .social TLD has relatively few restrictions on who can register a domain name, unlike the heavily regulated .edu.au namespace (well, except that kindergartens can get an .edu.au name, but that’s a different rant). Australian universities have not, in general, thought to register their .social equivalent as a brand protection measure. This is a risk, and adds to the integrity challenges universities already face with their brand being impersonated online.

Side note: if anyone from ANU wants the anu.social domain, I’m happy to give it to you for free on the proviso you actually host a Mastodon server with it. In the meantime, I promise I won’t try to impersonate you and drop malware from it, because, well, it’s been done at ANU before 🔥

Moderation

A major challenge with hosting a Mastodon server is the need to moderate content, users and imagery. Recent proposed legislation by the Australian government would vest additional powers in the Australian Communications and Media Authority (ACMA) to combat disinformation. While this is a positive step, it places additional onus on digital platform providers - here, Mastodon server administrators - to undertake additional record keeping efforts to show how disinformation is being detected and remediated. As a volunteer Mastodon administrator, this is additional work I’m not paid for - and for universities, they will not want to take on board this requirement.

Moreover, universities don’t want to undertake content moderation tasks, as these connote very little benefit while presenting a workload - that all Australian universities in a post-COVID era - cannot justify. Registering on another, external Mastodon server, negates this need. But this simply shifts the onus - on to the (usually volunteer) moderators on the Mastodon server on which the university has an account.

The choice of a Mastodon server on which to establish a presence is crucial to the integrity of an organisation’s brand; if your organisation is in the same digital space as homophobic, transphobic or otherwise discriminatory content, it reflects negatively on your brand. But someone has to do that moderation work.

There are many other platforms to invest in

Currently, many academics are on Mastodon, and their interactions are with each other. There are already platforms that seek to facilitate (and, let’s be honest, monetise) those interactions - ResearchGate, Academia.edu, ORCID, Google Scholar. Why would universities invest in Mastodon when they could turn their efforts to making sure their academics have comprehensive profiles on these platforms instead?

Addendum April 2024: You can’t tell when a Mastodon server has referred traffic to your own website

As outlined in this GitHub Issue by Jim Killock, URLs published via Mastodon do not contain any referrer information as a privacy measure. This means that the Mastodon server does not show up in the Referrer information on web stats programs, such as Google Analytics or AWstats. While this preserves the privacy of the Mastodon server, it makes it difficult for the web server owner to understand how much traffic comes to the server from Mastodon, as Mastodon referrals are all counted as “direct hits” (because they don’t have a referrer).

When universities ask the question “is it worth my social media person’s time investing in creating content for this platform”, they will usually use web stats to answer this question. If an organisation is not aware of this detail, the rational response here is “no” - because to web stats, it looks like not a single visit to your organisation’s website was referred from a Mastodon server. In reality, what the organisation should be looking for is an increase in “direct hits” when they publish content to a Mastodon server.

The other option here is to use a UTM (universal tag manager) code specifically for Mastodon content - for example using a URL like https://example.com/my-content?UTM=Mastodon. Google Analytics should then register Mastodon as a specific traffic source. However, this means creating separate URLs for content posted to Mastodon - and this leads into another recent issue:

Addendum April 2024: Social media automation tools are not as well integrated with Mastodon as other platforms

Most universities will use automated tools - such as Tweetdeck, Sprout Social or Loomly - to streamline their social media operations, including analytics. Most of these tools do not yet support Mastodon, although Mastodon has an openly available API. The exception appears to be Buffer, which has recently announced support for scheduling Mastodon posts.

In the meantime, I have been using this Mastodon Scheduler tool by David Libeau, however you cannot schedule images with it unfortunately.

What should universities do about Mastodon right now?

Even if universities aren’t ready to establish a presence on Mastodon yet, there are some activities I believe they should be doing to ensure the integrity of their brands.

• Register their .social equivalent - primarily so others can’t register it and use it for nefarious purposes.
• Provide the ability for academics to list their Mastodon handle on departmental websites - in the same way that most CMSs allow Twitter or LinkedIn handles to be published
• Monitor the landscape - understand which academic organisations are on Mastodon, and what content appears to be resonating with the Mastodon audience. For example, a recent Toot (the Mastodon equivalent of a Tweet) by Professor Inger Mewburn about disabling her Twitter account recently attracted over 1300 “boosts” - the equivalent of a retweet. That’s some impressive reach!
• Start to think about what sort of content strategy would work for your department or area on Mastodon. What sort of content lends itself to the platform? How would this help you reach your research and teaching goals?

Are there any other thoughts you have about universities and Mastodon? Let me know in the comments.

You might contribute data, write data specifications for collection, perform filtering or pre-processing, train ASR or TTS models, or design or perform evaluations on ML speech models.

If so, I’d love your help to understand current dataset documentation practices, and what we can do to make them better as part of my PhD research at Australian National University’s School of Cybernetics.

The survey takes 10-20 minutes to complete, and you can opt in to win one of 3 gift cards valued at $AUD 50 each.

Research Protocol 2021/427 approved by ANU Human Research Ethics Committee

https://anu.au1.qualtrics.com/jfe/form/SV_cSFODa5osYtm96e

Why EverythingOpen?

EverythingOpen is a re-imagining of linux.conf.au. linux.conf.au - affectionately known as LCA - started life in 1999 - as the Conference of Australian Linux Users (CALU), and has been a deeply-technical fixture on the Australian open source calendar since. There have been deliberate moves (which, for transparency, I strongly support) to widen both the audience of the conference, and the pool of volunteers which delivers it. EverythingOpen, the brainchild of long-term Linux Australia leaders Sae Ra Germaine and Joel Addison - aims to connect the many open* communities in Australia together.

There are open source communities - those who create software free for everyone to use, and enable that software to have impact (with thanks to Donna Benjamin for her work on enablement) by building communities, resources and other artefacts to bring software to life.

There are open hardware communities - Jon Oxer, Andy ‘Geekscape’ Gelme and Hack Melbourne - and their incredible work delivering the Open Hardware Miniconf at LCA for nearly a decade with projects such as the Swag Badge and the Dingo Car.

There are open data and open gov communities - Pia Andrews’ work here comes to mind in her founding of the GovHack movement, which now has a life of its own; Fiona Tweedie’s work with Open Knowledge, the work of Liz Stokes and Sara King and others who work with open research data and the researchers who need the skills to work with that data.

There are open security and open telemetry communities, bringing the ethos of open data and open source to the prescient problem of infosec and cyber security; Cameron Tudball’s talk in this space is definitely in my favourites list.

There are the open GLAM communities - Galleries, Libraries, Archives and Museums who often use, create or adapt open source tools into their curation and collection processes. The work here of Hugh Rundle, Ebe Kartus and the many good folx of VALA comes to mind.

There are open science communities - researchers, infrastructure specialists, high performance computing (HPC) experts - who increasingly use the power of machine learning and artificial intelligence to make new discoveries and advances - but who need a wealth of expertise in software, hardware and infrastructure to do so - the work of Kiowa Scott-Hurley

There are open education communities - educational developers, digital literacy specialists, knowledge management people - who create resources to educate, to inform, and to share learnings.

EverythingOpen is glue: helping these communities connect, collaborate, and converge.

Theme: Supporting our students, researchers and educational developers to create new knowledge and make discoveries (#OpenScience, #OpenData, #OER, #OpenEducation)

Liz Stokes, Australian Research Data Commons - Hammer Time: Building Carpentries community in Australia

Liz’ talk introduced The Carpentries - a volunteer, membership-funded collective which aims to teach foundational coding and data science skills to research communities worldwide, using a train-the-trainer approach, and engaging with the social factors of learning. In Australia, The Carpentries is supported by the Australian Research Data Commons, and the National Research Infrastructure Strategy.

As Liz explained, The Carpentries is a response to several trends:

• the increasingly computational nature of advanced research - with everything from linguistics, to pharmaceuticals, to physics, to art and design - now harnessing high performance computing (HPC) in the form of GPU clusters - to produce research outputs;
• the lack of basic computational courses offered in non-STEM courses - for example in the humanities and arts, graduates may never use tools such as Git, or Python;
• and the increasing challenging finding assistance with these tools within the academy, as budgets are squeezed - and communities of practices often don’t exist internally.

My key takeaway in this talk is just how much research infrastructure skills training takes place on a volunteer basis; universities are crying out for Carpentries volunteers - but with grad students taking on tutoring work just to stay afloat - who has time to take this additional workload on? An alternative model here would be for universities to fund Carpentries volunteers - but again, universities themselves are strapped for cash. It wouldn’t surprise me if startups spring up in this space - offering a similar “Git and Python bootcamp for $$$” style offering - which privileges these skills even further.

Another interesting aspect that Liz mentioned was the growing support for a new employment category within universities in Australia to support research software development. The skills required here often don’t fit neatly into the traditional “Academic” versus “Professional” dichotomy - instead straddling competencies of both - resulting in the inability to progress professionally in either.

Kat Cain, Deakin University Library - Open by design: co-designing and producing an OER interactive textbook

In this informative talk, Kat Cain, Library Partner at Deakin University, reported on the design and publication of an OER (open educational resource) textbook, which was designed using the Pressbooks platform. I hadn’t come across this platform previously, and discovered how it could be used to create engaging and interactive resources for courses that were being delivered in hybrid or online mode - which, given the pandemic, is likely to be a feature of the educational landscape for some time to come.

The aspects of Kat’s talk that I really enjoyed were the interdisciplinary nature of the project, and the challenges this presented - across methodologies, toolsets and in terms of project processes. The team adopted a Peopling Technologies philosophy, which was again something I hadn’t encountered before. In Kat’s words, this perspective holds that:

“People need to be positioned at the heart of any engagement with technology.”

Kat Cain, EverythingOpen

Dr Linda McIver, Australian Data Science Education Institute - Raising Heretics on a diet of Open Data

Linda has a background in computer science education, and her passion is in teaching kids about “data in the real world” - showing them how they can use critical thinking with real data to pique - and sustain- an interest in data science.

Linda’s talk highlighted some of the challenges of data science education - such as that late high school - when students are often first introduced to data science concepts - is too late to stop them dropping out of the STEM pipeline - and, moreover - that teachers have often never been taught to teach STEM - and are terrified at it.

Linda’s engaging talk relayed several examples of how students, using real-world data, were able to ask questions that had never been asked before - another proof point for the value of open data.

Theme: Representing our world in the age of generative AI (#OpenData, #OpenSource)

Peter Neish and Alex Lum - Sustainable open data using WikiData

This very informative talk by Peter and Alex covered the WikiData project - which started life as a way to create re-usable elements for Wikipedia pages across languages. WikiData creates linked open data through defining ontologies. What this means is that WikiData creates and connects structures to help represent our world. It can be used to map people - and their characteristics; places - and the places that are related to each other - and products - and the people who make or use them, and the places they’re found. These relationships have meaning - they are semantic - and can be queried to help provide answers to questions. Wikidata is an implementation of what’s known as the semantic web - a movement that’s over a decade old, but only now starting to gain momentum and recognition as its applicability to technologies like the metaverse, and augmented and virtual reality - becomes clearer.

The key implication I see from the WikiData work is actually in generative AI.

With the advent of generative AI, one of the challenges we have is to represent the relationships between objects in the world. While large language models like ChatGPT can scrape swathes of text from the web, and infer relationships in that text using attention-based transformer models, having a set of object ontologies - structures, hierarchies and mapped relationships - is a way of representing or validating relationships between entities in the world. For example, an LLM might infer that “Paris” is closely related, statistically, to “France”, but it doesn’t know that Paris is a city, that France is a nation-state, and that the relationship between the two entities is “is the capital of” (which also implies a one to one cardinality).

The challenge with LLMs at the moment is that they don’t have valid representations of relationships in the world. They can infer, statistically, what word comes next in a sequence, or generate plausible text, but we don’t trust the text they generate. Without real-world representations to validate against, we’re simply going to stop believing what ChatGPT and its kin generate.

And this is where the WikiData work has a huge role to play. By mapping relationships between entities, these can be used to validate the output from LLMs. “Is Paris the capital of France?” - using WikiData, and linked open data, the answer to this question can be validated.

Theme: Supporting our public service to deliver digital government services (#OpenGovernment, #OpenData)

With efforts in the Australian Public Service to create whole-of-government digital services in tatters with the already-poor reputation of the Digital Transformation Authority eroded further by a damning Auditors’ Report, there is a renewed opportunity to connect government efforts to broader open communities.

Pia Andrews - Creating open source legislation as code

Pia is well known in open government circles, and as a founder of the GovHack movement, and here she advocated for creating open source legislation as code - helping to build trustworthiness into government services. By having legislation as code, the legality and validity of government decisions can be verified, which in turn helps to ensure repeated, correct decision making.

You can read more about the concept of trustworthiness in government decision making in this paper, which I helped to peer-review:

@misc{andrews2022trust,
    title={A Trust Framework for Government Use of Artificial Intelligence and     Automated Decision Making},
    author={Pia Andrews and Tim de Sousa and Bruce Haefele and Matt Beard and Marcus Wigan and Abhinav Palia and Kathy Reid and Saket Narayan and Morgan Dumitru and Alex Morrison and Geoff Mason and Aurelie Jacquet},
    year={2022},
    eprint={2208.10087},
    archivePrefix={arXiv},
    primaryClass={cs.CY}
}

Parting thoughts

EverythingOpen undoubtedly delivered on its promise - it started to connect our disparate, diverse open communities of practice together, forging new relationships, and hopefully, fresh collaborations. My key concern is whether this initial impetus can be maintained; Joel and Sae Ra are both long overdue for a break! Will another group of community practitioners pick up the mantle - and help ensure Everything Open remains a fixture on the calendar? I hope so!

For full transparency, it’s important that I state that as part of my contract arrangement with Mycroft - I contracted for around 18 months between September 2017 and January 2019 - I was offered, and took up, stock options - although they’re now worthless.

The Mycroft Mark II Hardware

Unboxing

The box for the Mark II was sturdy, and well-designed, carrying forward many of the design features of its predecessor, the Mark 1: folded cardboard which provided protection and buffering for the device inside. It survived a long international journey without a scratch. Inside was a useful "Get started" card that provide Quick Start links, and a hardware warranty leaflet.

The smiley face upon opening the box was a particularly thoughtful touch. We are used to anthropomorphising voice assistants - imbuing them with human characteristics. I'm torn on this - on one hand, they perform part of the function of interacting with another human, but lack so much of the nuance and meaning that emerges from engaging with humans.

One other very pleasing aspect of the unboxing was that I didn't have to buy an Australian general power outlet adaptor. Australia uses [Type I sockets](https://en.wikipedia.org/wiki/AS/NZS_3112) - and foreign-manufactured devices often require an adaptor (thank goodness for USB). *The Australian Type I socket isn't shown here because it's connected to the Mycroft Mark II.*

Setting up the device

I found setting up the Mark II to be incredibly smooth. It ships with “Dinkum” software, which, according to the Mycroft website, is a stable operating system for the Mark II. However, it diverges significantly from the previous Mycroft Core software - meaning that Skills previously developed for previous iterations of Mycroft software - for Picroft, or the Mark I, or Mycroft on Linux - won’t work on the Mark II. I found Dinkum to be very stable - it “just worked” out of the box.

Connecting to WiFi

The Mark II uses WiFi connection technology from Pantacor. Once the Mark II had finished booting, I was advised to connect to an SSID that the Mark II was advertising using my phone or laptop, and from there, enter my WiFi credentials. I wasn’t surprised that only my 2.4GHz SSID was detected - I have a dual band router that also advertises a 5GHz SSID - and I wasn’t surprised that enterprise WiFi - WPA2 - wasn’t supported. This appears to be a common issue with IoT devices; they will readily connect to consumer grade networks, but cannot handle the additional complexity - such as MSCHAPv2 authentication - required by WPA2 networks. I suspect this presents a barrier to enterprise IoT adoption - and certainly voice assistant adoption.

Pairing the device with the backend platform

When I worked at Mycroft.AI, pairing the device with the backend platform - https://home.mycroft.ai - was one of the most frequently problematic parts of the device. This was seamless. The Mark II device displayed a pairing code on screen, along with a URL to visit, and it paired almost immediately. I was then able to set the device’s physical location and timezone.

Testing out Skills

One of the frequent criticisms of the Mark II running Dinkum software is its lack of Skills; this criticism is well founded. I can ask basic questions - which hit the Wikipedia backend - or ask for time and weather information - but anything more is unavailable. I was particularly disappointed that there wasn’t any music integration with the Mark II - which boasts dual 1.5” 5W drivers - but I can’t hold this against Mycroft.

In late 2020, Spotify revoked the API access that Mycroft was using to provide premium Spotify subscribers (Spotify’s API only works for subscribed users) with access to Spotify via Mycroft. Local music is an option, but because I use music streaming - well, apart from my now-nearly-30-year-old CD collection - this wasn’t of much use.

Hardware features

The Mark II sports some impressive hardware. There is an inbuilt camera (not that any Skills make use of it), with a physical privacy shutter.

The microphone is a 2-mic array with noise cancellation, and a physical mute switch - very important for those who like privacy - although I don't know the brand of the mic. The mute button integrated very well with the design of the touch screen - with the NeoPixel indicator taking on a red hue, and the border of the touch screen rendering in red also when hardware mute is on. [Seeed Studios](https://wiki.seeedstudio.com/) hardware are generally considered best-of-breed for embedded Linux devices, but I don't think this is a Seeed microphone.

The screen is a 4.3” IPS touchscreen, and it boasts bright, bold colours. I’m guessing it has a resolution of 800px by 400px, but don’t hold me to that. The board itself is based on the Raspberry Pi 4, and the Pi’s GPIO pins 1, 12, 13, GND and 3.3V power are exposed so that you can integrate other hardware with the device; that is, it is extensible if you’re into open hardware. There’s also a 1GB Ethernet RJ45 socket - so you are not reliant on WiFi - which was quite useful. The case itself is very sturdy, and needs a torx screwdriver to open.

SSH’ing into the device

SSH on the default port 22 is blocked and you need to use a custom port to SSH into the device - this is well documented. For security, the Mark II uses SSH security keys; requiring you to generate an SSH key, and enter this on the home.mycroft.ai platform. The key is then sent to the device, and you then SSH in to the custom port using the key. In my opinion this is far more secure than depending on passwords, however, this introduces a closer dependency on the home.mycroft.ai platform - and as we’ll see later, this closely ties the hardware to a supporting platform.

What’s under the hood?

Once SSH’d into the device, I was able to take a closer look at the mycroft-dinkum software in operation.

Voice activity detection / keyword spotter

One of the software layers in a voice assistant stack is a wake word listener or keyword spotter. This software constantly listens for spoken phrases - utterances - and classifies whether the utterance is a wake word. If the wake word listener detects a wake word, then the voice assistant listens for the next utterance, and assumes the next utterance spoken is a voice assistant command.

Previous versions of Mycroft used the Precise wake word listener, and I found it regularly had a high number of false positives - triggering when I didn’t say the “Hey Mycroft” wake word - and false negatives - not responding when I did utter the wake word. The Mark II was incredibly accurate in detecting my wake word utterances - especially considering I am female, and have an Australian accent - groups for which Precise performed poorly.

The mycroft-dinkum software uses several services. The voice service within mycroft-dinkum provides speech recognition, voice activity detection and microphone input. Unsurprisingly, because this service is constantly listening, it consumes a lot of the CPU of the device, which we can see if we run htop while ssh‘d in:

I was curious about what models were being used for wake word detection, and by looking into the GitHub code, I was able to determine that the Mark II was using the Silero VAD engine. Silero is a pre-trained model. However, there is no information available about the data it was trained on - the GitHub page doesn’t say - or the type of algorithm - such as a recurrent neural network - used to train the model.

I have a hunch - and it’s just a hunch - that Silero uses some of the Mycroft Precise wake word data that was captured via opt-in mechanisms. I was the main female contributor to that dataset - and if Silero was trained on Precise data, it would explain why it’s so accurate for my voice. But, because the provenance of the training data isn’t clear, I won’t be able to tell for sure.

Speech recognition

Finding out what is used for speech recognition was slightly more challenging. In previous versions of Mycroft software, the speech recognition layer is configurable through the mycroft-config.conf file, allowing the use of on-device speech recognition, or a preferred cloud provider. That configuration is stored in a different location under mycroft-dinkum, but I was able to find it. The STT module was set to mycroft - I’m not sure what this means, but I think it means Mark II is using the Google cloud service, anonymised through the home.mycroft.ai platform. Again, there’s a cloud - and network - dependency here.

Intent parsing and Skills

Once an STT utterance transcription has been identified by the speech recognition layer, the voice assistant needs to know how to tie the utterance to a command and pass it to a Skill for handling that command. In the Mark II, from what I could tell in the source code, that layer is provided by Padatious, a neural-network based intent parser.

The Skills range on the Mark II is very limited - you can search for weather, find out the time in different timezones - which was one of my most-used Skills, play from a pre-defined list of internet radio stations, and query Wikipedia. Limiting the range of Skills means that intent parsing is easier - because you have fewer Skills to choose between. However, passing the right query to a Skill can be problematic if your speech recognition isn’t accurate. Intent parsing for Named Entities - people, places, products, especially Indigenous-language-derived ones, worked reasonably well. Finding out the weather for “Yarrawonga”, “Geelong” and “Canberra” were all recognised correctly; “Wagga Wagga” wasn’t. Queries to Wikipedia were parsed accurately - “tell me about pierogi” (Polish derivation) and “tell me about Maryam Mirzakhani” (Persian derivation) - were both correctly identified.

Text to speech

Text to speech, or synthetic speech, is a complement to speech recognition, and takes written phrases and outputs speech as audio. For this layer, the Mark II uses the Mimic 3 engine, a TTS engine based on the ViTS algorithm. The GitHub repo for Mimic 3 doesn’t contain the original data the Mimic 3 voices were trained on, but in this blog post, developer Mike Hansen provides a technical walkthrough of how the models were trained, including the challenges of training new languages, and the phonetic and normalisation challenges that entails.

By default, the Mark II uses the UK English “Pope” Mimic 3 voice, which is a male-gendered voice. I was pleasantly surprised by this, given the industry default of gendering voice assistants as female, which is covered at length in the book The Smart Wife by Yolande Strengers and Jenny Kennedy ³. There were no Australian accents available.

I have a lot of my own (female, Australian) voice data recorded, but I didn’t want to contribute it to the Mimic 3 project. There are growing concerns around how voice and video data of people is being used to create realistic avatars - digital characters that sound, and look, like real humans - and until we have better regulation, I don’t particularly want a “Kathy” voice out in the wild - because I have no control over how it would be used. For example, I know that sex doll companies have approached voice startups wanting access to voices to put into their products. As we say in ‘Strayan: yeah, nah.

Mimic 3 works offline, and I was surprised at how fast it generated speech - there was a slight delay, but not very noticeable at all. Some of the pronunciations in the Pope voice were definitely a little off - a key issue in TTS is getting the generated voice to have the correct emphasis and prosody - but I was still pretty impressed. When I’m done with my PhD work, I’d love to train a Mimic 3 voice on my own voice data to understand the process a bit better.

Screen

Mark II uses the Plymouth splash screen manager for the touch screen visuals, and looking at the htop output, this one component uses over 16% of the available CPU. I found the visuals very well-designed and complementary to the audio information that was presented by the TTS layer.

The screen adds additional complexity and consideration for Skill designers however; not only do they have to design a pleasant voice interaction experience in a Skill , but for devices like the Mark II which are multi-modal, the Skill designer must now ensure a complementary visual experience.

Overall impressions

The Mark II packs some impressive hardware, and mature software into a solid form factor. The visual design is well integrated with the conversational design, and it’s intended to be extensible - a boon for those who like to experiment and tinker. The software layers that are used are generally matured, and predominantly run on-device, with the exception of the speech recognition layer (I think). The Skills all have a dependency on external services - Wikipedia, weather and so on. The integration with the home.mycroft.ai backend serves to protect my privacy - as long as I trust that platform - and there was no evidence in any of the digging I did that the device is “leaking” data to third party services.

These positive factors are tarnished by the high price point (listed at $USD 499, although the Kickstarter price was a lot lower at $USD 199) and the lack of Skills that integrate with other services - like Spotify. This is a device that is capable of so much more - it’s a diamond in the rough. But what would give it more polish?

And that takes me to part 2 of this post - ecosystems, infrastructures and the friction of privacy.

A critical lens on an inflection point in voice assistant technology: we are in the trough of disillusionment

It’s a difficult time for the voice assistant industry.

Amazon has recently announced huge layoffs to its Alexa voice assistant department (see write ups by Ars Technica, Business Insider, and Vox), as it struggles to find a path to monetisation for the loss-leading hardware devices that its been shipping. Google has followed this trend. This also comes off the back of an architectural change at Google where it dropped support for third party applications on its Assistant voice apps and devices, which has the effect of more tightly integrating its hardware with the Android ecosystem. Apple - the other big player in the voice assistant space, with Siri, has begun quietly laying off third-party contractors, according to reports. Baidu, the largest voice player outside of Western Europe and America, with its Xiaodu assistant, which had a $5b valuation just two short years ago, has also shed 10-15% of its staff. The outlier here is the open source Home Assistant, and its commercial sister operation, Nabu Casa, who have announced the impending launch of a voice-assistant-with-screen competitor to Google Nest Hub and to the Amazon Echo Show, the Home Assistant Yellow.

Previous predictions that there will be 8 billion voice assistant devices in use by this year now appear idealistic indeed.

Voice assistants and theories of innovation

It’s fair to say that voice assistants have reached the “trough of disillusionment”. This term belongs to Gartner and their Hype Cycle; a frame they use to plot the adoption and integration of emerging technology. The “trough of disillusionment” refers to a period in a technology’s history where:

Interest wanes as experiments and implementations fail to deliver. Producers of the technology shake out or fail. Investments continue only if the surviving providers improve their products to the satisfaction of early adopters.

Gartner Hype Cycle

The Gartner Hype Cycle is a theory of innovation; different technologies move through the Hype Cycle at varying rates, influenced by a range of factors. Other theories of innovation use divergent analytical lenses, or ascribe primacy to differing drivers and constraints on technology. The diffusion of innovation theory gave us the terms “early adopters” and “laggards”. Induced innovation ⁴ for example places emphasis on economic factors such as market demand. Evolutionary theory focuses on the search for better tools and technologies, with the market selecting - and ensuring the survival - of the best. Path dependence models valorize the role of seemingly insignificant historical changes that compound to shape a technology’s dominance or decline. The multi-level perspective blends the micro and macro levels together, showing how they interact to influence technological development. Disruptive innovation theory ⁵ takes a contingent approach; different innovations require different strategies to challenge and unseat established incumbents. Apple unseated Nokia with touch screens. Linux dominated the data centre due to higher reliability and performance. Netflix swallowed Blockbuster by leveraging increasing internet speeds for content delivery. Disruptions harnesses interacting social, economic and political developments.

I digress. What all of these views of innovation have in common, regardless of their focus - is inter-dependency of factors that influence a technology’s trajectory - of adoption, success, return on investment.

So what are the inter-dependencies in voice assistant technology that lead us to our current inflection point?

Platform and service inter-dependencies

Voice assistants are interfaces. They enable interaction with other systems. We speak to our phones to call someone. We speak to our televisions to select a movie. We speak to our car console to get directions. A voice assistant like Mycroft, or Alexa, or Google Home or Siri is a multi-purpose interface. It is reliant on connecting to other systems - systems of content, systems of knowledge, and systems of data. Wikipedia for knowledge articles, a weather API for temperature information, or a music content provider for music. These are all platform inter-dependencies.

Here, large providers have an advantage because they can vertically integrate their voice assistant. Google has Google Music, Amazon has Amazon Music, Apple has Apple Music. Mycroft has no music service, and this has no doubt played into Spotify’s decision to block Mycroft from interfacing with it. Content providers know that their content is valuable; this is why Paramount and HBO are launching their own platforms rather than selling content to Netflix.

Voice assistants need other platforms to deliver value to the end user. Apple knew this when they acquired Dark Sky and locked Android users out of the platform; although platforms like Open WeatherMap are filling this gap. We’ve seen similar content wars play out in the maps space; Google Maps is currently dominant, but Microsoft and Bing are leveraging OpenStreetMap to counter this with Map-Builder - raising the ire of open source purists in the process.

Voice assistants need content, and services, to deliver end user value.

Discovery - or identifying what Skills a voice assistant can respond to

Voice assistants are interfaces. We use myriad interfaces in our everyday lives; a steering wheel, a microwave timer, an espresso coffee maker, an oven dial, a phone app, natural language, a petrol bowser, an EV charger. I doubt you RTFM’d for them all. And here’s why: interfaces are designed to be intuitive. We intuitively know how a steering wheel works (even if, for example, you’ve never driven a manual). If you’ve used one microwave, you can probably figure out another one, even if the display is in a foreign language. Compare this with the cockpit of the Concorde - a cacophony of knobs, buttons and dials.

If you’ve used Alexa or Siri, then you could probably set a timer, or ask about something from Wikipedia. But what else can the assistant do? This is the discovery problem. We don’t know what interfaces can do, because we often don’t use all the functions an interface provides. When was the last time you dimmed the dashboard lights in your car? Or when did you last set a defrost timer on your microwave?

The same goes for voice assistants; we don’t know what they can do, and this means we don’t maximise their utility. But what if a voice assistant gave you hints about what it could do? As Katherine Latham reports in this article for the BBC, it turns out that people find that incredibly annoying; we don’t want voice assistants to interrupt us, or suggest things to us. We find that intrusive.

How, then, do we become more acquainted with a voice assistant’s capabilities?

For more information on skill discovery in voice assistants, you might find this paper interesting.

White, R. W. (2018). Skill discovery in virtual assistants. Communications of the ACM, 61(11), 106-113.

Privacy and the surveillance economy - or - who benefits from the data a voice assistant collects?

There is a clear trade-off in voice assistants between functionality and privacy. Amazon Alexa can help you order groceries - from Amazon. Google Home can read you your calendar and email - from Gmail. Frictionless, seamless access to information and agents which complete tasks for you requires sharing data with the platforms that provide those functions. This is a two-fold dependency; first the platforms that provide this functionality must be available - my vertical dependency argument from above. Secondly, you must provide personal information - preferences, a login, something trackable - to these platforms in order to receive their benefits.

I’m grossly oversimplifying his arguments here, however the well-known science and technology studies scholar, Luciano Floridi, has argued that “privacy is friction” ⁶. The way that information is organised contributes to, or impinges upon, our privacy. Voice assistants that track our requests, record our utterances, and then use this information to suggestively sell more products to us reduce friction by reducing privacy. Mireille Hildebrandt, in her book Technology and the Ends of the Law, goes one step further: voice assistants impugn our personal agency through their anticipatory nature ⁷. By predicting, or assuming our desires and needs, our ability to be reflective in our choices, to be be mindful of our activities, is eroded. Academic Shoshana Zuboff takes a broader view of these developments, theorising that we live in an age of surveillance capitalism; where the data produced through technologies which surveil us - our web browsing history, CCTV camera feeds - and yes - utterances issued to a voice assistant - become a form of capital which is traded in order to more narrowly market to us. Jathan Sadowski has argued, similarly, for the concept of datafication; when our interactions with the world become a form of capital - for trading, for investing, and for extraction.

Many of us have become accustomed to speaking with voice interfaces, and glossing over how those utterances are stored, or linked, or mined - downstream in an ecosystem. In fact, Professor Joe Turow argues this case eloquently in his book, The Voice Catchers: ⁸ by selling voice assistants at less than cost, their presence, and our interactions with them have been normalised, and backgrounded. We don’t think anything of sharing our data with the corporate platforms on which they rest. Giving personal data to a voice assistant is something we take for granted. By design. We trade the utility of frictionless access to services with the friction to privacy.

And this points to a key challenge for open source and private voice assistants - like Mycroft and like Home Assistant; in order to deliver services and content through those voice assistants, we have to give up some privacy. Mycroft handles this by abstraction; for example, the speech recognition that is done through Google’s cloud service is channelled through home.mycroft.ai - and done under a single identifier so that individual Mycroft users’ privacy is protected.

How do voice assistants overcome the tradeoff between utility and privacy?

Hardware is hard

One of my favourite books is by the developer of the almost-unheard-of open source assistant, Chumby - Bunnie Huang - The Hardware Hacker ⁹. It is a chronicle of the myriad challenges involved in designing, manufacturing, certifying and bringing to market a new consumer device. For every mention of “Chumby” in the book, you could substitute “Mycroft Mark II”. Design tradeoffs, the capital required to fund manufacturing, quality control issues, a fickle consumer market - all are present in the tale of the Mark II.

Hardware is hard.

Hardware has to be designed, tested, integrated with software through drivers and embedded libraries, it has to be certified compliant with regulations. And above all, it has to yield a profit for the hardware manufacturer to keep manufacturing the hardware. If we think about the escalating costs of the Mark II - the Chumby - and then look at how cheaply competitor devices - Alexa, Google Home - are sold for - then it becomes clear that hardware is a loss leader for ecosystem integration. I have no way to prove this, but I strongly suspect that the true cost of an Alexa or Google Home is four or five times more than what a consumer pays for it.

Voice assistants are interfaces.

And by having a voice assistant in a consumer’s home, it becomes an interface into a broader ecosystem - more closely imbricating and embroiling the customer in the manufacturer’s ecosystem. And if you can’t transform that loss leader into a recurrent revenue stream - through additional purchases, or through paid voice search, or through voice advertising revenue - then the hardware becomes a sunk cost. And you start laying off staff.

An open source voice assistant strategy is different - its selling point is in opposition to a commercial assistant - its privacy, its interoperability, its extensibility. Its lack of lock in to an ecosystem. And the Mark II is all of those things - private, interoperable and extensible. But it still hasn’t achieved product market fit, particularly at its high, true-cost price point.

How do voice assistants reconcile the cost of hardware and the ability to achieve product market fit?

Overcoming the trough of disillusionment

So how might voice assistants overcome the trough of disillusionment?

Higher utility through additional content, data sources and APIs

Voice assistant utility is a function of how many Skills the device has, how useful those Skills are to the end user, and how frequently they are used. Think about your phone. What apps do you use the most? Text messaging? Facebook? Tik Tok? What app could you not delete from your phone? Skills require two things; a data source or content source, and a developer community to build them. Open source enthusiasts may build a Skill out of curiosity or to slake a desire to be able to build a Skill, but commercial entities will only invest in Skill building if it generates revenue for a product or service. And then what does the voice assistant manufacturer do to share in that revenue? So we need to find ways to incentivise Skill development (and maintenance) as well as revenue sharing models that help support the infrastructure, Skill development - and the service or data that the Skill interfaces with. For example, Spotify understands this - and will reserve access to their highly-valued content only for business arrangement that help them generate additional revenue.

I also see governments having a role to play here - imagine for example accessing government services through your voice assistant - no sitting in queues on a phone. The French Minitel service was originally a way to help citizens access services like telephone directories, and postal information. But governments want to both streamline the development they do in-house, and control access to API information; will there be a level of comfort in opening access - and if so, who bears the cost of development?

Distinguishing voice assistants in the home from the voice assistant in your pocket (your mobile phone)

Most of us already have a voice assistant in our pocket - if we have an Android or iPhone mobile phone. So what niche does a physical voice assistant serve? One differentiator I see here is privacy; a voice assistant on a mobile phone ties you to the ecosystem of that device - to Apple, or Google or another manufacturer. Another differentiator is the context in which the voice assistant operates; few of us would want to use a voice assistant in a public or semi-public context, such as on a train or a bus or in a crowd. But a home office is semi-private; and many of us are now working from home. Is there an opportunity for a home office assistant? That isn’t tied to a mobile phone ecosystem? But thinking this through its trajectory of development, if we’re working from home, then we’re working. How will employers seek to leverage voice assistants, and does this sit in opposition to privacy? We are already seeing a backlash against the rise of workplace surveillance (itself a form of surveillance capitalism), so I think there will be barriers to employment or work-based technology being deployed on voice assistants.

Towards privacy, user agency and user choice

I’m someone who places a premium on privacy: I pay for encrypted communications technology and pay for encrypted email that isn’t harvested for advertising. I pay for services like relay that hide my email address from those who would seek to harvest it for surveillance capitalism.

But not everyone does; because privacy is friction, and we have normalised the absence of friction for the price of privacy. As we start to see models like ChatGPT and Whisper that hoover up all the public data on the internet - YouTube videos, public photographs on platforms like Flickr - I think we will start to see more public awareness of how are data is being used - and not always in our own best interests. In voice assistants, this means more safe-guarding of user data, and more protection against harnessing that data for profit.

Voice assistants also have a role to play in user agency and user choice. This means giving people choice about where intents - the commands used to activate Skills - lead. For example, if a commercial voice assistant “sells” the intent “buy washing powder” to the highest “washing powder” bidder, then this restricts user agency and user choice. So we need to think about ways that put control back in the user’s hands - or, in this case, voice. But this of course constrains the revenue generation capabilities of voice assistants.

For voice assistants to escape the trough of disillusionment, they will need to prioritise privacy, agency, utility and choice - and still find a path to revenue.

At a glance

Hardware, wearables and accessories

• My main laptop is an ASUS ROG Zephyrus G15 GA502IV (no change since last report)
• Google Pixel 4a 5G running stock Android (changed since last report)
• Mobvoi Ticwatch Pro 2020 model (changed since last report)
• Fellowes book stand - again, very light and easily portable
• Logitech C920 web cam (no change since last report)
• Stadium USB microphone (no change since last report)
• Sennheiser Momentum 3 wireless bluetooth headphones (changed since last report)

Software

• Atom with a range of plugins for writing code, thesis notes (no change since last report)
• Pandoc for document generation from MarkDown (no change since last report)
• Zotero for referencing (using Better BibTeX extension) (no change since last report)
• OneNote for Linux by @patrikx3 (no change since last report)
• Nightly edition of Firefox (no change since last report)
• Zoom (no change since last report)
• Microsoft Teams for Linux (no change since last report)
• Gogh for Linux terminal preferences (no change since last report)
• Super productivity (instead of Task Warrior) (changed since last report)
• Cuckoo Timer for Pomodoro sessions (changed since last report)
• RescueTime for time tracking (no change since last report)
• BeeMindr for commitment based goals (no change since last report)
• Mycroft as my Linux-based voice assistant (no change since last report)
• Okular as my preferred PDF reader (instead of Evince on Linux and Adobe Acrobat on Windows) (changed since last report)
• NocoDB for visual database work (changed this report)
• ObservableHQ for data visualistion (changed this report)

Techniques

• Pomodoro (no change since last report)
• Passion Planner for planning (no change since last report)
• Time blocking (used on and off, but a lot more recently)

What’s changed since the last report?

There’s very little that’s changed since my last State of My Toolchain report in 2021: I’m still doing a PhD at the Australian National University’s School of Cybernetics, and the majority of my work is researching, writing, interviewing, and working with data.

Tools for PhD work

My key tools are MaxQDA for qualitative data analysis - Windows only, unfortunately, and prone to being buggy with OneDrive. My writing workflow is done using Atom. One particularly useful tool I’ve adopted in the last year has been NocoDB - it’s an opensource alternative to visual database interfaces like Notion and AirTable, and I found it very useful - even if the front end was a little clunky. Working across Windows and Linux, I’ve settled on Okular as my preferred PDF reader and annotator - I read on average about 300-400 pages of PDF content a week, and Adobe Acrobat was buggy as hell. Okular has fine-grained annotating tools, and the interface is the same across Windows and Linux. Another tool I’ve started to use a lot this year is ObservableHQ - it’s like Jupyter notebook, but for d3.js data visualisations. Unfortunately, they’ve recently brought in a change to their pricing structure, and it’s going to cost me $USD15 a month for private notebooks - and I don’t think the price point is worth it.

Hardware and wearables

The key changes this year are a phone upgrade - my Pixel 3 screen died, and the cost to replace the screen was exorbitant - a classic example of planned obsolescence. I’ve been happy with Google’s phones - as long as I disable all the spyware voice enabled features, and settled on the Pixel 4a 5G. It’s been a great choice - clear, crisp photos, snappy processor, and excellent battery life.

After nearly four years, my Mobvoi Ticwatch Pro started suffering the “ghost touch” problem, where the touch interface started picking up non-existent taps. A factory restart didn’t solve the problem, so I got the next model up - the Ticwatch Pro 2020 - at 50% off. This wearable has been one of my favourite pieces of hardware - fast, responsive, durable - and I can’t imagine not having a smartwatch now. I’ve settled on the Flower watch face after using Pujie Black for a long time - both heavily customisable. The love Google is giving to Wear OS is telling - I have much smoother integration between phone apps and Wear OS apps than even 1-2 years ago.

After having two Plantronics Backbeat Pro headphones - one from around 2017 and the other circa 2021, both still going, but the first with a very poor battery life and battered earpads, I invested in my first pair of reasonable headphones - the Sennheiser Momentum Pro 3. The sound quality is incredible - I got them for $AUD 300 which I thought was a lot to pay for headphones, but they’ve been worth every penny - particularly when listening to speech recognition data.

With so much PhD research and typing, I found my Logitech MK240 just wasn’t what I needed - it’s a great little unit if you don’t have anything else, but it was time for a mechanical keyboard because I love expensive hobbies. After some research, and a mis-step with the far too small HuoJi Z-88, (the keypresses for linux command line tasks were horrendous) I settled on the Keychron K8 and haven’t looked back. Solid, sturdy, blue Gateron switches - it’s a dream to type on, and works well across Windows and Linux. However, on Linux it is using a Mac keyboard layout and I had to do some tweaking with a keymapper - and used keyd. My only disappointment with Keychron is the hackyness needed to get it working properly on Linux.

Productivity

My Passion Planner is still going strong, but I haven’t been as diligent as using it as a second brain as I have been in the past, and the price changes this year meant that shipping one to Australia cost me nearly $AUD 120 in total - and that’s unaffordable in the longer term - so I’m actively looking at alternatives as as Bullet Journalling. The Passion Planner is great - it’s just expensive.

I’ve also dropped Task Warrior in favour of Super Productivity this year. Task Warrior isn’t cross-platform - I can’t use it on Android, or on Windows, and thanks to MaxQDA software, I’m spending a lot more of my time in Windows. The Gothenberg Bit Factory are actively developing Task Warrior - full transparency, I’m a GitHub sponsor of theirs - but the cloud-based and cross-platform features seem to be taking a while to come to fruition.

I’m also using time-blocking a lot more, and am regularly using Cuckoo as a pomodoro timer with a PhD cohort colleague, T. We have an idea for a web app that optimises the timing of Pomodoros based on a feedback loop - but more on that next year.

Current gaps in my toolchain

Visual Git editor

In my last State of My Toolchain report, I lamented having a good Visual Git Editor. That’s been solved in Windows with GitHub’s desktop application, but as of writing the Linux variant appears to be permanently mothballed. I’m sure this has nothing to do with Microsoft buying GitHub. So I am still on the lookout for a good Linux desktop Git GUI. On the other hand, doing everything by CLI is always good practice.

Second Brain

In my last report I also mentioned having taken Huginn for a spin, and being let down at its immaturity. It doesn’t seem to have come very far. So I’m still on the lookout for “Second Brain” software - this is more than the knowledge management software in the space that tools like Roam and Obsidian occupy, but much more an organise-your-life tool. The Microsoft suite - Office, Teams, and their stablemates - are trying to fill this niche, but I want something that’s not dependent on an enterprise login. But I’ve decided to reframe this gap as a “Second Brain” gap - after reading Tiago Forte’s book on the topic.

The Fediverse

Triggered by Elon Musk’s purchase, and subsequent transformation of Twitter into a flaming dumpster fire, I’ve become re-acquainted with the Fediverse - you can find me on Mastodon here, on Pixelfed.au here, and on Bookwyrm here. However, the tooling infrastructure around the Fediverse isn’t as mature - understandably - as commercial platforms. I’m using Tusky as my Android app, and the advanced web interface. But there are a lack of hosting options for the Fediverse - I can’t find a pre-configured Digital Ocean Droplet for Mastodon, for example - and I think the next year will see some development in this space. If you’re not across Mastodon, I wrote a piece that uses cybernetic principles to compare and contrast it with Twitter.

5 years of toolchain trends

After five years of the State of My Toolchain report, I want to share some reflections on the longer-term trends that have been influential in my choice of tools.

Cross-platform availability and dropping support for Linux

I work across three main operating systems - Linux, Windows (because I have to for certain applications) and Android. The tools I use need to work seamlessly across all three. There’s been a distinct trend over the last five years for applications to start providing Linux support but then move to a “community” model or drop support altogether. Two cases in point were Pomodone - which I dropped because of its lack of Linux support, and RescueTime - which still works on Linux for me, albeit with some quirks (such as not restarting properly when the machine awakes from suspend). This is counter-intuitive given the increasing usage of Linux on the desktop. The aspiration of many Linux aficionados that the current year will be “The Year of Linux on the Desktop” is not close to fruition - but the statistics show a continued, steady rise - if small - in the number of Linux desktop users. This is understandable though - startups and small SaaS providers cannot justify supporting such a small user base. That said, they shouldn’t claim to support the operating system then drop support - as both Pomodone and RescueTime have done.

Please don’t make me change my infrastructure to work with your product

A key reason for choosing the Ticwatch Pro 2020 over other Mobvoi offerings was that the watch’s charger was the same between hardware models. I’d bought a couple of extra chargers to have handy, and didn’t want to have to buy more “spares”. This mirrors a broader issue with hardware - it has a secondary ecosystem. I don’t just need a mobile phone, I need a charger, a case, and glass screen protectors - a bunch of accessories. These are all different - they exhibit variety - a deliberate reduction in re-usability and a buffer against commodification. But in choosing hardware, one of my selection criteria is now re-usability or upgradeability - how can I re-use this hardware’s supporting infrastructure. The recent decision by Europe to standardise on USB-C is the right one.

I’m happy to pay for your product, but it has to represent value for money, or it’s gone

Several of my tools are open source - Super Productivity, NocoDB, Atom, Pandoc - and where I can, I GitHub sponsor them or provide a monetary contribution.On the whole, these pieces of software are often worth a lot more too me than the paid proprietary software I used - for example, MaxQDA is over $AUD 300 a year - predominantly because it only has one main competitor, NVIVO. I have no issue paying for software, but it has to represent value for money. If I can get the same value - or nearly equivalent - from an open source product, then I’m choosing open source. Taguette wasn’t there over MaxQDA, but Super Productivity has equivalent functionality to Pomodone. Open source products keep proprietary products competitive - and this is a great reason to invest in open source where you are able.

That’s it! Are there any products or platforms you’ve found particularly helpful? Let me know in the comments.

Because there are many stages to interviews - identifying participants, approaching them for interviews, obtaining consent, scheduling, transcription and coding - I needed a way to manage the pipeline. My PhD cohort colleagues use a combination of AirTable and Notion, but I wanted an opensource alternative (surprise!).

Identifying alternatives and choosing one to use

I did a scan of what alternatives were available simply by searching for “open source alternative to AirTable”. Some of the options I considered but discarded were:

• BaseRow: While this is open source, and built in widely adopted frameworks such as Django and Vue.js, and available in Docker and Heroku deploys, the commercial framing behind the product is very much open core. That is, there are a lot of features that are only available in the paid / premium version. I’ve worked with open core offerings before, and I’ve found that the most useful features are usually those that are behind the paywall.
• AppFlowy: While this looked really impressive, and the community behind it looked strong, the use of Flutter and Rust put me off - I’m not as familiar with either of them compared to Vue.js or Django. I also found the documentation really confusing - for example, to install the Linux version it said to “use the official package”, but it didn’t give the name of the official package. Not helpful. On this basis I ruled out AppFlowy.
• DBeaver: This tool is more aimed at people who have to work with multiple databases; it provides a using GUI over the top of the database, but is not designed to be a competitor to Notion or AirTable. I wanted something more graphically-focused, and with multiple layout styles (grid, card etc).

This left me with NoCoDB. I kicked the tyres a bit by looking at the GitHub code, and read through the documentation to get a feel for whether it was well constructed; it was. Importantly, I was able to install it on my localhost; my ethics protocol for my research method prevented it being hosted on a cloud platform.

Installation

Installation was a breeze. I set up a database in MySQL (also running locally), then git clone‘d the repo, and used npm to install the software:

git clone https://github.com/nocodb/nocodb-seed
cd nocodb-seed
npm install
npm start

nocodb uses node.js’s httpd server, and starts the application by default on port 80, so to start using it, you simply go to: http://localhost:8080/. One slightly frustrating thing is that it does require an email address and password to log in. nocodb is a commercial company - they’ve recently done a raised and are hiring - and I suspect this is part of their telemetry, even for self-hosted solutions. I run Pihole as my DNS server, and I don’t see any telemetry from nocodb in my block list, however.

Next, you need to provide nocodb with the MySQL database details that you created earlier. This creates some additional tables. nocodb then creates some base views, but at this point you are free to start creating your own.

Deciding what fields I needed to capture to be able to visualise my interview pipeline

Identifying what fields I needed to track was a case of trial and error. As I added new fields, or modified the datatypes of existing ones, nocodb was able to be easily re-synced with the underlying database schema. This makes

Identifying what fields I needed to track was a case of trial and error. As I added new fields, or modified the datatypes of existing ones, nocodb was able to be easily re-synced with the underlying database schema. This makes nocodb ideal for prototyping database structures.

In the end, I settled on the following tables and fields:

Interviewees table

• INTERVIEWEE_ID - a unique, auto-incrementing ID for each participant
• REAL_NAME - the real name of my participant (and one of the reasons this is running locally and not in the cloud)
• CODE_NAME - a code name I ascribed to each participant, as part of my Ethics Protocol
• ROLE_ID - foreign key identifier for the ROLES table.
• EMAIL_ADDRESS - what it says on the tin.
• LINKEDIN_URL - I used LinkedIn to contact several participants, and this was a way of keeping track of that information.
• HOMEPAGE_URL - the participant’s home page, if they had one. This was useful for identifying the participant’s background - part of the purposive sampling technique.
• COUNTRY_ID - foreign key identifier for the COUNTRIES table - again used for purposive sampling.
• HOW_IDENTIFIED - to identify whether people had been snowball sampled
• HAS_BEEN_CONTACTED - Boolean to flag whether the participant had been contacted
• HAS_AGREED_TO_INTERVIEW - Boolean to flag whether the participant had agreed to be interviewed
• NO_RESPONSE_AFTER_SEVERAL_ATTEMPTS - Boolean to flag whether the participant hadn’t responded to a request to interview
• HAS_DECLINED - Boolean to flag an explicit decline
• INTERVIEW_SCHEDULED - Boolean to indicate a date had been scheduled with the participant
• IS_EXPLORATORY - Boolean to indicate the interview was exploratory rather than in-depth. Having an explicit Boolean for the interview type allows me to add others if needed (while I felt that a full blown table for interview type was overkill).
• IS_INDEPTH - Boolean for the other type of interview I was conducting.
• INTERVIEWEE_DESCRIPTION - descriptive information about the participant’s background. Used to help me formulate questions relevant to the participant.
• CONSENT_RECEIVED - Boolean to flag whether the participant had provided informed consent.
• CONSENT_URL - A space to record the file location of the consent form.
• CONSENT_ALLOWS_PARTICIPATION - A flag relevant to specific type of participation in my ethics protocol, and my consent form
• CONSENT_ALLOWS_IDENTIFICATION_VIA_PARTICIPANT_CODE - A flag relevant to how participants were able to elect to be identified, as part of my ethics protocol.
• INTERVIEW_CONDUCTED - Boolean to flag that the interview had been conducted.
• TRANSCRIPT_DONE - Boolean to flag that the transcript had been created (I used an external company for this).
• TRANSCRIPT_URL - A space to record the file location of the transcript.
• TRANSCRIPT_APPROVED - Boolean to indicate the participant had reviewed and approved the transcript.
• TRANSCRIPT_APPROVED_URL - A space to record the file location of the approved transcript
• CODING_FIRST_DONE - Boolean to indicate first pass coding done
• CODING_FIRST_LINK - A space to record the file location of the first coding
• CODING_SECOND_DONE - Boolean to indicate second pass coding done
• CODING_SECOND_URL - A space to record the file location of the second coding
• NOTES - I used this field to make notes about the participant or to flag things to follow up.
• LAST_CONTACT - I used this date field so I could easily order interviewees to follow them up.
• LAST_MODIFIED - This field auto-updated on update.

Countries table

Get the SQL to replicate the database table structure

I’ve exported the table structure to SQL in case you want to use it for your own interview tracking purposes. It’s a Gist because I can’t be bothered altering my wp_options.php to allow for .sql uploads, and that’s probably a terrible idea, anyway ;-)

Creating views based on field values to track the interview pipeline

Now that I had a useful table structure, I settled on some Views that helped me create and manage the interview pipeline. Views in nocodb are lenses on the underlying database - that restrict or constrain the data that is shown so that it’s more relevant to the task at hand. This is done through showing or hiding fields, and then filtering the selected fields.

• Data entry view - this was a form view where I could add new Interviewees.
• Views for parts of the pipeline - I set up several grid views that restricted Interviewees using filters to the part of the interview pipeline they were in. These included those I had and hadn’t contacted, those who had a scheduled interview, those who hadn’t responded, as well as several views for where the interviewee was in the coding and consent pipeline.
• At a glance view - this was a gallery view, where I could get an overview of all the potential and confirmed participants.

A limitation I encountered working with these views is that there’s no way to provide summary information - like you might with a SUM or COUNT query in SQL. Ideally I would like to be able to build a dashboard that provides statistics on how many participants are at each part of the pipeline, but I wasn’t able to do this.

Updating nocodb

nocodb is under active development, and has regular updates. Updating the software proved to be incredibly easy through npm, with a two-line command:

Uninstall NocoDB package

npm uninstall nocodb

Install NocoDB package

npm install --save nocodb

Parting thoughts

Overall, I have been really impressed by nocodb - it’s a strong fit for my requirements in this use case - easily prototypable, runs locally, and is easily updateable. The user interface is still not perfect, and is downright clunky in places, but as an open source alternative to AirTable and Notion, it hits the spot.

One definite bugbear I have though is how MaxQDA interacts with SharePoint. As part of my ethics protocol, I am storing my PhD data on university systems - not to external cloud tools like DropBox or Next Cloud. Instead, I save the MaxQDA files to my local (Windows - MaxQDA doesn’t have a Linux client, unfortunately) machine. This is then synced with OneDrive to the University’s SharePoint server.

This works well. Except when it doesn’t.

A couple of months ago I had an error that seemed like a once-off; an error where MaxQDA apparently couldn’t convert the project file. This error was presented when I opened the MaxQDA file (a .mx22 file):

Like so many error messages, it violated design principles for good error messages; it wasn’t a precise description of what had gone wrong, it wasn’t human readable, and it didn’t give me any helpful advice on how to solve the problem. So, I had to figure it out myself.

I tried the obvious things first;

• I closed MaxQDA and re-launched the software; the error persisted.
• I restarted my computer and then re-launched the MaxQDA software; the error persisted.
• I stopped and started the OneDrive service; the error persisted.

At this point, it was clear I’d have to dig deeper into OneDrive. In File Explorer, I could see that the file was still synchronising with OneDrive:

By rights, stopping and starting OneDrive should have re-synchronised the file; but it hadn’t.

OneDrive was also showing a synchronisation error:

Clearly, OneDrive thought that the MaxQDA file was still open; and was not syncing the file to the cloud for this reason. However, closing MaxQDA, OneDrive and a whole reboot had not fixed this error.

My conclusion from this investigation is that MaxQDA somehow leaves an open file handle; for example if the application closes unexpectedly. The open file handle is not cleared via MaxQDA, via OneDrive, or via the underlying Windows operating system. So how else might you clear an open file handle?

Windows isn’t my preferred operating system; and I don’t know enough about the OS internals to go digging into file handles and how to clear them. So I went rm -rf; or about as close to it as you can get on Windows …

The solution

The only thing that did fix the issue was uninstalling OneDrive, re-installing OneDrive, and then re-authenticating OneDrive and allowing it to sync to the cloud. My working hypothesis is that the uninstallation of OneDrive forces Windows to clear any open OneDrive file handles; then the re-installation returns the MaxQDA file to a known good state.

All in all, this took about an hour of investigation to identify the issue and find a workaround. And to be clear - the solution is just a workaround - it doesn’t address the underlying problem - which is that MaxQDA files that synchronise from a local machine to the cloud via OneDrive or SharePoint are prone to synchronisation failure that manifests in an open file handle; which in turn leads to an obscure error message.

This has now happened to me twice - but at least now I know how to fix it next time …

Update: Resetting the OneDrive cache appears to resolve this issue

So, after encountering this issue with MaxQDA for around the fifth time, and even after uninstalling and reinstalling OneDrive, I did a bit more digging, and found some blog posts that suggested resetting the OneDrive cache.

This is covered in this how-to-guide, but the commands are essentially:

1. Open the Windows command tool as Administrator (you need to be an administrator to clear the OneDrive cache)
2. Run the command %localappdata%\Microsoft\OneDrive\onedrive.exe /reset\
3. Then restart OneDrive by running the application

This worked for me - so it’s another possible workaround for this very frustrating issue!

Web camera software

While web cameras on Linux these days tend to “just work”, there are no Linux counterparts to the Logitech or other vendor-specific software that the hardware usually ships with. Way in back in January when I presented at linux.conf.au, part of my tech check (big thanks, Chris and jwoithe!) involved using the qv4l2 utility to provide finer-grained control over settings such as white balance, contrast, and brightness. This handy utility also controls auto and manual zoom on cameras, and I adjusted the zoom to better fill the screen.

From memory, I didn’t have to add an apt repository for this - it’s in the Ubuntu repos by default - and can be installed with:

sudo apt install qv4l2

Recording video and setting up a virtual background

Open Broadcaster Software

The next challenge was to record the video, using a virtual background. The problem of recording video on Linux is mostly solved these days using the Open Broadcaster Software (OBS) software.

Background removal plugin

While this software comes with a “chroma key” plugin (for green screens), it doesn’t come with an automatic background removal plugin. I read up on what plugins were available and ended up choosing this one from Roy Shilkrot (big thank you, Roy!) -

https://github.com/royshil/obs-backgroundremoval

It was well documented, and very clear about the installation steps. It did require compiling from source, but the instructions were detailed. Because I already had CUDA installed on my machine (I have an RTX2060 NVIDIA GPU, don’t get me started on Nvidia and Linux drivers and how hard they are to get working, but there are signs of hope, with NVIDIA recently open sourcing their Linux drivers - albeit through moving most of the functionality to firmware), I opted to compile with GPU support. Compilation was very straightforward, however the instructions provided for sym-linking the compiled binaries were out of date. Luckily, there was already an Issue that had been raised for this, and I used the information in the Issue to fix the sym-linking. This exercise gave me a deep appreciation for the effort that goes into packaging, and the complexity that it abstracts away for the end user.

Machine vision algorithms

This plugin uses machine vision (which is why OpenCV is one of the package’s dependencies) to determine the outline of the subject from the background, and then uses the Chroma Key to fill the non-subject space with a single colour. It allows you to choose from a range of algorithms to do the subject outline, such as SINet, MODnet, mediapipe and a model for selfies. I found I got the best results using mediapipe. The plugin also allows you to configure other inference settings, such as the frame interval between which inference samples are taken. Again, there is another trade-off here; if you resample the image frequently, this takes more computational power, but if you sample less frequently then the outline may not “keep up” with the movement of the subject. I chose to inference every frame - primarily because I had the compute power to do so 😈.

All of these computer vision (CV) algorithms are very recent - for example SINet [0] was developed just over a year ago, and MODnet [1] is less than 6 months old; I continue to be surprised at how fast the ML space is moving, and how commodotised it has become. If you have ever used the “virtual background” feature in a videoconferencing application like Zoom or Teams, it will be using a similar algorithm. However, a lot of the implementation decisions - like how often to do inference, which algorithm to use, or the characteristics of the algorithm such as smoothing or contrast, are abstracted away from the end user in these applications. What we lose in complexity and control, we gain in ease of use.

Running a noise removal filter on the video using ffmpeg

Although I had managed to do a video recording, sans background, when I played back the video, the audio had a lot of background noise. This came from the sound of the fans on my laptop; when the GPU is under load, the fans increase their speed to dissipate the heat that is generated. The inference of the background removal algorithm was causing the fans to spin up - and to generate background noise that was captured by the microphone. Tradeoffs, tradeoffs and more tradeoffs!

Rather than re-record the video, or reduce the inference frequency of the background removal plugin, I decided to see if I could improve the noise quality by running the video through a noise reduction filter. Signal processing, and acoustic signal processing, is definitely not an area that I feel comfortable with, but I had a general idea of how noise reduction filters worked - by gating (cutting off) noise above or below a particular frequency (the gating floor or gating ceiling). When I worked in videoconferencing, I’d seen a similar approach used with special digital signal processing equipment in videoconferencing-equipped spaces. Again, there’s a tradeoff - if you have a quietly spoken person, and a loud, deep speaker, but use the same gating settings, you’re going to get unexpected results.

Although I had Audacity installed, I reached for the command-line tool ffmpeg. The newer versions of ffmpeg have noise reduction filters built in, and the documentation includes several command line examples. Running the video through the noise reduction filter improved the quality markedly.

Video editing, compositing and rendering in OpenShot

Next, I used OpenShot to do video editing, addition of slides, and “topping and tailing” with the conference branding. OpenShot is a breeze to use, and although it doesn’t have a lot of fancy transitions, it’s an excellent choice for this step of the workflow. I previously converted the PowerPoint I was using for slides into PDF, then used GIMP to create a 1920 x 1080 px PNG image of each PDF. The PNG images were then imported into OpenShot and overlaid over the video tracks.

One of the branding artefacts was in a format (Quicktime MOV) that OpenShot didn’t particularly work well with; so I changed the format using Handbrake.

Again, having a reasonable GPU helped with the rendering process; each conference video is around 15 minutes long, and OpenShot rendered them at 1080p 30fps resolution in around 5 minutes 😈.

Subtitles

Subtitles are helpful for many people, and in particular they increase the accessibility of video content for folx who are hard of hearing. Several video creation tools can produce automatic subtitles. However, these are usually inaccurate at best, and at worst can semantically change the meaning of the video content. Moreover, I knew that in this video we would be using some domain-specific vocabulary (the video is all about accent bias in language technologies like speech recognition) - and that the automatic subtitles would be even less accurate.

For this task, I tried out the application Subtitle Editor, but found the interface difficult to use. I’d heard good things too about Aegisub, but in the end I just manually created the .srt file by hand-editing it in Atom. The .srt file format is surprisingly basic - and hand-editing was made a lot easier because we created a script for the video beforehand. It may not work as well if you are transcribing the subtitles from the spoken audio.

Edit: I eventually figured Subtitle Editor’s interface out, when I needed to move all the subtitles back by 7 seconds after topping and tailing the video; this feature itself is worth installing the application.

The one thing that tripped me up in hand-editing subtitles was forgetting that the time format is HH:MM:SS,ms. My video was only 16 minutes in length, and I erroneously placed the “minutes” in the “hours” column. For example, I wrote 00:05:00, 400 to mean 5 seconds into the video, but this is actually 5 minutes into the video. The millisecond separator - the comma - is unusual in English, but the SRT format was developed in France; which uses the comma. This reminded me of Lawrence Busch’s excellent book Standards: Recipes for reality, which articulates the path dependencies that standards create.

1
00:00:05,400 –> 00:00:12,300
[Kathy] Hi everyone, I’m Kathy Reid, a PhD candidate
at the School of Cybernetics
at Australian National University,

2
00:00:12,400 –> 00:00:22,300
and I’m recording today from the unceded lands
of the Wadawarrung people here in Geelong,
on the south-west coast of Victoria.

Conclusion

Video production for conferences is becoming more prevalent, particular as the COVID-19 pandemic has taken many conferences online. Tools like Zoom and Camtasia, while mature, abstract away many of the underlying algorithms, and simplify their interfaces for usability by removing fine-grained controls from the user. By exploring open source tools for video production workflows, we obtain a deeper understanding of “what’s under the hood”.

To extend this work in the future, one of the capabilities I would like to explore is automation of the workflow pipeline. This may be possible for tasks such as topping and tailing videos with conference branding, but the actual composition of video and slides is still going to require human attention.

Footnotes

[0] Fan, D. P., Ji, G. P., Sun, G., Cheng, M. M., Shen, J., & Shao, L. (2020). Camouflaged object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2777-2787).
https://github.com/DengPingFan/SINet

[1] Ke, Z., Sun, J., Li, K., Yan, Q., & Lau, R. W. (2020). MODNet: real-time trimap-free portrait matting via objective decomposition.
https://github.com/ZHKKKe/MODNet