AI safety is a contentious topic. While some prominent figures of the AI community have argued that destructive general artificial intelligence (AI) is on the horizon, others derided their warning as a marketing stunt to sell large language models (LLMs). “If the call for ‘AI safety’ is couched in terms of protecting humanity from rogue AIs, it very conveniently displaces accountability away from the corporations scaling harm in the name of profits,” tweeted Emily Bender, a professor of computational linguistics at the University of Washington. Focusing on potential future harm from ever more powerful AI systems distracts from harm that is already happening today.
Most of us do not set out to make software that is actively harmful. And yet there is a plethora of AI systems that are harmful to queer people. Text-to-image systems turn trans people into hyperexualized caricatures, translation systems obliterate neo-pronouns and sentiment analysis tools place queer identity terms on par with negative adjectives. How did we get there and how can we fix this?
Unfortunately, the reasons are baked into the foundations of machine learning. Machine learning picks up and reproduces statistical patterns from large amounts of data. This makes it an especially good method whenever it is difficult to define hard rules or algorithms to solve a problem, but there are plenty of examples to learn from. But being part of a minority means that data instances representing the minority are much more rare than instances representing the majority. Or even worse, as it is often the case with the queer community, much of the representation could be negative. While recent methods for the alignment of LLMs are moving in the right direction, they are no panacea: rather than removing harmful instances from the training data, they add another training signal to counterbalance undesired outputs. Alignment is still easily circumventablea and covers only one specific use case of machine learning systems, namely LLMs with chat interfaces.
If the pictures of trans people in an image dataset are mostly scraped from pornography websites, then AI-generated pictures of trans people will be hyper-sexualized. If machine translation datasets contain no instances of neo-pronouns it is not surprising that non-binary characters are misgendered in automatically translated video subtitles. And if queer identity terms are used in a derogatory way in the training data for LLMs, it is not surprising if sentiment analysis systems that build on LLMs assume queer identity terms are in fact swear words and not neutral descriptors.
Which leads us to the next question: How can computer science education and especially continuous training of the programming workforce address these shortcomings? Let’s imagine possible scenarios of how a group of developers in an industry setting might come to consider the impact of their software on the queer community and how computer science education can come into play.
ExampleCorp is a company that produces and markets software to a wide range of commercial clients. One of their products is a text-to-image system that generates personalized advertisements. For the upcoming pride month companies want to use the raised awareness to create ads that show their products in domestic scenes with queer couples, but this goes horribly wrong. One Monday morning the development team has an emergency meeting with Felicity, the head of the customer service department. She is faced with angry clients who claim that the text-to-image system produces homophobic caricatures when prompted with texts like “Lesbians at breakfast” or “Gay couple baking a cake.”b
In a first knee-jerk reaction the development team blocks the text-input-API for all queer identity terms, so that typing any prompt with “gay” or “lesbian” in it will lead to an error message. Similar routes have been taken by other large tech companies: for some time the question “What is gay?” to Google’s Bard led to a canned response that denied further information, while “What is straight?” was treated as inoffensive by the system. At ExampleCorp the team looks through the training data, filtering for images with descriptions with queer identity terms in it. They quickly find the culprit: a big proportion of the tagged images were scraped from homophobic Web forums and some from pornography pages. But after removing these harmful samples there are many fewer pictures with queer descriptors left in the dataset. While the ads generated are not offensive any more, they are also of much lower quality than ads without queer identity terms. As a workaround the team communicates that customers should use prompts such as “Two women at breakfast” or “Two men baking a cake,” and replace prompts like “trans woman” with “woman holding a trans flag.” The team then bolsters the training dataset with stock photos of queer people, which have to be bought rather than scraped, and which are then manually annotated by crowd workers, adding to the price tag. After that the model performance gets a little bit better. But design oversights are not easily remedied on-the-fly: the solution comes too late and the public image of the company is damaged.
To prevent future mishaps like these, the company contacts a local queer organization that teaches diversity workshops. Together with the development team they come up with new design and testing criteria. They take an intersectional view of the problem too, making sure that other axes of discrimination are addressed. Armed with this new knowledge, the development team sets about disseminating it to other teams within the company, running peer-led workshops and giving lunchbreak talks. Two developers in particular, Sandja and Tom, take a liking to the topic and become unofficial experts who keep up with new developments and literature. They also join the Slack channel of Queer in AI, an organization that represents queer people who work in AI and that is an active forum of exchange. Community members point them to helpful literature and upcoming lectures or workshops.
A few months later ExampleCorp plans to start working on a tool that translates video subtitles in real time. Lee’s team is tasked with implementing it. Lee still vividly remembers the diversity workshop. He also is a big fan of the Netflix series One Day at a Time that features a non-binary character, Syd. Having learned Italian in college Lee wonders how Syd’s pronouns would be translated to Italian, where there are no established gender-neutral pronouns. How can the team make sure that the automatic translation does not misgender non-binary people?
After searching academic publications the team reaches out to a group of researchers who recently organized a workshop about the topic.c Discussions with them makes the team at ExampleCorp consider a different approach to the application. Instead of integrating seamlessly into a video-application, the subtitle tool asks the user which translation strategy should be used for the translation of English gender-neutral pronouns into other languages. Should gendered pronouns be avoided entirely or should a diacritic marker be used, or a completely new declination? Implementing this requires more time and effort, but the leadership is willing to provide the necessary resources to avoid another publicity fallout. The company even actively points out the new feature in its advertising, promoting it as an advantage over competing products. The additional user-interaction gives the team information about preferred translation strategies and helps them to improve the app continuously after deployment.
The negative experience with the image generation system and the following fallout has also raised uncomfortable questions in the hiring department: several developers are convinced that the harmful images would have been noticed earlier if there were more queer people at ExampleCorp. Is ExampleCorp an unwelcoming environment? Managers notice that they’ve mostly recruited people who are similar to themselves: straight white men who graduated from the same university as them. Now job ads do not only go out to LinkedIn and alumni lists, but are also sent to the message boards of groups like Queer in AI, Black in AI, and Latinx in AI. Before the image generation disaster a mention of social activism in the job interview would have only elicited a polite nod, but now managers have come to see it as a valuable asset.
But is our task done now? Acknowledging privilege and making amends takes more than just a seminar, or a knowledgeable colleague or two. The newly minted queer bias experts, Sandja and Tom, continue reading on the topic, which leads them to a daring suggestion: instead of having AI generate ads for pride month, why not recommend customers queer modeling agencies that could be hired for individual shoots or for supplying stock photography? After all, harm is being done when companies decide to pay for an AI that generates pictures of queer people rather than hire queer models and artists. The people whose pictures were scraped by ExampleCorp to create the AI in the first place were never asked for permission or remunerated either. Maybe they should do something about that, too?
It is not difficult to imagine the outcome. Chances are that any suggestion that threatens the profit model of the company will be rejected. What if we embrace the idea that users should be in charge of disclosing gender information rather than having their gender predicted based on their behavior? That would be in tune with the fact that people might change their gender expression over time or situationally, but would make targeted advertising more difficult. What if hate speech and misinformation would be silenced rather than amplified by recommender systems? That would increase user happiness but lower their engagement with the app and therefore revenue for the company. What if a product solved a problem for a demographic that is historically oppressed? Probably people of that demographic have less money to spend on technology, which makes their problems uninteresting as a business case. Only recently have companies started to target the queer community, with Subaru’s famous lesbian car ads as one of the first examples. But often enough, the concern does not extend past a rainbow logo for pride month.
That is why truly out-of-the-box thinking and liberating queer AI development happen outside the commercial sphere in open source projects and grassroot organizations, and in protest movements against big companies rather than in hackathons sponsored by them. A company can invite queer people to audit their systems for biases and might even pay them to do so, but at the end of the day there is no real accountability or co-ownership that would allow a system to be vetoed.
This column has outlined a few ways we can build queer-friendly systems: With post hoc haphazard adjustments, with sensitive and value-driven design, or by community co-creation. And how can we teach people to build better systems? This needs first and foremost a diverse workforce and a company structure that allows for compassion, thorough thinking and the freedom to ask tough questions. Is ExampleCorp realistic for what is possible and likely in today’s world? I am not sure, but I sure hope so.
