本文以2016年微软推出的Twitter机器人Tay为例，探讨了人工智能开发中可能出现的风险和问题。Tay由于学习了用户发布的带有种族主义和性别歧视等有害信息，导致其发布了大量不当言论，最终被紧急下线。文章分析了Tay事件中数据、设计、测试和监控等环节存在的不足，并强调了在人工智能开发过程中，数据质量、模型设计、测试和监控的重要性，以及这些环节对AI项目成功与否的关键作用。

🤔**数据质量至关重要：**Tay事件中，Twitter上的不良用户发布了大量带有偏见和歧视性的信息，Tay学习了这些信息，导致其输出也带有偏见和歧视。这表明，训练AI模型的数据质量会直接影响模型的输出结果，使用低质量、有偏见或不完整的数据会训练出有问题的模型。

💡**模型设计需谨慎：**Tay的设计允许其持续学习外部环境中的信息，这导致其学习了大量有害信息。这表明，AI模型的设计需要考虑其可能面临的各种风险，并采取措施避免模型学习到有害信息。

🧪**充分的测试必不可少：**Tay在发布前没有进行充分的测试，导致其有害行为在发布后才被发现。这表明，AI模型在发布前需要进行充分的测试，包括压力测试和业务指标评估，以识别潜在的风险和问题。

📊**持续监控至关重要：**Tay的持续监控最终导致其被下线。这表明，AI模型在发布后需要持续监控其行为，及时发现和解决潜在问题，防止其产生负面影响。

🤝**责任共担：**AI开发的风险和责任不仅在于开发者，也需要业务相关人员共同承担，对每个环节都进行充分的思考和设计，才能最大程度地降低风险，提高AI项目的成功率。

Of late, we’ve been hearing about Twitter bots in the news due to the whole saga of Elon Musk buying Twitter. One of the reasons the deal took so long to pan out was Musk’s concerns about the number of spam bots running rampant on the platform. While Musk believes that bots make up more than 20% of accounts on Twitter, Twitter states that the number of bots on its platform is marginal.

So, what’s this Twitter bot thing?

A Twitter bot is essentially a Twitter account controlled by software automation rather than an actual human. It is programmed to behave like regular Twitter accounts, liking Tweets, retweeting, and engaging with other accounts. 

Twitter bots can be helpful for specific use cases, such as sending out critical alerts and announcements. On the flip side, they can also be used for nefarious purposes, such as starting a disinformation campaign. These bots can also turn nefarious when “programmed” incorrectly.  

This is what happened with Tay, an AI Twitter bot from 2016.   

Tay was an experiment at the intersection of ML, NLP, and social networks. She had the capacity to Tweet her “thoughts” and engage with her growing number of followers. While other chatbots in the past, such as Eliza, conducted conversations using narrow scripts, Tay was designed to learn more about language over time from its environment, allowing her to have conversations about any topic. 

In the beginning, Tay engaged harmlessly with her followers with benign Tweets. However, after a few hours, Tay started tweeting highly offensive things, and as a result, she was shut down just sixteen hours after her launch.

You may wonder how can such an “error” happen so publicly. Wasn’t this bot tested? Weren’t the researchers aware that this bot was an evil, racist bot before releasing it? 

These are valid questions. To get into the crux of what went wrong, let’s study some of the problems in detail and try to learn from them. This will help us all see how to handle similar challenges when deploying AI in our organizations. 

Data

Data is often a big reason why AI models fail. In the case of Tay, shortly after her release,  Twitter trolls started engaging the bot with racist, misogynistic, and anti-Semitic language. And because Tay had the capacity to learn as she went, it meant that she internalized some of the language taught by the trolls. Tay just repeated some of this language. Tay uttered bad language because she was fed bad data.

Take note: Poor-quality, prejudiced, or downright bad training data can significantly impact how machine learning models behave. You train ML models with nonrepresentative data, and they will churn out biased predictions. If you starve models of data or feed models incomplete data, they will make random predictions instead of meaningful ones. Questionable learning/training data = questionable output. 

Questionable training data = questionable ML model output 

Design 

While we don’t often relate model or solution design to erratic model behaviors, it’s often more common than you think. By design, Tay continuously learned from external input (i.e., the environment). Among all the benign Tweets that Tay consumed from her environment were also abrasive Tweets. The more abrasive Tweets Tay saw, the more she learned that those were typical types of responses to Tweet. 

This is true of any ML model. The dominant patterns influence the predictions of the ML models. Fortunately,  it’s not necessary for ML models to learn continuously from their environment. ML models can learn from controlled data. So, Tay’s design itself was risky. 

Take note: The design of your ML models impacts how it behaves in reality. So, when designing ML systems, developers and business stakeholders should consider the different ways in which the system can fail, operate suboptimally, be breached, and adjust the design accordingly. In the end, you need a fail-safe plan. 

In the case of Tay,  such thinking early on would’ve made clear that not all Tweet engagements would be benign. There could be bad actors tweeting and engaging in a highly offensive manner, not far-fetched at all from reality. The realization that the bot could be consuming bad data may have stopped the team from using data from other Twitter accounts. They may also have considered consuming data from approved Twitter accounts. 

The design of your ML models impacts how it behaves in reality.

Testing

One of the key steps in the machine learning development lifecycle is testing—not just during development, but testing right before full deployment. I call this post-development testing (PDT). 

The ML Development Life Cycle

In the case of Tay, It’s unclear how much PDT went on before releasing the bot, but obviously, it wasn’t enough! Had Tay been subjected to different types of tweet engagements during PDT, the dangers of releasing Tay would’ve become obvious.

Take note: In practice, PDT is often overlooked due to a rush to release a new feature or product. It’s often assumed that if a model works well during development, it will naturally perform well in practice. Sadly, that’s not always the case. So, take note that PDT is critical when it comes to AI deployment.

During PDT, you can stress test your AI solution to find points of failure. In the case of Tay, subjecting it to different types of Twitter users (e.g., trolls, benign users, and passive aggressives) could’ve surfaced risky behaviors of the bot.  PDT can also help evaluate your solution’s impact on relevant business metrics. For example, suppose your business metric measures speed improvement in completing a particular task. PDT can give you early insights into such metrics. 

During PDT, you can stress test your AI solution to find points of failure. PDT can also help evaluate your solution’s impact on relevant business metrics.

Monitoring

Another critical component in the ML development lifecycle is monitoring after deployment. With Tay, monitoring the bot’s behavior eventually led to it being shut down within 24 hours of its release (side note: negative press also had a hand in it). If the bot hadn’t been monitored long after its release, this could’ve led to a whole lot more negative press and many more groups being offended. 

Take note: While model monitoring is often done as an afterthought, it should be prioritized before its release to end users. The initial weeks after a model’s release is the most crucial, as unpredictable behaviors not seen during testing could emerge. 

The initial weeks after a model’s release is the most crucial, as unpredictable behaviors not seen during testing could emerge. 

Summary

While what went wrong with Tay may be surprising and intriguing to many, from a machine learning best practices perspective, Tay’s behavior could’ve been predicted. Tay’s environment wasn’t always positive, and she was designed to learn from that environment which led to a perfect recipe for a dangerous experiment. 

So decisions around data, model design, testing, and monitoring are critical to every AI initiative. And this is not just the responsibility of the developers but also the business stakeholders. The more thought we put into each element, the fewer the surprises and the higher the chances of a successful initiative. 

That’s all for now!