AI Is Learning to Identify Toxic Online Content人工智能学习识别网上恶意内容

Machine-learning systems could help flag hateful， threatening or offensive language. 机器学习系统可帮助标记仇恨性、威胁性或攻击性言论。

Social platforms large and small are struggling to keep their communities safe from hate speech， extremist content， harassment and misinformation. One solution might be AI： developing algorithms to detect and alert us to toxic and inflammatory comments and flag them for removal. But such systems face big challenges.

The prevalence of hateful or offensive language online has been growing rapidly in recent years. Social media platforms， relying on thousands of human reviewers， are struggling to moderate the ever-increasing volume of harmful content. In 2019， it was reported that Facebook moderators are at risk of suffering from PTSD as a result of repeated exposure to such distressing content. Outsourcing this work to machine learning can help manage the rising volumes of harmful content. Indeed， many tech giants have been incorporating algorithms into their content moderation1 for years.

One such example is Google’s Jigsaw2， a company focusing on making the internet safer. In 2017， it helped create Conversation AI， a collaborative research project aiming to detect toxic comments online. However， a tool produced by that project， called Perspective， faced substantial criticism. One common complaint was that it created a general “toxicity score” that wasn’t flexible enough to serve the varying needs of different platforms. Some Web sites， for instance， might require detection of threats but not profanity， while others might have the opposite requirements.

Another issue was that the algorithm learned to conflate toxic comments with nontoxic comments that contained words related to gender， sexual orientation， religion or disability. For example， one user reported that simple neutral sentences such as “I am a gay black woman” or “I am a woman who is deaf” resulted in high toxicity scores， while “I am a man” resulted in a low score.

Following these concerns， the Conversation AI team invited developers to train their own toxicity-detection algorithms and enter them into three competitions （one per year） hosted on Kaggle， a Google subsidiary known for its community of machine learning practition-ers， public data sets and challenges. To help train the AI models， Conversation AI released two public data sets containing over one million toxic and non-toxic comments from Wikipedia and a service called Civil Comments. Some comments were seen by many more than 10 annotators （up to thousands）， due to sampling and strategies used to enforce rater accuracy.

The goal of the first Jigsaw challenge was to build a multilabel toxic comment classification model with labels such as “toxic”， “severe toxic”， “threat”， “insult”， “obscene”， and “identity hate”. The second and third challenges focused on more specific limitations of their API： minimizing unintended bias towards pre-defined identity groups and training multilingual models on English-only data.

Our team at Unitary， a content-moderation AI company， took inspir-ation from the best Kaggle solutions and released three different models corresponding to each of the three Jigsaw challenges. While the top Kaggle solutions for each challenge use model ensembles， which average the scores of multiple trained models， we obtained a similar performance with only one model per challenge.

While these models perform well in a lot of cases， it is important to also note their limitations. First， these models will work well on examples that are similar to the data they have been trained on. But they are likely to fail if faced with unfamiliar examples of toxic language.

Furthermore， we noticed that the inclusion of insults or profanity in a text comment will almost always result in a high toxicity score， regardless of the intent or tone of the author. As an example， the sentence “I am tired of writing this stupid essay” will give a toxicity score of 99.7 percent， while removing the word “stupid” will change the score to 0.05 percent.

Lastly， all three models are still likely to exhibit some bias， which can pose ethical concerns when used off-the-shelf3 to moderate content.

Although there has been considerable progress on automatic detection of toxic speech， we still have a long way to go until models can capture the actual， nuanced， meaning behind our language—beyond the simple memorization of particular words or phrases. Of course， investing in better and more representative datasets would yield incremental improvements， but we must go a step further and begin to interpret data in context， a crucial part of understanding online behavior. A seemingly benign text post on social media accompanied by racist symbolism in an image or video would be easily missed if we only looked at the text. We know that lack of context can often be the cause of our own human misjudgments. If AI is to stand a chance of replacing manual effort on a large scale， it is imperative that we give our models the full picture.

大大小小的社交平台都在竭力保障用户远离仇恨言论、极端内容、网络骚扰及错误信息。人工智能或可成为一种解决方案：开发算法来检测恶意和煽动性言论，打上删除标记，并向我们发出警告。但此类系统面临重大挑战。