Linguistics Markers in False, Opinionated, and Verified News Headlines and Social Media Posts

As part of their Advanced Corpus Linguistics course project, a group of UBC Master of Data Science (MDS) Computational Linguistics students (Rachelle De Jager, Shiao-li Green, Nicole Shantz, and Jennifer Flake) wanted to explore linguistic markers in false, opinionated, and verified news headlines and social media posts.

The students sourced their data from the Hugging Face Twitter Misinformation dataset, which is comprised of four different sources:

1. Fake and real news articles;
2. Tweets using disaster-words and whether they were about disasters or something like ‘my phone blew up”;
3. Tweets on real-life disaster events, where all were labelled as factual; and
4. MuMiN: Multilingual Multimodal Fact-Checked Misinformation Social Network, which consists of claims made on social media that have been professionally fact-checked. 

They then enriched the dataset with web-scraped headlines from Google News.

The students wanted a dataset compiled from multiple sources in order to best capture the difference between factual reporting, misinformation, and personal opinions.    

Their original dataset had 102,661 items and they annotated a subset of 1,000 with the following linguistic markers:

1. adjectives: Adjectives with suffixes -ful, -less, -ment, -ness, -ing, or -ible;
2. all_caps: Capitalised words that are not acronyms;
3. exclamation_marks: Any number of exclamation marks;
4. hedging: Words found in the hedging lexicon (word or phrase used in a sentence to express ambiguity, probability, caution, or indecisiveness); and
5. unk: Profanity.

Their approach was to study misinformation detection differences between an undergraduate student, AI and themselves. By evaluating the language used in each data point with their annotation markers, they were able to measure inter-annotator agreement (a statistical measure of consistency between different human annotators labelling the same data). 

These results informed what subsample would be used in the final dataset. This process allowed them to implement the three-class label system to train their models.

In addition, during tagging data with misinformation marker tags, the students saw a lot of profanity in text that appeared to contain misinformation. After this observation, they decided to add profanity as a marker that should be labelled.

What the students discovered was that AI did not annotate as well as humans did. Humans and AI differed in annotation ability according to which label they were applying. AI was particularly bad at identifying the specific types of adjectives that were used in the annotation scheme. For example, rather than only labelling words with an “-ing” suffix that are adjectives, AI tended to label all words with an “-ing” suffix, regardless of part of speech. On the other hand, AI excelled at identifying exclamation points.

Interestingly, humans disagreed most on what words qualified as profanity. Some considered acronyms of phrases containing profanity to qualify, while others did not. Similarly, some considered profanity any word that you wouldn’t use in the workplace, while others thought profanity should be used for only the really offensive words. Therefore, the human and AI inter-annotator agreement score was slightly higher than the humans-only inter-annotator score (for profanity).

The end result of their project was a dashboard that allows users to search the texts by keyword and view the associated annotations. The results can be toggled to compare human and AI annotations and also highlight specific markers which were labelled within each text.

The students feel that their corpus can be used by media literacy educators. Their annotated examples show how misinformation differs from verified reporting based on content and language usage. Their reported statistics also demonstrate how human and AI annotation decisions can differ.

Explore Computational Linguistics Explore Other Data in Action Stories