-
Services
computer vision
self-serve
- Industries
- Pricing
- About
- Careers
-
Resources
- talk to us
Academia
Linguistic Annotation for Technological University Dublin
Ireland
Overview
Kyle Hamilton from Technological University Dublin collaborated with Label Your Data for her research on rhetorical devices of propaganda used in news articles.
Client
Kyle Hamilton is a PhD Researcher at TU Dublin who focuses on applying neuro-symbolic AI to detect propaganda in news feeds.
Challenges
Kyle needed a high-quality text dataset containing classified propaganda-related sentences to compare humans and ChatGPT in detecting propaganda in news.
Solutions
Label Your Data provided skilled annotators with linguistic background to classify and label 357 sentences using the Client’s platform.
Results
Despite inconsistencies in human annotations, they prove more reliable in propaganda analysis due to ChatGPT’s lack of real-world knowledge.
Client
Kyle Hamilton is a PhD Researcher at Technological University Dublin. Her research work is centered around neuro-symbolic AI for detecting propaganda in news articles.
Holding a Master’s in Information and Data Science and a Bachelor of Fine Art, Kyle brings a unique interdisciplinary approach to her work.
Rhetoric Annotation Tool
Challenges
Most efforts to automate the detection of propaganda and misinformation are mainly focused on using natural language processing (NLP).
Following the same idea, Kyle aimed to create a tool that helps identify propaganda in news. But first, she wanted to compare ChatGPT and human linguists in performing the same task.
Kyle Hamilton needed help with:
Exploring the possibilities of using ChatGPT for automated propaganda detection.
Hiring an expert annotation team with domain experts to compare human annotations to ChatGPT’s responses.
Getting high-quality annotated text dataset for the research.
Dealing with the subjective nature of the sentence classification task.
Solution
Ratio of partial agreement among all three annotators for each feature
Chat GPT Agreement among itself when prompted 3 times
Verb choices
Tropes
Tense
Subject choices
Series
Sentence architecture
Prosody and punctuation
Predication
Phrases built on verbs
Phrases built on nouns
Parallelism
New words and changing uses
Mood
Modifying phrases
Modifying clauses
Lexical and semantic fields
Language varieties
Language of origin
Figures of word choice
Figures of argument
Emphasis
Aspect
Label Your Data helped with:
Hiring 3 dedicated data annotators with linguistic background for classification of 357 sentences.
Working in a flexible mode to seamlessly integrate into Kyle's annotation platform.
Conducting a cross-reference QA to address the project’s subjectivity.
Delivering high-quality annotated text corpus.
Results
This initial phase aimed at supporting Kyle Hamilton’s research and securing the grant. While findings are initial, Label Your Data expects a larger dataset to annotate.
Though human experts’ annotations vary, their real-world knowledge surpasses AI models like ChatGPT, which solely relies on internet-derived data.
Analyzed the discrepancies in agreement between annotators and ChatGPT.
Identified the most challenging areas to achieve consensus:
Series
Prosody and punctuation
Phrases built on verbs
Parallelism
Figures of word choice
Figures of argument
Despite inconsistencies, defined human annotators as more reliable for propaganda analysis.
Delivering high-quality annotated text corpus.
Request a pilot
Tell us more about your project and data
Thank you for contacting us!
We'll get back to you shortly
After running pilots with several annotation providers, Label Your Data delivered the strongest results by a clear margin, standing out on turnaround time, annotation quality, and the responsiveness of their feedback loops.
Maxime Debarbat
Senior ML Engineer (GenAI)
Trusted by ML Professionals
Why AI Teams Choose Label Your Data
Data Annotation for Complex Environments
Rely on consistent, high-quality output for complex datasets, detailed taxonomies, and edge cases.
Structured Quality from Pilot to Production
Get quality engineered into every step through onboarding, evolving guidelines, QA, and continuous feedback.
Flexible and Scalable Operations
Adjust team capacity, project size, and delivery model as you scale, with no setup fees or long-term lock-ins.
An Integrated Delivery Partner
Align on goals, workflows, and expectations with a team that integrates into your process from day one.
Projects Led by Annotation Experts
Work with former annotators who understand annotation complexity, quality standards, and high-volume delivery.