Data Integrity Series. Chapter 5: AI vs. Humans

Online surveys have become an invaluable tool for researchers, businesses, and organisations seeking to gather insights from a wide range of respondents. However, ensuring the quality of the data collected in online surveys is an ongoing challenge. One of the emerging issues is the presence of fraudulent AI-like responses, which can skew results and lead to erroneous conclusions.

In this blog post, we will explore the importance of identifying and filtering out such responses, as well as the role of both AI tools/programs and human oversight in maintaining data quality.

Tackling the Rise of AI-Generated Respondents in Online Surveys

Distinguishing AI-Like Respondents from Poor-Quality Real Respondents

AI-like respondents, often generated by automated scripts or artificial intelligence models, differ from poor quality real respondents in that AI-like responses exhibit a pattern of consistent, robotic behaviour devoid of genuine human engagement. These responses may mimic human language and behaviour to a high degree but lack the nuanced context, variability, and emotional authenticity typically present in responses from real people.

In contrast, poor quality, real respondents may provide inconsistent or unreliable answers due to factors like inattention, lack of understanding, or survey fatigue, but their responses still carry traces of genuine human variability and context. Distinguishing between the two is crucial for maintaining the reliability and accuracy of online survey data.

Removing Fraudulent AI-Like Responses

While the increasing number of AI tools offer promising solutions for detecting fraudulent AI-like responses in online surveys, human involvement remains an indispensable element in the quest for data quality. In this section, we explore the methods used for identifying and removing fraudulent AI-like respondents and the vital role played by both AI and human reviewers in filtering them out.

1. Use of AI Tools and Programs:

The development and use of AI tools and programs as a line of defence against fraudulent AI-like responses has suddenly been increasing. These tools employ a range of techniques to detect anomalies in survey responses. Some common methods include:

Pattern Recognition: AI tools can identify patterns that suggest automated responses, such as consistent response times, unusual mouse movements, pasting text responses or real-time translation of survey screens.

Language Analysis: Advanced AI can analyse the language used in responses to detect machine-generated text or inconsistencies in writing style.

IP Address, Fingerprinting and Geo-Location Monitoring: Tracking IP addresses, fingerprinting and geo-location can help identify multiple responses from the same source, indicating potential fraud.

2. Human Involvement is Crucial:

While AI tools are effective at detecting many fraudulent responses, they are not 100% accurate and reliable. Human involvement is essential for several reasons:

Contextual Understanding: Humans can interpret responses in the context of the survey and its objectives, allowing them to detect nuanced issues that AI may miss.

Complexity of Fraud: Fraudsters continually evolve their methods. Humans can adapt and identify new patterns of fraudulent behaviour that AI tools may not have encountered before.

Ethical Considerations: Some responses may not be fraudulent but rather the result of genuine misunderstanding or confusion. Human reviewers can exercise judgment and discern intent.

3. Combining AI and Human Oversight:

A hybrid approach that combines the strengths of AI tools and human reviewers is the most effective way to ensure data quality. AI can quickly identify potential issues and flag responses for human review. Human reviewers can then delve deeper into these flagged responses, applying their expertise to make final determinations.

Continuous Monitoring is Essential

Maintaining data quality in online surveys is an ongoing battle, especially in the face of fraudulent AI-like responses. Online surveys should not rely solely on initial data checks. Continuous monitoring throughout the survey period is essential to catch evolving fraudulent tactics and maintain data integrity.

While AI tools and programs play a crucial role in detecting and flagging potential issues, human involvement is equally indispensable. Human reviewers provide the contextual understanding, adaptability, and ethical judgment needed to ensure the reliability of survey data. The combination of AI and human oversight represents the most effective strategy for identifying and mitigating fraudulent responses while preserving the integrity of online survey results. At CMR, we embrace this hybrid approach, so that researchers, businesses, and organisations can trust that their survey data reflects genuine insights from real respondents.