Data Integrity Series. Chapter 2: Ensuring High-Quality Respondents in Online Surveys

In recent times, the world of market research has been facing increasing data quality issues due to the growing presence of bad quality respondents, bots, fraudsters, and even AI (as we mentioned in our previous blog post). The reliability and accuracy of online surveys’ data poses a significant challenge for businesses looking to gather meaningful insights.

In this blog post, we’ll explore various measures and methods CMR has in place to guarantee good quality respondents in online surveys, helping combat fraudulent activity and ensuring the integrity of our data.

CMR Strategies to Combat Fraud and Guarantee Data Quality

1. Survey Design:

One of the initial steps in ensuring quality respondents is crafting an engaging survey. A well-designed questionnaire can minimise respondent fatigue and inattentiveness. To achieve this we:

Keep it interesting: By creating surveys with engaging and relevant questions that maintain the respondent’s interest throughout the process.

Keep it concise: By limiting the Length of Interview (LOI) as much as possible. Long surveys are more likely to lead to dropouts and reduced data quality.

Diverse question types and exercises: By mixing different question types, including visual materials and different types of exercises to keep respondents engaged and encourage thoughtful responses.

2. Survey Programming:

During the survey programming phase, several techniques are employed to identify and filter out low-quality respondents:

Programmed Verification Methods: A range of techniques are implemented in our online surveys, such as CAPTCHAs and other human verification methods to thwart bots; Geolocation and IP tracking for location validation; and the inclusion of Honeypots and Trap Questions to identify and deter potential bot activity.

Encrypted Survey Links: Survey links are encrypted to prevent tampering or corruption of survey URLs.

3. Data Quality Checking:

In the subsequent data quality checking phase, our additional rigorous methods come into play. We employ various strategies to assess respondent behaviour and scrutinise responses, such as:

Page-Level Time Tracking: The time spent on individual pages and the survey as a whole is monitored to identify speeders, suspicious respondents or bot-like behaviour.

Quality Check Flags at the Screener Level: Domain knowledge is leveraged to identify genuine respondents, with specific questions related to the topic being asked in the screener to filter out non-experts or dishonest participants.

Consistency Check Questions: Questions that mirror others in the survey are added with the intent of obtaining the same responses, making it easier to spot fraudsters or respondents who aren’t paying attention or providing inaccurate information.

Numeric Questions for Over/Under Usage Estimation: Numeric questions with wide ranges are added to identify responses that are statistically improbable.

Open-Ended Questions: Open end responses are thoroughly analysed case by case for duplications, nonsensical answers, and poor articulation. Additionally, auto-fill and copy-pasting in open-ended questions is disabled to make it more difficult for bots, AI respondents and fraudsters to provide their answer.

Identifying Straight Liners: Respondents who provide the same response for multiple questions without variation are detected, suggesting automated or inattentive behaviour.

4. Choosing our sample sources wisely

Selecting reliable partners who share our commitment to data quality is crucial. We build strong relationships with our sample sources, emphasising transparency and trust. Collaborating with partners who also implement robust verification methods and stringent screening ensures that the respondents they provide meet our quality standards. This strategic partnership allows us to access high-quality respondents while continuously verifying their authenticity, strengthening the reliability of our survey data.

5. Partnering with new technology

Our commitment to data quality doesn’t stop there. We’re constantly on the lookout for innovative technologies and methods to enhance our data checks further. We closely analyse new technologies and emerging trends, as pre-survey vetting systems and advanced algorithms to filter out unwanted responses, ensuring that we remain at the forefront of data quality assurance. By staying proactive and adaptive, we continuously improve our processes to maintain the highest standards in research and insights.

Safeguarding the quality of respondents in online surveys is crucial for obtaining reliable market research data. By implementing these strategies across the survey design, programming, sample sourcing and data quality check phases, it is possible to significantly reduce the impact of fraudulent activity and poor-quality data. It’s an ongoing battle but staying vigilant and continuously improving survey methodologies helps us, at CMR, to maintain the integrity of our research efforts and derive meaningful insights from the data collected.

Don’t forget to check out our previous blog post in this series on The Evolution of Data Quality Issues in Online Market Research Surveys.