Avoiding False Positives: Survey Data Protection is a Balance

January 21, 2021

If you’re like me, you’re sick of hearing about the “New Normal” that COVID has inflicted upon us. Some examples are very obvious while others not so much. Respondents in survey research are living through subtle changes that are impacting survey data in concerning ways. In short, changed consumer behaviors have exposed real flaws in outdated or simple data protection tools that block too many good respondents (aka “false positives”) while missing actual fraudsters.

Since survey data protection and improving quality is what OpinionRoute does all day, we are happy to share some recent trends from across the industry.


1- While certain fraud approaches are growing in prevalence, others are declining as efficacy wanes.

2- Newer, complex fraud approaches have been discovered right here in the USA.    

3- “False positives” are increasingly prevalent as common detection approaches are not understanding new consumer behavior in this new normal.  

While all trends are notable. Point 3 is the focus for this piece.


With our Data Quality as a Service product, CleanID, OpinionRoute has participated in scores of studies designed by clients to measure the effectiveness of various fraud prevention tools. Through these experiences, we’ve noticed that the pace of change in normal consumer behavior has accelerated so quickly during COVID, that “new normal” common events are being incorrectly flagged as malicious behavior by older or less sophisticated systems. Let me share a few examples:


VPN usage

     Work-from-Home has exploded during the pandemic in white collar professions. In fact, a summer Stanford study estimated as many as 42% of the labor force was working from home full-time. Companies have adapted to this reality while raising their technical security game for remote devices. Virtual Private Networks (VPNs) are very commonly used to protect company data on devices logging in via household ISPs like Comcast and AT&T. Since fraudsters often use virtual networks as masks for their true device location and markers, this is a common misread for fraud flagging. Missing this nuance means your survey could systemically exclude 42% of the labor force. It’s happening right now and for some researchers, it’s drastically skewing the insights. Fortunately, CleanID understands and accommodates this nuance.

Mobile Usage

     When using a mobile phone, your IP address often routes through a cellular network that identifies location differently than your actual geography. For example, as I write this, my cell phone is in Cleveland, OH, but my IP points to Knoxville, TN. CleanID understands this. However other tools don’t, which introduces a massive bias potential.  

Insufficient IP Blacklist Usage

     IP Addresses often get renewed or refreshed in time.  It’s similar to how pay-as-you-go cell phone numbers often jump back into rotation. Any effective fraud prevention system must account for this by ensuring any blacklist check it uses is comprehensive, current and global. This represents an uncomfortably high level of bias if mismanaged.  

In our collective zeal to protect data and ensure only the best data quality, it is vital that researchers constantly investigate the relevancy of all approaches and tools.  The balance is to stay ahead of emerging threats, while avoiding systemic exclusion of large segments of the population. Let’s keep out the cheaters, but let’s also make sure real consumers can have their voices heard in our surveys. You can have both!

Popular Articles
© Copyright 2021 OpinionRoute. All rights reserved.