Step Up Your Survey Research


Most surveys are terrible. From poorly designed questions, to incoherent survey flow, to useless results, it’s no wonder data-driven organizations have so little faith in survey research. But this isn’t the fault of the tool, it’s because most surveys are built without adhering to some basic best practices, which once fixed can transform any survey from a zero to a hero. This blog post and the video below will show you how to create data-science quality surveys that provide unique and immediately actionable insight about your customers, competitors, and marketplace.

The Data Science Approach to Surveys

It’s a rarity for two Data Scientists of the “Academic” and “Computer Science” classifications to agree on anything (learn about the 4 types of Data Scientists here). However, one almost universal opinion they share is a complete lack of respect for surveys, believing them to be primitive and ineffective compared to other methods of data collection. In many cases, they are absolutely right. The most common causes of ineffective surveys include non-existent game plans for resulting data, cryptic/confusing questions, an unengaging user experience, and minimal pretesting. Establishing a clear connection between the information you need and how you plan to use it (S.M.A.R.T goals can easily fix this) ensures a sound strategy and immediately demonstrating legitimacy to survey takers through transparency and clear language will greatly increase the number of completed surveys. Integrating text piping and adaptive questions help customize the user experience and multiple pretests dramatically lower the possibility of encountering problems post-launch. Remedying these common mistakes is a very effective way of making your survey stand out, potentially becoming a great tool to compliment data science work, particularly observed problems.

Increasing Survey Engagement

If you’ve followed the four steps listed above, at this point you should have a polished, well thought-out survey. The next stage in the survey improvement process is fieldwork, beginning with a soft launch. This involves controlled, small batch experimental distributions of your survey done iteratively. A soft launch is an essential component of a successful survey, because no amount of pretesting will model everything respondents will do. The overall success of your soft launch can be gauged by 2 numbers:

  • Response Rates – This is the number of people your reached (impressions, clicks on ads, emails opened) divided by the number of people contacted, and measures your initial engagement success. It’s important to vary the solicitation channels used in your soft launch, and to track the performance of each. By taking this systematic approach, you can understand what’s happening in the response rates beyond the average and truly optimize your soft launch.
  • Completion Rates – This measures the number of surveys finished divided by the number of surveys started, and it’s at this point where you learn if your survey actually works or not. Patterns in question drop-offs and response times indicate which questions may be confusing to respondents. With this insight you must decide on either rephrasing the question or removing it completely. Additionally, strategically placing a free response question at the end of your survey gives respondents an opportunity to share their thoughts (positive and negative), so observe these intently.

After your soft launch is completed, it’s time to begin your full launch. Your full launch should be considered “The point of no return”, and involves things like spending the rest of your advertising budget as well as soliciting all remaining channels for potential respondents. However, even after full launch tracking metrics and reading feedback is still required to work out the possibility of undetected issues.

Blending Survey Data

As insightful as survey data is by itself, it’s impact can be exponentially increased by blending it with other data sources. But before attempting this, it’s crucial that you ensure your data is “analysis ready” by cleaning it. Carefully inspecting the respondents who sped through the survey, straight lined, or took it multiple times is the most effective way of identifying bad data points and removing them from your data set. Additionally, there are survey tools available(i.e. Qualtrics) that enable you to precode values and labels of your data before (or after) your launch process is completed, which greatly simplifies and shrinks the time you spend performing this process. After cleaning/removing these types of responses, you’re ready for analysis.The power of survey data can be increased substantially by connecting specific survey responses to other data you have on the same individuals, like observational data, direct product data, or customer records. In order to accomplish this, the data must be organized in a manner you are able to attribute an identifier to each individual, whether it be user ID, name, or email address. Doing this makes it possible to blend as many different data sets as you want.

Become a Data Demigod with our Free Lecture Series