Bob Graff - MarketVision Archives - GRBN.ORG

Course Correction for Discrete Choice with Mobile Audiences

GRBN_News — Mon, 09 Apr 2018 06:05:20 +0000

Are traditional ‘best practices’ limiting the business potential for clients? It’s an important question but not often asked. Having a set of best practices gives us confidence in our processes for execution with the promise of a strong research process plan delivering high quality data. But what if your best practices require you to exclude a key group of research participants by removing mobile audiences? How does that impact the data you’re collecting and do the results adequately reflect the marketplace opportunity? A team of researchers in partnership with the GRBN (Global Business Research Network) are conducting research to assess the impact of excluding mobile audiences in pricing research. Three considerations driving best practice decisions for online research

Mobile survey participation is increasing but consistent representation is lagging
Survey platforms are improving but multiplatform designs remain too few
Sample representation and sourcing needs greater transparency

With the prevalence of mobile response and a renewed focus on the respondent experience, there’s an opportunity to revisit our assumptions for what research may be appropriate for certain devices.

Increase in Mobile Participation

No surprise here. We’re tied to our phones and there are inherent opportunities associated with that level of access. The percentage of participants responding to a survey request on a mobile device continues to climb, and the increase is even greater among younger audiences. While mobile access is on the rise, the focus should be on participant inclusion regardless of device.

Survey Platforms

The industry is responding to the challenge of improving the respondent experience but there’s more work to do to improve the consistency of presentation of surveys across devices. Many platforms continue to render surveys differently for mobile participants or not render adequately for participant accessibility. Researchers are doing a tremendous disservice to clients if they don’t optimize the respondent experience and aren’t focused enough on inclusion of sample through platform design.

Sample Representation

How is your data different because mobile audiences weren’t included in the sample? Sample companies have a responsibility to consult with clients about the potential inclusion or exclusion of groups due to poorly designed surveys. The exclusion of a portion of sample should never come as a surprise. Clients should understand the sample plan prior to research and be given time to make appropriate adjustments to maximize participation. The opportunity is to improve the respondent experience through better platform and survey design and focus on widening the net of available participants in research. Higher quality data is the outcome and participant inclusion is imperative.

Revisiting Discrete Choice and Mobile Participation

Assumptions have been made over the past several years that certain types of surveys aren’t a good fit for a mobile audience given screen size and platform limitations. The limitations are still there with certain types of research, but as the levers have changed we have an opportunity to revisit what is deemed ‘best practice’ for specific types of research. Discrete choice research presents some challenges and for many the default course of action has been to exclude mobile participants. Depending on the design, there can be a lot of information presented and too many options to display on a small screen. The presentation of questions can be inconsistent and could lead to different results by subsets of your sample. As we consider the potential for mobile audiences and discrete choice, two questions come to mind.

How does the discrete choice exercise render on a smaller screen?
What is the impact on data if we exclude mobile participants from sample?

The focus to date has been on point #1 – the rendering of the survey exercise on the screen. If the exercise is different by device, we exclude the smaller screens and control the presentation of the exercises by limiting participation to laptop and desktop respondents. It then becomes ‘best practice’ to default to exclusion of mobile audiences for discrete choice studies. But we haven’t paid sufficient attention to point #2 – how the exclusion of the sample impacts the findings from research. How are insights limited as a result? Thankfully platforms are improving to better support multiplatform surveys with a consistent display across devices. This means the presentation of the questions is the same on a phone as it is on a laptop or desktop, without the need to scroll or pinch to navigate the survey. We know this consistency is crucial to delivering data comparability. The improvement of online survey platforms is no small achievement. It makes it possible to offer research to participants previously deemed a ‘bad fit’ for the design. What was once not a ‘best practice’ might now warrant consideration.

Pricing Research and Unintended Consequences

What if ‘best practice’ of excluding mobile participation in a discrete choice pricing study results in more conservative pricing recommendations? Much the way we ask participants in discrete choice surveys to make ‘trade-off’ decisions over product features, researchers go through a similar series of ‘trade-off’ decisions to determine their optimal approach for executing research. Decisions regarding sample, programming, question types and exercises to include, incentives to offer, and other areas have a major impact on the potential for insights. In the case of pricing research, these decisions on execution, specifically on the inclusion or exclusion of mobile audiences, may fundamentally change the recommendations from research. Could pricing research without mobile audiences lead to more conservative or aggressive pricing decisions?

Research Plan and NEXT Presentation

Through our partnership with Mindbody, a B2B software provider serving the wellness and fitness community, we intend to better understand how pricing decisions may be affected by the inclusion of mobile audiences. Our team of researchers will present findings on this topic with the industry at the Insights Association NEXT Conference May 1^st, so we hope to see you there. Research Team: Andrew Cannon, GRBN Dyna Boen, UBMobile Lisa Wilding-Brown, Innovate MR Bob Graff, MarketVision Research David Lau, Mindbody Bob Graff MarketVision Research

The post Course Correction for Discrete Choice with Mobile Audiences appeared first on GRBN.ORG.

Flipping the Quality Discussion to Focus on Participant Inclusion

GRBN_News — Mon, 24 Apr 2017 07:14:34 +0000

The efforts to ensure quality in survey research have focused too heavily on the development of tools designed to detect and remove bad and/or fraudulent data. This approach is incomplete. As a full-service research company in business for almost 35 years we deploy the full arsenal of tools to detect and remove bad data. From digital fingerprinting to reliability tests to probability algorithms and open-end response validation and others, we cover a lot of territory to ensure we’re delivering a clean and reliable data set. And we’re not alone – most of the industry is engaged in catching cheaters and excluding poor data. It’s cost of entry in this business. Perhaps the industry is spending too little time on specific efforts to improve the survey experience through design and efforts focused on inclusion of underserved participant groups. We can flip the discussion and continue to push for better survey design and participant engagement, not relying simply on providing a better user experience but also ensuring specific design elements lead to more reliable data.

The Future of Surveys is not Mobile

Participants choose how and when they engage in research. Research organizations no longer make those decisions, but we do get the opportunity to optimize the participant experience. Focusing our collective efforts to provide the right experience for participants is our imperative and the purpose should be to facilitate survey completion across demographic groups and device types. The future of surveys is not mobile; the future is in designing a consistent experience to a participant regardless of device or platform. This means going beyond a simple Responsive Design solution for making surveys “mobile optimized” and instead focusing on specific designs which render consistently, providing the same question presentation, across devices. We began conducting research-on-research focused on multi-platform survey design 18 months ago, across project types and categories to determine optimal survey designs for multi-platform survey engagement. This process has resulted in data-supported designs which improve the participant experience and ensure sustainable data quality in online data collection. New designs ensure no difference in data across device types (laptop/tablet/phone) and reduce mid-term dropout rates by mobile-first participants, and allows us to get rid of troublesome questions types like grids to improve the experience.

ADA Accessibility and Participant Inclusion

Instead of looking for new ways to exclude participants, how can we bring more people into our surveys and encourage them to continue to participate in survey research? Do you know how many people with visual impairment or blindness are currently trying to complete your surveys? In the U.S. there are approximately 25 million people living with high levels of visual impairment or blindness and this number is expected to climb with the aging population. With ADA accessibility mandates expected in 2018 we need to start thinking about how to bring this group into the survey experience and how to do it in a way that goes beyond simple compliance for creating a good experience. It’s not easy. The challenges with survey design and for creating an engaging survey experience with the visually impaired and blind are significant and important for the future of representative survey research. Those with visual impairment require a high contrast solution while those with blindness need a solution which works with screen readers and braille pads, and the specific design solutions are unique. I would encourage you, if you haven’t already, to engage with this community to begin developing the solutions needed to be compliant and to ensure their needs for participant experience are being met. This is an under-served community when it comes to online participation, but they want to do research and we need them to be part of our research if we want true representation. And in this case and others, we simply need to try harder to create opportunities for research participation.

Participant Inclusion

So much of our efforts towards quality are focused on excluding participants. We’re starting to flip the discussion to engagement and focus instead on participant inclusion. Multiplatform design and ADA accessibility are two ways we’re moving the initiative forward. Better survey design with a renewed focus on the participant experience, and optimizing those elements which produce quality across projects and across categories should be our beacon for future industry efforts. Robert Graff Vice President at MarketVision Research, Inc

The post Flipping the Quality Discussion to Focus on Participant Inclusion appeared first on GRBN.ORG.