SKOPOS-Digital-Offensive auf der GOR 2018
Vom 28.2. bis 2.3. feiert die General Online Research als internationale Fachkonferenz rund um das Thema Online-Forschung ihr 20-jähriges Jubiläum.
Ganz im Zeichen der Digitalisierung, freuen wir uns, unsere neuesten Erkenntnisse aus den Bereichen künstliche Intelligenz, automatisierte Text-Analyse und Mobile Research vorzustellen. Doch auch das wichtige Thema Qualität kommt nicht zu kurz. Deshalb diskutieren wir die Herausforderungen bei der Reproduzierbarkeit von psychologischen Studien und zeigen, wie man die Ergebnisse auch in die Welt der Marktforschung überführen kann.
Und das sind unsere Vorträge im Überblick:
Learning From All Answers: Embedding-based Topic Modeling for Open-Ended Questions
Speaker
Harms, Christopher 1,2 & Schmidt, Sebastian1
1 SKOPOS GmbH & Co.KG 2 Rheinische Friedrich-Wilhelms-Universität Bonn
Session
B2: Using Big Data Tools in „Small“ Surveys
Time
Thursday, 01/Mar/2018: 10:30 – 11:30
Location
Room 149
Relevance & Research Question
Open-ended questions in surveys pose a challenge to researchers especially in large data-sets. While for small surveys, manual coding is often a feasible and commonly used option, this becomes increasingly expensive with increasing number of participants and/or questions. Machine learning approaches can help to reduce costs while maintaining a high accuracy for practical purposes. In many projects, however, a code plan might not be available and a more exploratory approach is desired. Recent advances in unsupervised machine learning and natural language processing (NLP) allow to effectively explore a data-set containing answers to open-ended survey questions. More than quantitative classification measures, the utility for practical market researchers is of interested for us.
Methods & Data
We compare three different approaches to analyzing text data: (1) a part-of-speech-based extraction of keywords, (2) topic modeling through latent Dirichlet allocation (LDA) and (3) embedding-based Topic Modeling (ETM). We compare the results through expert evaluations on two different datasets: First a standard Twitter data-set in English that is regularly used for evaluation of text analytics methods, and second a data-set from a recent market research project in German. Market research professionals will rate the resulting information on how well the generated topics represent the underlying responses and how confident they are to make a decision based on the produced results.
Results
Results of the expert evaluations will be available in December 2018.
Added Value
Researchers and their clients can gain additional insights from open text data if they take an exploratory approach. Using machine learning technologies, this becomes feasible even for large data sets and when no pre-defined coding plan is available. Our comparison will show the utility of these methods in a real-world research context. Combining qualitative data from text analytics methods with quantitative data from the survey, allows for even more informative results.
We need to talk: Reproducibility in Online Research
Speaker
Harms, Christopher1,2
1 SKOPOS GmbH & Co.KG 2 Rheinische Friedrich-Wilhelms-Universität Bonn
Session
B9: Device Preference and Device Effects
Time
Friday, 02/Mar/2018: 12:00 – 1:00
Location
Room 149
Relevance & Research Question
Psychological research has undergone extra scrutiny in the recent years. Several projects to investigate the reproducibility of several findings and effects have shown that large parts of psychological research are not replicable in independent replication attempts (see e.g. paper on “False-positive psychology”, ”Reproducibility Project: Psychology”, replication attempts for social priming effects or “Power Posing”). For online research, both in academic and in business contexts, this is a critical observation as customers and researchers rely on the reliability of our results.
The debate about the complex causes and consequences is still ongoing. It concerns all parts of empirical research: theory, data collection and data analysis. This submission will give a brief overview on the replication crisis in psychology and how it translates to online research, since some problems (e.g. small, non-representative samples) are of minor relevance for online research while others are still relevant (e.g. post-hoc explanations of unexpected results). Especially the relevance of replicability in singular market research projects is discussed and how recommendations for online research need to differentiate between scientific investigations and commercial market research projects.
Results
Several recommendations have been proposed for psychological science and can be adapted for online research. These involve: pre-registration of research hypotheses, public access to raw research data, justification for alpha levels in traditional significance testing and alternatives to significance testings. I will provide an overview on the current recommendations and how they can be applied in research projects. Not all are equally adequate for all (commercial market) research projects, so the discussion needs to continue on the specific implementations to improve our work and needs to also involve representatives of commercial researchers.
Added Value
Commercial and academic recipients of online research results need to rely on the integrity, quality and soundness of our work. The recommendations provided here are a first, in parts easy-to-implement step towards a more reproducible way of doing research. Ultimately, more careful planning of studies and data analysis can help to increase the value of online research results.
Mobile Devices in online surveys: Drivers for participation on mobile devices and effects on data quality in using propensity score matching
Speaker
Schlickmann, Patrick & Schmidt, Sebastian
SKOPOS GmbH & Co.KG
Session
A10: Device Effects
Time
Friday, 02/Mar/2018: 2:15 – 3:15
Location
Room 248
Relevance & Research Question
Due to their widespread popularity and ever-increasing (technical) capabilities, Tablets and Smartphones constitute great advantages for gathering survey data. Especially with regard to presentation and input, mobile devices differ considerably from PCs – a characteristic that researchers are strongly advised to make use of when planning and designing a web-based survey.
Firstly, even though “mobile first” is often claimed to be mandatory for all kinds of surveys, reality is far more complicated as the mobile participation rate widely differs due to different target groups, sampling sources etc. This is why the first part of the study focusses on structural drivers for mobile participation rates.
Secondly, there are numerous theoretical reasons to assume that differences in data quality are caused by the device. However, as selection and measurement effects might lead to prone analysis, the question arises whether or not the device or correlated demographic and psychographic factors play a major role when it comes to differences in data quality.
Methods & Data
The present analysis towards data quality is conducted by using the GESIS Panel with data quality indicators like item nonresponse, answering behavior for open-ended and multiple choice questions, etc. To disentangle selection and measurement effects propensity score matching is used to create a quasi-experimental design. To assess drivers for mobile participation, a post-hoc analysis of 30 different online-surveys is conducted, taking into account different target groups, demographic criteria and sample sources.
Result
PCs and Smartphones are found to differ significantly with regard to data quality. However, no such differences can be found between PCs and tablets. Smartphone usage leads to higher item nonresponse and shorter open answers, as well as an increase in left-alignment and primacy effects. Additionally, it overall took Smartphone users longer to complete the survey.
Results for mobile participation drivers will be available by Mid December 2017.
Added Value
The authors will present key drivers for mobile online survey participation. That is, tangible guidelines on creating a significant share of mobile respondents will be given. In addition, specific measures, that are found to minimize device-specific data quality loss, will be highlighted – as well as survey layout optimization and phrasing.
Keine Angst vor Ihrem neuen Kollegen, der Maschine!
Speaker
Harms, Christopher
SKOPOS GmbH & Co.KG
Session
D10: Künstliche Intelligenz in der Marktforschung
Time
Friday, 02/Mar/2018: 2:15 – 3:15
Location
Room 154
