Survey: Key questions pre-tested or drawn from validated instruments

What to Check

Survey questions can be poorly understood, differently interpreted across respondents, or subject to social desirability bias. Novel survey instruments must be pre-tested before deployment. Questions measuring established constructs (depression, trust, risk aversion) should be drawn from validated scales with published reliability evidence rather than invented ad hoc.

How to Check

1. Identify the key outcome and covariate measures. Were they:
   • From a validated scale: e.g., PHQ-9 (depression), CES-D (depressive symptoms), GAD-7 (anxiety), GHQ-12 (general health), World Values Survey trust module, Holt-Laury risk aversion task, WHODAS 2.0 (disability), SF-36 (health). If yes, cite the original validation paper.
   • Pre-tested with cognitive interviews: small-sample qualitative pre-test in which respondents “think aloud” while answering to identify comprehension problems.
   • Pilot-tested quantitatively: administered to a small sample to check distributions, skip patterns, and completion rates before main fieldwork.
   • Adapted from prior literature without revalidation: this is acceptable only if the adaptation is minor (translation, minor wording) and the context is similar.
2. For novel instruments measuring psychological or attitudinal constructs, check whether internal consistency (Cronbach’s α ≥ 0.7) or test-retest reliability is reported.
3. Check whether the question ordering and context effects are discussed (earlier questions affecting later answers).

Pass Condition

Key measures are either (a) drawn from published validated instruments with citation, or (b) explicitly pre-tested (cognitive interview or quantitative pilot). Novel instruments measuring latent psychological constructs report reliability evidence (Cronbach’s α or similar).

Failure Examples

1. Ad hoc trust measure: Paper measures social trust using a single novel question invented for the study, without any pre-test or reference to validated trust scales (e.g., WVS or GSS trust module). Claims strong results on trust. Fails.
2. No pre-test for complex instrument: Paper administers a 45-minute survey covering sensitive topics (income, health, sexual behavior) with no mention of pilot testing or cognitive interviews. Comprehension failures may be widespread. Fails.
3. Translated instrument without validation: Paper translates an English-language scale into Swahili for the first time and administers it without assessing cross-cultural validity or internal consistency. Fails.

References

• Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50(1), 537–567.
• Willis, G. B. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Sage.
• Nunnally, J. C. (1978). Psychometric Theory (2nd ed.). McGraw-Hill.