The Anatomy of a Bad Survey Question

The Anatomy of a Bad Survey Question

What happens when a survey question leads respondents toward a specific answer? Consider asking, "How short was Napoleon?" This phrasing assumes he was notably short and pushes respondents to agree, regardless of historical fact. Such leading language distorts feedback before it’s even collected. Even subtle wording shifts can dramatically alter results—Pew Research found that adding a clause about potential U.S. casualties in Iraq dropped favorability for military action from 68% to 43%, proving how fragile survey data can be when questions aren’t neutral.

For service businesses using tools like CallMyCustomers to gather post-service feedback, flawed questions mean reacting to illusions. A double-barreled question like "Please rate the speed and accuracy of our response" forces customers to evaluate two distinct traits with a single score, making it impossible to know which aspect drove their rating. Ambiguous phrasing—such as using internal jargon or vague terms like "satisfactory" without definition—leads to inconsistent interpretation. When response options don’t cover all possibilities or overlap (e.g., "often" and "sometimes"), respondents must choose inaccurately or disengage entirely.

These flaws compound: leading questions introduce bias, double-barreled items muddy analysis, and unclear language erodes trust. Together, they turn survey data into a shaky foundation for decisions about scripts, offers, or customer reactivation strategies. Before sending another review request, audit your questions for these pitfalls—because in repeat-revenue work, acting on misleading feedback risks reconnecting with customers based on what you wanted to hear, not what they actually experienced.

Leading, Double-Barreled, and Ambiguous: The Three Most Common Wording Flaws

Leading questions subtly steer respondents toward a desired answer by embedding assumptions or emotionally charged language. For instance, asking "How excellent was our technician's service?" presupposes excellence and pressures participants to agree, especially in service contexts where social desirability bias is strong. This flaw compromises data integrity by inflating positive ratings and masking genuine customer sentiment. According to Pew Research Center, even minor wording shifts can dramatically alter responses, as seen in their end-of-life survey where 51% favored doctor-assisted dying when phrased as "giving terminally ill patients the means to end their lives," but only 44% agreed when framed as "assisting in committing suicide"—demonstrating how language shapes perception on identical topics.

Double-barreled questions combine two distinct ideas into a single query, making it impossible for respondents to evaluate each component fairly. A common example in home services is asking clients to "rate our speed and accuracy" after a repair visit—someone might praise the technician’s punctuality but criticize the fix quality, yet have no way to express that split judgment. This flaw creates ambiguous data that cannot be reliably interpreted, as noted by Nielsen Norman Group, which explains such questions force respondents into inaccurate compromises. Splitting these into separate questions—one for timeliness, another for workmanship—ensures each dimension is measured independently and honestly.

Ambiguous or jargon-heavy wording reduces comprehension and introduces measurement error, particularly when service businesses use internal terms customers don’t understand. Asking "How satisfied are you with our HVAC system’s SEER rating?" assumes familiarity with technical efficiency metrics most homeowners lack, leading to guesswork or skipped questions. Similarly, vague prompts like "Was our service adequate?" leave interpretation wide open—does "adequate" mean functional, satisfactory, or exceptional? As SurveyMonkey highlights, unclear language reduces completion rates and skews results because respondents interpret questions differently. Using plain language—such as "How well did your heating system work after our visit?"—ensures clarity and aligns with what customers actually experienced. For businesses using review requests to gather feedback, avoiding these flaws means designing questions that reflect real interactions, not internal assumptions. A plumbing company, for instance, should ask about leak resolution or cleanup quality rather than hypotheticals like "Would you recommend us if we offered 24/7 emergency service?"—which predicts future behavior unreliably. Grounding questions in actual service encounters, verified through the client-approved scripts CallMyCustomers uses, yields actionable insights that drive meaningful improvements in retention and satisfaction.

Broken Response Options and Surveys That Ask Too Much

Sometimes the question is fine — it's the answer choices that break the survey. When response options overlap, leave gaps, or demand absolutes, respondents are forced into answers that simply aren't true, and your data quietly rots.

Pew Research Center's methodological guidance is blunt: closed-ended questions should include all reasonable responses (exhaustive), and options should be mutually exclusive, with no overlap. Nielsen Norman Group echoes the warning, noting that answer options that are not all-inclusive or not mutually exclusive force respondents into inaccurate choices. Customer Thermometer flags both "non-mutually-exclusive answer options" and "incomplete answer options" among the most common mistakes it sees in real customer surveys.

A related trap is the absolute yes/no question. SurveyMonkey points out that questions using words like "always" or "never" paired with Yes/No options force unhelpful answers — almost nobody's experience is truly absolute, so the response is either a lie or a coin flip. Graduated scales and a "not applicable" escape hatch fix this.

Then there's the sin of asking too much. Genroe documented a 58-page airline survey with 75 questions — and fully 25% of those questions asked for data the airline already had. A Club Med survey ran 25+ pages with roughly 100 questions. As Genroe puts it, your customer never owes you a response.

The numbers on forced answers are lopsided, too. Making questions mandatory to protect against incentive-gamers backfires badly: roughly 3% of respondents game incentives, while 97% are annoyed by forced answers, according to the same real-world analysis. You're irritating nearly everyone to catch a tiny minority — and the irritated ones either abandon the survey or answer randomly.

The fix is straightforward discipline:

• Cut every question you can answer from your own records — CRM data, booking history, point-of-sale records.
• Test whether options overlap or leave gaps before you send anything.
• Replace always/never yes-no questions with graduated scales.
• Keep the survey short enough to respect the respondent's time.

This is why CallMyCustomers segments a client's list before any outreach — recency, old quotes, lapsed memberships — so follow-up messages ask only what the business genuinely doesn't know. A post-service review request built on existing data takes a minute to answer, not an afternoon.

How to Fix a Bad Survey Question: Neutral, Single-Idea, Experience-Based

The good news is that every bad survey question has a fix — and most fixes take minutes, not a redesign. The Pew Research Center's methodological work shows why the effort matters: even small wording differences can substantially affect the answers people provide. One 2003 experiment found support for military action in Iraq dropped from 68% to 43% simply by adding a clause about potential casualties.

Split double-barreled questions. If a question measures two things at once, respondents can't tell you how they weigh each element — and neither can you. SurveyMonkey's example, "How satisfied are you with the pay and benefits of your job?" becomes two clean questions: one on pay, one on benefits. Genroe's real-world example — "rate the speed and accuracy of our response" — forces the customer to answer only one of the two ideas. Speed and accuracy deserve separate questions.

Rewrite leading questions neutrally. Replace "How short was Napoleon?" with "How would you describe Napoleon's height?" The same discipline applies to customer feedback: instead of "How excellent was our service?", ask "How would you rate our service?" Nielsen Norman Group warns that leading language pressures respondents into socially desirable answers rather than honest feedback — the opposite of what a survey exists to collect.

Make response options exhaustive and mutually exclusive. Closed-ended questions should include all reasonable answers with no overlap, and always add an opt-out:

• Include "Not applicable" or "No opinion" when a question may not fit everyone.
• Replace absolute Yes/No items with graduated scales.
• Check that no two options overlap, which forces arbitrary choices.

Customer Thermometer's guidance is blunt: always provide an opt-out to avoid bad data.

Ask about actual recent experience, not hypotheticals. SurveyMonkey notes that hypothetical questions are unreliable because most people cannot accurately predict their behavior in imagined situations. "How likely are you to refer us?" produces wishful thinking. "Did you refer anyone in the past six months?" produces data.

This is why CallMyCustomers reviews every script and survey question with the business owner before anything reaches a customer. A question that skews responses doesn't just waste the survey — it misinforms the follow-up campaign built on it. Approval-first means flawed questions get caught before they generate flawed answers.

Put Better Questions to Work: From Feedback to Booked Appointments

Knowing what makes a survey question bad only pays off when you put better questions to work. For a US service business — an HVAC company, a dental clinic, an auto repair shop — the goal isn't research for its own sake. It's turning honest feedback into follow-up that books real appointments.

Start by keeping it short. The Nielsen Norman Group puts it plainly: "keep them short" and ask only what's essential. A peer-reviewed review in the Journal of Graduate Medical Education reaches the same conclusion: a shorter survey is almost always better than a longer one. Real-world failures back this up — Genroe documents an airline survey that ran 58 pages with 75 questions, and a quarter of those questions asked for data the airline already had.

Before anything goes out, test it. The research recommends cognitive interviews and pilot testing with your target audience before deployment — thoroughly evaluating surveys catches leading wording, double-barreled questions, and answer options that don't fit before your customers do. A quick read-through by someone outside your office will surface most of these flaws.

Once clean feedback comes in, route it somewhere useful. A post-service survey that ends in a dead file is a wasted touch. Instead, let responses feed directly into follow-up campaigns:

• Happy responses trigger a Post-Service Follow-Up & Reviews request — especially powerful given that 72% of customers tell at least six people about a good experience, per feedback research.
• lukewarm responses feed a Customer Survey-to-Offer campaign, giving the customer a relevant reason to rebook.
• Negative responses get a personal recovery touch before the customer goes dormant — most customers forget a business within about 12 months.

This is exactly how CallMyCustomers approaches the work: the business owner approves every question, script, and offer before anything is sent, so the survey and the follow-up feel like one coherent conversation rather than a marketing blast. Replies route straight into your normal booking process, and the messaging stays in your business's name.

The stakes are real. The same feedback research found that 13% of unhappy customers share their negative experience with 15 or more people — and a bad survey question is often what turns a mild complaint into a public one. Good questions get honest answers; honest answers get followed up; follow-up gets booked work.

Want to see what your list can actually produce? Get a free list review from CallMyCustomers — you'll see your rate, setup, and what better questions plus approved outreach could turn your past customers into booked jobs, before you spend a dollar.