पाठशाला Pathshala · ग्राहक Grāhak, The customer · Lesson 10 · Start

Surveys that do not lie to you

A survey is the cheapest way to hear what a thousand people think and the easiest way to hear what you hoped. How to measure behaviour and willingness to pay, and spot the questions that produce false comfort.

Pathshala, The Founder Library · 11 October 2026 · 7 min read

A marker beside a row of hand-ticked circles on white paper — Photograph: Towfiqu barbhuiya · Pexels

A survey is the cheapest way to hear what a thousand people think and the easiest way to hear what you hoped. The difference between the two is mostly in the wording, a little in the sample and almost never in the software.

This lesson sets out what a survey can and cannot do, the question types that produce false comfort, how to ask about behaviour and price, how many responses you need, and a pre-flight check to run before anything is sent.

What a survey can and cannot tell you

A survey counts. It tells you how common something is that you already know exists: how many of your users reconcile payments by hand, how often, with which tool. It is weak at discovery, because a closed question can only return the answers you thought to offer. That makes the order of work fixed. [Interviews](/library/the-customer-interview-done-properly) and [watching](/library/field-research-watching-instead-of-asking) find the problem and the words customers use for it. A survey then measures how widespread it is.

Pew Research Center, which writes survey questions for a living, has the cleanest evidence of why the order matters. In a 2008 poll asking for the most important issue in choosing a president, 58% chose the economy when it was offered in a list and 35% volunteered it when the question was open. Only 8% of the closed-question respondents gave an answer outside the list, against 43% of the open-question group. The list did not measure opinion. It shaped it. Pew’s own remedy is the founder’s too: pilot the open version first, then build the closed options from what people actually said.

The questions that produce false comfort

Agree or disagree. “Managing credit is a challenge for my business: agree or disagree.” People tend to agree with statements put to them, and Pew notes the tendency is stronger among less-informed respondents and when an interviewer is present. Offer a choice between two alternatives instead: “Which is closer to your experience: most credit customers pay on time, or I spend time every week chasing them?”

Several blank clipboards laid out on a pastel surface. — A blank form looks neutral. The wording written on it decides most of the answers before anyone picks up a pen. Photograph: DS stories · Pexels

Two questions in one. “How satisfied are you with our pricing and support?” has no honest answer when the pricing is fine and the support is not. Pew calls these double-barrelled and the fix is to split them.

The flattering question. People overstate what reflects well on them and understate what does not. Pew lists overstated church attendance, charitable giving and voting, and understated drinking and tax evasion. In a founder’s survey the flattering answers are “I review my accounts weekly”, “I would switch for a better product” and “I always read the terms”. Make the less flattering answer easy to give: “Some weeks things come up and the books wait. In the last month, did you review them every week, some weeks, or not at all?”

The order effect. Earlier questions colour later ones. In a December 2008 Pew poll, 88% said they were dissatisfied with the way things were going in the country when the question followed one about the president’s job approval, against 78% without it. Ask about frustration with current tools just before overall satisfaction and you will measure the frustration twice. In self-administered surveys people favour the first options in a list, so randomise any list without a natural order and keep it to four or five options.

The prediction. Jakob Nielsen’s example is a survey in which 50% of respondents said they would buy more from e-commerce sites offering 3-D product views. They were reporting that 3-D sounds appealing. Any question that begins “would you” measures the appeal of the idea, not future behaviour.

Ask about behaviour, with a time frame

Every question that matters should be about something the respondent did, within a period short enough to remember. Not “how often do you order groceries online” but “in the last thirty days, how many times did you order groceries online?” with bands to choose from. Not “is reconciliation a problem” but “last month, roughly how many hours did you or your staff spend matching payments to invoices?” Not “which features matter” but “which of these did you use in the last week?”

Past behaviour with a time frame has three virtues. It has a true answer, so politeness has less room to work. It can be compared with data you hold, which lets you check the survey against reality for the users you can see. And it gives you a frequency and a cost, the two numbers that decide whether a problem is worth a product.

Willingness to pay without asking “would you pay”

“Would you pay ₹499 a month for this?” is the most reassuring question in the founder’s survey and the least informative. It costs the respondent nothing to say yes. Three better instruments sit in ascending order of truth. Current spend: “What did you spend last month on this, in money or in someone’s time?” The answer is a ceiling and an anchor. A choice between priced options: show two or three real plans at real prices and ask which they would choose, including “none of these”. A choice forces a trade-off that a single yes does not. A real offer: end the survey with a pre-order or a paid pilot and a payment link. The share who pay is the only willingness-to-pay figure that will survive contact with the market.

In India two cautions apply. A survey in English samples the users comfortable in English, so translate it for the segment you mean, not the one you sit with. And an incentive large enough to matter, a recharge or a voucher, attracts people who answer for the incentive. Keep rewards small and send them after a completed response, not before.

Who answers, and how many you need

Margin of error falls with the square root of the sample, so the first hundred responses buy most of the precision and each further hundred buys less. Pew’s methodologist Andrew Mercer gives the reference points: about ±3 points on a random sample of 1,067, but about ±8 points on a subgroup of 160 within it, and ±16 on the difference between two answers within that subgroup. Founders rarely act on the whole sample. They act on a segment: the paying users, the clinics with more than three doctors, the stores in tier-2 cities. That segment is a subgroup, and its margin is the one that matters.

The bigger risk is not the margin but the sample. A link posted in a founders’ WhatsApp group and forwarded by friends reaches people who know you, like you and are not your customers. No number of responses corrects that. Send to a defined list, record how many you sent and how many answered, and compare the respondents with the list on anything you already know, such as city, plan or tenure.

A survey that asks what people would do measures how much they like the idea. A survey that asks what they did last month measures the market.

A worked example: home bakers in Mumbai

A founder building ordering and payment software for home bakers has interviewed fifteen bakers and watched three take orders on a Saturday. She has a hypothesis: bakers lose orders because enquiries on Instagram and WhatsApp go unanswered during baking hours. She surveys three hundred bakers in Mumbai from a list built through two baking-supply shops.

The first draft asked whether missed messages were a problem (agree or disagree), whether they would use an automatic reply tool and whether they would pay ₹299 a month. It would have returned a comfortable majority on all three. The second draft asks how many order enquiries they received last Saturday, how many they replied to within an hour, how many turned into orders, and what they spent last month on anything that helps them take orders. It ends with a real offer: a one-month paid pilot at ₹299 with a UPI link.

Sixty-two percent report replying to fewer than half of Saturday enquiries within an hour, a margin of about ±5.5 points on three hundred responses. The segment she cares about, bakers taking more than twenty orders a week, is sixty of the three hundred, and on that segment the margin is about ±12. She reports the segment figure with its range. Twenty-two bakers pay for the pilot. That number, not the sixty-two percent, goes on the first slide.

A pre-flight check for every survey

Before sending, read every question aloud and test it against six lines. It asks about something the respondent did, with a time frame. It asks one thing. It offers alternatives rather than a statement to agree with. Its options are randomised where they have no natural order and number no more than five. Its least flattering answer is easy to choose. And the survey ends with a real commitment to measure: a pre-order, a pilot, a call booked. Then send it to ten people from the list, read their answers, fix what confused them and only then send the rest.

Afterwards, write the response rate, the segment sizes and the margin on each segment next to every number you report. A number with its margin beside it is a measurement. A number without one is an opinion that looks like data.

The margins here assume random samples. Most founder surveys are not, so treat every figure as an upper bound on precision and say so when you present it.