पाठशाला Pathshala · संचालन Sanchālan, Operations · Lesson 15 · Build
Vanity metrics and the numbers that lie to founders
A number that cannot go down cannot warn you. Audit the dashboard with three questions, replace every cumulative and gross figure with one that can fall, and write down what each word on it means.
Pathshala, The Founder Library · 11 October 2026 · 7 min read
The most dangerous number on a founder’s dashboard is not a wrong one. It is a correct one that can only go up. It rises in good months and in bad ones, it photographs well for investors, and on the day the business starts to shrink it says nothing at all.
This lesson gives three questions that expose a vanity metric, the numbers that fail them most often in Indian startups, a figure that shows how a shrinking business can still draw a rising chart, the honest replacement for each common offender, and a quarterly audit that keeps the dashboard truthful.
Three questions every number must pass
Eric Ries, who popularised the term, described vanity metrics in a 2009 essay as numbers that might make you feel good but do not offer clear guidance for what to do, and argued that the only metrics entrepreneurs should invest energy in collecting are those that help them make decisions. That definition turns into three questions a founder can ask of any line on a dashboard.
Can it go down? A number that only accumulates, such as total downloads, total sign-ups or total orders since launch, cannot register a bad month. Does it have a time window and a denominator? “Orders” means nothing; “orders this week from customers acquired in the last ninety days, per active customer” can be compared, trended and owned. Would a different value change what the company does next week? If the number could double or halve and no meeting would end differently, it is decoration. A number that passes all three is worth a place. A number that fails any one of them belongs, at most, in the press release.
The numbers that lie most often
Cumulative users, downloads and sign-ups. Andreessen Horowitz’s 16 Startup Metrics is blunt: cumulative charts by definition always go up and to the right for any business that is showing any kind of activity, and they can do so even when a business is shrinking. Registered users. Every person who ever entered a phone number for an OTP, including those who never came back. Gross merchandise value before cancellations, returns and discounts. In Indian e-commerce, where cash on delivery is common, an order placed is not an order delivered, and an order delivered is not an order kept; a GMV figure that counts the first is counting hope.
Unweighted pipeline. The total value of every open opportunity in the CRM, including the ones nobody has spoken to in four months. Letters of intent, memoranda of understanding and pilots. Each is a promise to consider paying, and enterprise buyers in India sign them generously. Run-rate from the best period. The best day multiplied by 365, or the festive month multiplied by twelve. Followers, impressions and page views. Useful to a marketing team as inputs, misleading as evidence that anyone wants the product. None of these numbers is false. Each is true about something other than the health of the business.
How a shrinking business draws a rising chart
The mechanism is simple enough to see in one figure. A consumer app acquires users every month; some become active; a share of the active users leave each month. Cumulative sign-ups add every new user and subtract nobody. Monthly active users add the new ones who activate and subtract the ones who leave. When acquisition slows or churn rises, the two lines part company, and only one of them notices.
At the default settings, new sign-ups fall four per cent a month and a quarter of active users leave each month. Over the last six months the cumulative line rises by more than a third while active users fall by about a fifth. A board shown the first line approves the hiring plan. A board shown the second asks why active users peaked in month eight.
A number that cannot go down cannot warn you. Put the ones that can at the top of the page.
Replace each one with a number that can fall
Deleting a vanity metric leaves a gap that someone will fill with another. Replace it with its honest partner instead. Cumulative sign-ups become new sign-ups this month and the share still active in week four, read by [cohort](/library/cohort-analysis-for-founders-not-analysts). Registered users become monthly active users by a written definition. GMV becomes net revenue: delivered orders, after returns, cancellations and discounts, at the company’s take rate, with return-to-origin shown separately. Pipeline becomes weighted pipeline, each opportunity multiplied by its stage’s historical conversion, with anything untouched for sixty days removed. LOIs and pilots become paid pilots and their conversion to contracts. Run-rate becomes the last full month’s recurring revenue multiplied by twelve, and nothing else. Total customers become logo retention and net revenue retention.
Paul Graham’s advice in Startup = Growth gives the ordering: the best thing to measure the growth rate of is revenue, and the next best, for startups that are not charging yet, is active users. Put those at the top of the [weekly page](/library/weekly-metrics-review-one-page-one-hour) and let every other number explain them.
Definitions are where honest numbers go wrong
A metric can pass all three questions and still mislead if its definition drifts. The a16z list notes that companies have almost unlimited definitions of what active means, and asks only that a company be clear about its own. Write a definitions page and keep it with the dashboard. Active: a user who completed the core action, say placed an order or sent an invoice, in the calendar month; opening the app does not count. Customer: an account that paid in the period. Revenue: recognised, net of GST, refunds and credit notes. Churn: an account that paid last month and did not this month. Change a definition only with a dated note, and restate the history so that the trend survives.
Ratios can be vain too, because a ratio improves whenever its denominator shrinks for the wrong reason. Conversion rises when paid traffic is cut and only the most motivated visitors remain. Average order value rises when small buyers stop buying. Net promoter score rises when the unhappy stop answering the survey. Gross margin rises when a low-margin product line is quietly discontinued, which may be right but is not an improvement in the business that remains. For every ratio on the dashboard, show the numerator and the denominator beside it, as absolute numbers, for the same period. A ratio that improved while both its parts fell is reporting a smaller company, not a better one.
Graham’s essay also names the subtler failures in a footnote: counting inactive users as active, paying more for users than they are worth, releasing invites at a controlled pace to smooth the curve. His point is that these tricks hurt the founder more than the investor, because they corrupt the signal the founder uses to steer. An investor in diligence will recompute the numbers from raw data anyway. The founder who believed the dashboard is the one who is surprised.
A worked audit
An illustrative Jaipur consumer company selling home décor online opens its board dashboard. Line one: 4.2 lakh app downloads, up eighteen per cent this quarter. It cannot go down; it is replaced by monthly buyers, which turn out to have fallen six per cent. Line two: GMV of ₹3.1 crore for the quarter. It counts orders placed; after cancellations, return-to-origin on cash-on-delivery orders and customer returns, delivered and kept revenue is ₹2.2 crore, and the fall is concentrated in two states where cash-on-delivery runs highest. Line three: 1.1 lakh registered users, replaced by the share of each month’s new buyers who buy again within ninety days, which has been flat at nineteen per cent for three cohorts. Line four, Instagram followers, moves to the marketing team’s page. The new dashboard has fewer lines and worse news. Within a month the company has cut cash on delivery in the two states with the highest return rates and moved spend toward the channel whose buyers repeat.
The quarterly dashboard audit
In the first week of each quarter, take the dashboard and run every line through the three questions in a single meeting. For each line that fails, write its replacement and the person who owns it. Check the definitions page against how the numbers were actually computed this quarter; where they differ, fix the computation or date the change. Remove any line nobody cited in a decision during the quarter. Then read the top three numbers aloud and ask whether each one could have warned the company of a bad month. If any could not, the audit is not finished.
The companies and figures in this lesson are illustrative. Nothing here is investment advice.