Waiting longer will not tell you whether it worked.

SECTION 01The question

Prices went up in March. Or a new salesperson started, or the ads changed, or the shop started opening on Sundays. Six weeks later sales are up eight percent. Did it work?

Every owner asks this, and the usual answer is a length of time. Give it a month. Give it a quarter. Wait and see.

The real answer starts somewhere else. It depends on one number: how far their sales move on their own, with no single change being tested. A result smaller than that is not a result. It is a month.

So we measured it, on a real business, every way an owner might look.

SECTION 02One real business, every sale for two years

The University of California, Irvine publishes the complete transaction file of a real UK online retailer: gift-ware, many of its customers wholesalers, every line of every invoice from December 2009 to December 2011.[1] That is 1,067,371 lines.

It is a business squarely the size we work with. Net sales, after cancellations and leaving out postage and fees, came to £9,290,284 in its first 52 full weeks and £9,528,358 in the next 52, up 2.56%. A steady year, in other words, from 5,827 customers.

Then one question asked a few hundred times. Take any window, one week, two, four, eight or thirteen. Compare it with the obvious thing to compare it with. How big is the move? Every possible window across the 104 weeks, stepping a week at a time, and the Christmas shutdown left out because nobody mistakes Christmas for a result. A month, from here on, means four weeks.

The figure we report is the line that nine ordinary windows in ten stay under. Call it the band. A change has to land outside the band before it means anything.

SECTION 03A day tells you almost nothing

Start with the smallest window. Each trading day against the same weekday a week earlier, 562 of them.

The typical day moved 28.34%. Nine days in ten moved by less than 88.81%, which means one day in ten moved by more. A good Tuesday is not information. It is a Tuesday.

None of that is surprising. What follows is.

SECTION 04Against last month, waiting makes it worse

The comparison everybody makes is this month against last month. It is the one most monthly reports print.

Here is what it says in an ordinary year. January 2011 came in -50.08% on December. April came in -29.03% on March, and May came in +51.62% on April. September came in +44.23% on August. A business growing 2.56% a year swings by more than a quarter, and sometimes by half, from one month to the next.

Against the weeks just before, the band for one week is 44.96%. For four weeks it is 44.82%. Then it goes the wrong way: 54.47% at eight weeks and 63.69% at thirteen.

Against the weeks before, waiting longer makes the answer harder to read, not easier.

The reason is in the chart. A longer window reaches further into a different season, so the comparison measures the calendar more and the change less. A quarter of autumn against a quarter of summer is not a test of anything. This is the whole reason the Census Bureau runs a research programme on removing the season from a series, so that its movements are "easier to analyze over consecutive time intervals".[2]

SECTION 05Against last year, time starts to help

Compare the same weeks a year apart and the season mostly cancels. Now waiting helps: the band is 38.07% for one week, 21.58% for four, 15.27% for eight and 12.16% for thirteen.

Even so, look at where a month reading "up five percent" lands: in the thick of the cloud. 32 of the 45 ordinary four-week windows moved more than that against last year, with no single change behind them. A month up twenty percent is near the edge of the cloud and still inside it.

This comparison has a price, too. It needs a full year of history before it can be made at all, and a whole year of other things changing in between: a new competitor, a lost customer, Easter on a different week.

SECTION 06Against yourself, in the same weeks

There is a third comparison, and it is the one a data team reaches for first: change one half of the business and leave the other half alone. Then compare the two halves in the same weeks. Same season, same weather, same economy. The untouched half shows what would have happened anyway.

We tested it on this business with nothing changed at all. Split the products into two random halves, and measure how far the ratio between the halves moves from one window to the next. Twenty-five different random splits, so no single lucky one decides it.

Split by products, the band is 17.98% for one week and 11.73% for four. That is narrower in one week than last year's comparison manages in four, and it needs no history at all. At thirteen weeks it is 8.70%.

Split by customers and the picture changes completely: 62.03% for a week, 28.45% for four. Worse than last year's comparison. Many of this business's customers are wholesalers, and its ten biggest buyers alone are 13.93% of sales. Where one big order lands decides which half wins the week.

Which half you hold back matters as much as holding one back.

That is not something anyone could know in advance. For a shop with thousands of small customers and a few dozen products, it might run the other way. It has to be measured on the business itself.

SECTION 07The table to keep

Here is everything in one place. Each figure is the band: nine ordinary windows in ten move less than this, with no single change behind them.

Read it from the size of the change backwards.

A 20 percent change clears the band inside a week against untouched products, and inside eight weeks against last year. A 10 percent change takes between eight and thirteen weeks against untouched products, and does not clear the band inside a quarter against last year. A 5 percent change did not clear it inside a quarter by any comparison we measured.

Put a familiar change through it. A 6 percent price increase that keeps every customer adds 6 percent to sales. The narrowest band anywhere in this table is 8.70%. In this business, that price increase is invisible in every comparison, in every window up to a quarter.

SECTION 08Three ways to read a change

None of this means a change cannot be measured. It means the measuring is designed before the change, not after it. There are three real doors, each with a cost.

Wait for last year. No extra effort, and it works for big changes. It costs a year of history and months of waiting, and anything else that changed in that year gets credited to you.

Hold half back. The fastest and clearest read available, and the only one here that reads a 10 percent change inside a quarter. It costs leaving half the business unchanged for a while, and choosing which half takes a measurement of its own, because the wrong split here was worse than no split at all.

Decide on judgment, knowingly. Some changes are right whether or not they show up in a quarter's sales. Knowing the band at least means a bad month is not mistaken for a bad decision, and a lucky one is not mistaken for proof.

Which one fits depends on the change and on what you know about your customers that no file can show. Your call.

SECTION 09The point

The band in this piece belongs to one gift retailer. Yours is different, and it is sitting in your data right now: every sale already carries a date, a customer and a product, which is everything this took.

Every verdict on a change is a comparison against this number, whether anyone measured it or not.

A data team measures it before anything changes, because it is the one figure that decides what every later result means. Another change read the long way round, rain this time, is worked through here.