It has seemed to me lately that I'm getting far fewer reviews on Amazon than I used to, considering the number of units I sell. The thought has been nagging at me for some time, and I'm not alone — others seem to be observing the same thing and speculating about causes.
So — you know me. Time to actually crunch a few numbers and try to see if it's true and, if so, why I think it might be happening.
I started running Amazon AMS ads about 15 months ago, and my units sold have shot up gratifyingly. But not my reviews. My ratings are stable and the reviews I get are much the same as they've always been, but there are just fewer of them than I would expect.
First steps — collect the data
I've been meaning to copy my reviews off the retailer sites, especially Amazon, lest they vanish in one of the periodic Amazon purges. So far I've been lucky and haven't lost any, but that can change. It's useful to have them available, not just for ratings on the retail sites, but also as sources of blurb and other publicity text from real readers.
I checked my retailers and confirmed that, yep, I have almost no reviews except on Amazon, and almost all of those on Amazon USA, of course. That made it easy.
I set up a spreadsheet like the one I use for tracking unit sales to track reviews: month/year, source, rating, product, retailer, headline, review text, etc. Then it was off to the races with pivot tables.
Do I have enough data?
I don't make any big push for reviews, just a modest suggestion in the backmatter of the books. I don't have a ton of reviews, but they do keep coming in (slowly), so I'm going to assume there are enough for some valid conclusions. In any case, I don't have any special marketing that might confuse results.
Next — connect the review data to the units sold data
I put a worksheet up with one pivot table for the reviews-by-month, and another with the units-sold-by-month. Then I ran out the data for a comparison from the date my first book came out, in October, 2012.
A model to compare data
The question I wanted to answer was:
Has the percentage of reviews per units sold been declining lately?
Now, any measurement trial is improved if you can run more than one set of data for comparison and, as it just so happens, I have 2 completed fantasy series, each with 4 novels, the first book of which I'm been advertising on Amazon.
These series behave almost identically with regard to statistics. They get the same ratings (circa 4.5), they have the same amount of series read-thru, and they're both in the same genre. The second (wizard) is slightly more popular than the first (wild hunt) and has just about caught up — the same number of units sold for the first book of each. So this is as close to two identical test cases as one could hope for.
The rule of thumb for deciding how many reviews to expect used to be about 1-3% of units sold. It's easy enough to look at my cumulated reviews and cumulated sales to figure that out, but the question of interest is to see if that has changed over time. (Don't worry — there won't be any calculus questions on the test.)
Here's the first book of each series. They are scaled the same, and you can see that the second series doesn't start being published until after the first series was finished. AMS ads began shortly after the 2nd series was completed.
- (Trivial.) It takes a little while for the initial ratio of reviews to sales to settle down (jagged green line) for book 1 of a series as the next books are released soon afterwards.
- The red line which marks the number of reviews is on the same scale as the number of units sold, so it's difficult to see much movement. Based on the rule of thumb of 1-3%, that makes sense. The actual data has more than twice as many reviews for the first series as for the second, even though the number of units sold for the first books of these two series are almost identical. Still, you would expect the red line to at least approximate the shape of the blue line, even if 100 times smaller, but it doesn't, especially where it would be obvious — at the great run-up of the Amazon AMS ad-driven units sold.
- The percentage of cumulative reviews to units sold is more than twice as many for the older series.
But look at the green lines more carefully. For the older series, the percentage of cumulative reviews to cumulative units sold is almost flat from early 2014 to early 2017, essentially 3 years. For the younger series, there's a hint of something similar up to early 2017, as well.
But from early 2017 to the present, reviews are losing ground. As sales of units go up, the reviews posted do not keep pace. In fact, from that period to present, they decline by about 1 2/3 (not quite 2 times).
It becomes clear that the reason series 1 has a better percentage of cumulative reviews to units sold (a little more than 3%) is not because it's been around longer, but because it was a lot easier to get reviews before early 2017, so it starts its decline from a higher position. For series 2, the equivalent percentage is a bit less than 1.5%, because it didn't get to include earlier years.
So, I think I know “what”
I believe the apparent decline in review postings rate is real, and the two series corroborate each other well.
And I think I know “when”
There's fairly consistent evidence of a long-term (3-year) plateau followed by a hefty decline at a point that's similar for both series.
So now the question is “why?”
What started happening circa early 2017 at Amazon and has been getting worse? (Remember: I haven't lost any posted reviews, though it may be that reviews haven't reached me.)
If you think you have an answer, please reply in the comments.