Visit Page
Skip to content

Amazon AMS Ads — A Test (Part 1)

Posted in Just for Writers, and Marketing

(This article continues here.)

For about a year and a half I had a good run with basic AMS Manual Keywords ads — those ad farms of thousands of keywords. Then, in the fall of 2018, something significant changed at Amazon, besides just the AMS reporting system upgrade, and my ads which had been running at a smallish but steady profit basically stopped working well.

At the time, I didn't concentrate on the issue, content to spend several months working on getting my feet wet with Facebook Advertising while my AMS ads ran feebly in the background (at least I wasn't losing money on them). Let's just say that two separate attempts at Facebook product ads ended unsuccessfully and I shut them down to rethink my approach.

So, there I was in June with no effective advertising running and, let me tell you, that is no place you want to be. In the absence of any recent new release, my organic sales on my primary (1st-in-series) books are, at best, less than 1 per day.

The very visible effect of advertising (vs organic sales/new releases)

But there's an upside to poor organic sales numbers — they make a great background to test against for advertising!

So I've rebuilt my AMS Ads from the ground up, and put together a rigorous sort of test which I wanted to share with you.

I'll pass over how I made a pile of almost 19,000 manual keywords. They range from authors to titles, series, categories, search strings, etc. Since AMS only allows a maximum of 1000 keywords per campaign, each product I wanted to advertise required 19 campaigns. Happily, part of Amazon's reorganization of AMS last fall included a portfolio concept, so I was able to group all 19 campaigns into a single portfolio for each product.

Currently I have 2 heroic fantasy series (4 books each), and I advertise the first book in each series, and the first 2-book bundle in each series. I sell fewer bundles, by far, than first books in series, but they're not negligible. They have weaker stats — harder to sell a bundle by an unknown author than a single book. In the past, the ads for the bundles have been ROI positive, but I now suspect they might not be.

To help you interpret the charts below, here are the names/abbreviations:

  • ANNWN-N1 (Novel 1 of The Hounds of Annwn)
  • ANNWN-B2 (Bundle of 1st 2 books of The Hounds of Annwn)
  • CHAIN-N1 (Novel 1 of The Chained Adept)
  • CHAIN-B2 (Bundle of 1st 2 books of The Chained Adept)

Now, the last time I ran AMS ads successfully, I was bidding $0.51/click. I briefly experimented with $0.75 last fall in a simple stab at seeing if that would help revive my ads and, while I generated a welcome number of sales, they were not profitable, even taking series read-thru into account, so I dropped the bid back down to $0.51 and left them alone.

Guesstimate of attribution of KDP Unit Sales to Organic, AMS Ads, Facebook Ads sources

Still, I had to start somewhere. Since the last time I looked, a bid of $0.51 wasn't doing much and at least a bid of $0.75 had generated some activity, if too expensively, I decided to start a formal test there. I would run the new (to me) “Dynamic Up/Down” method of ads (which adjusts the bid in either direction to chase clicks) at $0.75, with 4 portfolios of 19 campaigns, one for each of the 4 products listed above.

So, what happened?

I shut down the old ads so I could have a clean background to measure against, and used this as the source of KDP Units Sold for the products being advertised.

Test 1 – Dynamic Up/Down, bid $0.75, 3 days

TEST 1 – Dynamic Up/Down, bid $0.75, 3 days

There's a lot to look at here…

The numbers in peach come from the AMS reporting system, one line per product portfolio. The numbers in blue come from the new (beta) KDP graphic sales report (example above). At the time this picture was taken, 19 days had passed since the test ended.

My goal is to have a positive ROI (Return on Investment), comparing the ad spend to the series read-thru estimated earnings (which I would not receive for 2-3 months). When someone buys the first book of a series, I know how many additional books in a series will be bought (on average) — it's a very stable number for me. So I can tell what the purchase of the first book in the series is worth to me.

Note: for series read-thru, I use the cumulative lifetime full series read-thru percentage for each of the products. Now, if I can enhance my series read-thru via better marketing content and processes, and thus increase the percentage, it will have a significant impact on these profitability results.

I could use the most-recent full series read-thru percentage instead, but I find that can be distorted whenever there has been a recent push to move 1st-in-series books — it can take a while for rest-of-series to catch up. But keep an eye on gradual change over time.

Cost per Click (CPC) and Click-Thru Ratio (CTR)

My bid was $0.75, and the average CPC was $0.62. The CTR was a healthy 0.09%.

Units Sold and ACOS and ROI

AMS claims I only sold 5 units. KDP shows 20. While it took a few days for the units sold value to finish updating after the test completed, they haven't budged from 5 as of 19 days since the end of the test, even after crossing a month-end boundary.

The rumor is that it can take up to 2 weeks for the data to be complete. I think that's a fair margin, based on my series of tests here, though the only data I found to shift after the event date was units sold (and related data such as ACOS), and impressions, which drift around a bit. (I think it very likely that there are multiple independent databases at Amazon which update each other on various timeframes.)

When I ran my year and a half of Amazon keyword ad farms before, I tracked the match between AMS Units Sold as reported, and my best estimate of equivalent KDP Units Sold, and concluded there was about an 11% discrepancy — underreporting by AMS. This was based on 1200 units advertised and reported by AMS.

However, the difference between 5 and 20 is far larger than 11%.

AMS's calculation of ACOS is, of course, calculated on the units sold that it reports, so I ignore that and look instead at my own calculation of ACOS. I add the product sold price and the series read-thru price, reduce to 70% for the actual (eventual) payment by Amazon, and come up with my Earnings. (I ignore the impact of file size on the KDP payout.)

Comparing my Earnings to my AMS Spend yields my Return on Investment (ROI). As you can see, my ROI was negative — I lost $81.84 on this test, and there was no reason to expect that to improve.

Test 2 – Dynamic Up/Down, bid $0.65, 3 days

I killed the portfolio campaigns from the first test and set up an exact duplicate, with a reduced bid of $0.65.

TEST 2 – Dynamic Up/Down, bid $0.65, 3 days

Cost per Click (CPC) and Click-Thru Ratio (CTR)

My bid was $0.65, and the average CPC was $0.55. The CTR was a healthy 0.08%.

Units Sold and ACOS and ROI

16 days have passed since this test was stopped, but the units sold count for AMS hasn't changed for several days (AMS = 3, KDP = 6)

The ROI is still negative — I lost $140.58 running this test.

Test 3 – Dynamic Down, bid $0.65, 4 days

I killed the portfolio campaigns from the second test and set up an exact duplicate, the only change being that I changed the bidding method to Dynamic Down, which never moves the bid up opportunistically.

TEST 3 – Dynamic Down, bid $0.65, 4 days

Cost per Click (CPC) and Click-Thru Ratio (CTR)

My bid was $0.65, and the average CPC was $0.47. The CTR was still a healthy 0.08%.

Units Sold and ACOS and ROI

12 days have passed since this test was stopped, but the units sold count for AMS hasn't changed for a while (AMS = 8, KDP = 14).

I almost broke even, but the ROI is still negative — I lost $17.07 running this test.

Test 4 – Dynamic Down, bid $0.55, 11 days (ongoing)

I killed the portfolio campaigns from the third test and set up an exact duplicate, the bidding method still Dynamic Down, and the bid now $0.55.

TEST 4 – Dynamic Down, bid $0.55, 11 days

Cost per Click (CPC) and Click-Thru Ratio (CTR)

My bid was $0.55, and the average CPC was $0.41. The CTR is an adequate 0.05%. Note that the two bundle products never sell as well as the two first-in-series novels, so I pay more attention to the latter with regard to CTR.

Units Sold and ACOS and ROI

2 weeks has not yet passed since this test was started. At this time the units sold count for AMS is still changing, and will continue to do so as the ads run (I will run them for months). Right now, AMS = 9 and KDP = 20.

The ROI is great — I spent $122.47 on these ads and, by the KDP count and my standard adjustments for expected series read-thru, I will eventually receive $196.46 from Amazon, for an ROI of 60.4%.

Analysis

There's gold lurking in this data, if you look for it.

What is Amazon doing?

I believe you can perceive certain underlying principles in how Amazon presents my ads based on this simple series of 4 tests.

Let's look at some of these numbers again…

Some key statistics

Serving impressions

The higher your bid, the easier it is for Amazon to serve it (to win the bid auction). The difference between Test 2 and Test 3 is probably not significant, but look at the range across the set of tests.

Click Thru Rates (CTR)

When you bid less, you are served to lesser quality viewers who are less likely to click, and thus the CTR goes down pretty dramatically. 0.05% is the minimum goal you want, before which rumor claims that Amazon penalizes you by showing your ad less often. (Their reasoning: if customers don't click on your ad, it must not be interesting to them.)

Cost per Click (CPC)

Naturally, since I am bidding less for 3 of these tests, the CPC goes down. And switching from Dynamic Up Down to Dynamic Down is almost as good as dropping my bid by $0.10, all by itself.

Conversion (Sales/Clicks)

These tests are a little brief to really compare conversion stats, but I'll keep an eye on Test 4 as it goes on to see if the nice healthy conversion rate continues. The goal is to keep that higher than 5% — you want the clicks you get to convert into sales at a reasonable rate.

Tentatively, it looks as if the conversion for the more expensive clicks isn't as good, and I'm not sure why that should be. Perhaps it's just a misleading coincidence (the perils of small numbers/limited testing days). Still, while the CTR is lower for Test 4 (impacts Amazon exposure), the much lower CPC (impacts cost) and the much better conversion (impacts cost by reducing the clicks needed for sales) both contribute to the big difference in ROI. In other words, I don't sell more units (more limited exposure) but the units I sell are significantly more profitable.

Keep in mind that, as for CTR, the bundles should perform less well than the 1st-novel-in-series, since they're more expensive and thus a harder stretch for a new-author-to-the-buyer purchase. So focus particularly on the statistics for the 1st-novel-in-series ads. I sell less than 10% as many 1st-bundle-in-series books than 1st-novel-in-series books, but I earn more from them, and the series read-thru to the next bundle is reasonable.

AMS Orders (Units Sold) vs KDP Units Sold

This is a puzzlement, and the major source of uncertainty behind analyzing how your AMS Ads are doing. It's clearly the case that AMS has something of a lag in reporting orders, and one would be wise to wait at least 2 weeks before assuming the AMS count is final.

I'm pretty confident, since I have no other ads running and my organic sales are so low, that all the KDP ads I've counted in these tests are correct — at worst, I might be off by only a handful of true organic sales. But, as it stands, there's a current discrepancy of AMS 25 units vs KDP 60 units for these 4 tests, and that is a very large difference.

From my old data, I thought I could trust the AMS count to be at worst 11% under KDP actual and could treat it accordingly, but these tests seem to indicate that's not the case, or perhaps just not any longer. I'm going to keep a close eye on this while test 4 (now the current ads) continues to run. Right now it looks as if the AMS Units Sold is rather grossly undercounted, not just lagged. But see the Addendum below for other considerations.

Remember, if AMS Units Sold is that unreliable, it makes the AMS ACOS unreliable, too, and thus a dubious measure to base decisions on. You're better off trying to pull out the exact KDP sales from the KDP side to do a KDP-based ACOS calculation, and that can be very difficult when you have significant and varying organic sales and other ad systems running, like Facebook. That's why I was glad to run these tests in isolation in a simplified environment with my organic sales that are too small to mess up the analysis.

Going forward, with a longer run of Test 4 to draw upon, I'll have to draw some statistical conclusions if I can, similar to my old 11% corrective, once I add Facebook Ads to the mix and it becomes impossible to really assign KDP Units Sold to the appropriate source. It will also be difficult to test Facebook (unless I pause AMS Ads while doing a similar set of tests on Facebook — I think that's the approach I'll be taking.)

See the Addendum below for other thoughts on the match between AMS and KDP Units Sold from these tests.

ROI vs Volume

You can't draw too many conclusions based on the ROI for tests that run just three or so days (random variability is the probable cause of the difference between Test 1 & Test 2), but the trend is quite clear.

On the one hand, I'm delighted with the ROI on the final test, but on the other hand I'd like to be served to more viewers, to generate more sales and take advantage of Amazon's push (more impressions) for better performing ads. But it doesn't matter how many sales I get in the long run, if they're not profitable.

This is the old problem for Amazon ads about scalability. You can have profit, or you can have volume — hard to get both.

Professional agencies who serve AMS Ads for authors rely on long series (many more than my 4 books) so that the long series read-thru can pay for the cost of higher bids. They try to take advantage of Amazon's greater support for higher performing ads (based on the ad clicks) — but Amazon can't see the additional read-thru considerations and thus the rationale that supports the ad spend.

I can't compete with bids at that level, so I'll have to take my profits at a lower level of bid. It's pointless to chase the high spenders.

The problem with looking at things the way the agencies and some of the high spenders do (taking the maximum bid you can support) is that you want to earn real profit from the series read-thru if you can, not spend it all chasing higher bids.

As in all things, it's a matter of finding a balance you can live with.


Addendum

There is uncertainty in all of these metrics, and you should devise ways to test further and make deliberate decisions that you can review later.

Organic sales assumption

What if I'm wrong that my organic sales for the advertised products (as a group) are effectively 0 units per day? What if they were closer to 1 per day or 2 per day? How would that impact this test?

How sensitive are the results to my assumption?

Key Statistics incl. different assumptions for organic units

Well, for one thing, raising the number of assumed organic sales/day would have the virtue of having the remainder more closely approximate the AMS Orders (Units Sold). There would still be a question as to why Test 1 has such a great disparity.

The actual KDP units which are not-organic (which are attributable to AMS ads) can't be less than the reported AMS Orders. That is, if KDP reports 14 total units for Test 3 and AMS reports 8, the KDP units which are not-organic can't be less than 8, regardless of the formula I plug in for estimated daily organic sales.

If my organic sales are closer to 1/day than 0/day, then the ROI for Test 4 becomes -29.3% and I am losing money.

However… remember the problem with small numbers.

The sales over the 11 days of Test 4 are uneven, and when you're dealing with small numbers that can have a large impact on testing. When I looked at these numbers as recently as 2 days previously, the ROI was positive.

11 days of running Test 4

I believe, through long tracking data from before I began advertising, that my organic sales for the advertised books are on average significantly less than 1/day (when not involved in a new-release situation). But it's a good idea to test that hypothesis and see what the results might be if I'm wrong. If I were convinced that had changed and moved to 1/day or more, then I would want to test a bid of $0.45.

Decision: I will run a test to confirm my assumptions about organic sales (which are based on old data).

Update: I paused the $0.55 ads at the end of July and ran with no ads or promos at all for 3 weeks, to confirm my assumption that my organic sales were effectively zero/day for the advertised products.

Here's what that looked like.

All ads and promos paused — organic sales only for 21 days

The result? My current organic sales rate for these two books is 0.2 units per book per day.

Decision: That's close enough to zero that I'm not going to worry about it. I resumed the $0.55 DYNDW ads.

Decision: I will monitor my ROI for the resumed $0.55 ads. If it drifts down, I may lower it to $0.50 or $0.45, just in case the organic rate creeps up over time. The penalty will be fewer impressions, but the ROI is the important part, for me.

Running ads for 1st-bundle-in-series assumption

In my previous run of AMS Ads, I got decent results advertising for 1st-bundle-in-series as well as 1st-novel-in-series. But in these tests, the results have been rather disappointing. Out of 60 KDP Sales presumed to be attributable to AMS, only 7 came from bundles. It's not that 7 is such a poor number — it's that the daily AMS spend overwhelms it, and the CTR for Test 4 is poor.

What would the metrics look like if I removed both the sales and the expense for the bundles?

Key Statistics incl. different assumptions for organic units and comparison with or without ads for bundles

Well, it wouldn't make much difference if I'm wrong about the organic units (that overwhelms the distinction), but if I'm right about the organic units (significantly less than 1/day), then look how the ROI for Test 4 improves, as well as all the other metrics.

Decision: I will pause the ads for bundles in Test 4 for and see if the improvement in ROI and the other metrics is sustained.

Additional tests

See this detailed additional test re: overcounts and lags here. It examines in detail some of the weaknesses in the AMS units sold data.

Business-as-usual assumption

I have other assumptions related to full series read-thru percentages, which I mentioned above, but since I don't expect those to worsen (they're already a worst-case scenario), I don't need to calculate variants to make decisions.

But, like all of my old metrics, this is based on my not being able to improve organic sales through other means, such as newsletter prods, backmatter enticements, and other background changes to tune the passive sales engine better. If I can improve that, then the organic sales will increase, and my assumptions for calculating ROI from advertising will have to take that into account. Otherwise the metrics I look at to monitor apparent adequate returns on advertising could degenerate into ROI-level losses without my even noticing the change.

Always test your assumptions so you understand what their impact might be on your decisions.

Subscribe to My Newsletter

...and receive a free ebook: The Call, a short story that precedes the start of The Hounds of Annwn.

2 Comments

  1. Thanks for sharing – you did an amazing amount of work to get this data.

    So many people have recommended that you not even bother with advertising unless you have a series of books, especially a long one – you, Kris Rusch, and people who say they’re profitable in the FB groups I’m in – that it’s starting to feel like dogma.

    Without the nimbleness of being a multi-novel author, I can see that Amazon ads are probably not the way to go.

    But I still like to see how one end of the price spectrum works for indie authors, even if I can’t imitate the speed or results.

    July 6, 2019
    |Reply
    • You know, Alicia, that my 4th test above is profitable WITHOUT the series read-thru for the ANNWN-N1 1st-in-series book, and breakeven for CHAIN-N1, at 70%.

      That means if these were standalone, all else being equal, they could probably make a small profit. And therefore, they could be tested at a $0.45 bid, to see if that profit could be solidified.

      It’s not impossible to get a profit from advertising AMS for a standalone book. Just harder — less cushion.

      July 6, 2019
      |Reply

Leave a Reply

Your email address will not be published. Required fields are marked *