It’s (always) a good time to double down on customer retention. The trouble is, while the ROI is high, it’s sometimes hard to figure out whether you’re doing the best you could. The question that comes up often is – what kinds of campaign mechanics are doing well? Is it multi-armed bandits? What should we do more of? What should we do less of? And this, I’ve often found, is not an easy question to answer. Here’s why:
Fearlessly Try Stuff
If you want to figure out what works, you need to have tried a whole lot of things and monitored them for a while. Does reaching out to small sets of customers with targeted messages work better than mass outreach with offers? Can’t say till you’ve tried (no it’s never as straightforward as “Yes, targeted works better!”). Does personalization help? Does sending it on Friday work better than on Wednesday? Does a 50% Off work better than BOGO? Many organizations just haven’t tried enough to be able to get good insights so the point is – keep trying stuff. If you haven’t been doing it already, start now. If all you’re doing is mass campaigns with an offer and no personalization you aren’t going to be able to optimize campaigns.
Benchmarks From “Out There”
Ever so often I get asked to help with “what’s the benchmark”, how well are we doing compared to our peers. The only real answer, I’m afraid, is: You are your peer. Do better than you did, and keep optimizing. Every firm operates in a different context. Even if they look similar, maybe their loyalty program enrollment criteria are different, maybe the number of outreaches per month they do is different, so many maybes… So don’t bother with what’s being curated and put out there in conferences and case studies, just keep doing better than your past self.
Why A/B Testing Isn’t The (Complete) Answer
This line of discussion often gets around to A/B testing. Nothing against it, but to me, you can’t have a winner-takes-it strategy when it comes to campaign mechanics. A/B testing is winner takes it all – you see which option worked better and then divert all weight there. Variant A has a % Off offer, Variant B has an Upgrade offer – if A does better than B, it doesn’t mean all comms should now go out with the % Off offer. It does however mean one should amplify the winner strategy and suppress the loser – till perhaps things change. And that’s the key point – things change, context changes, people change – so what they respond to will change, and if you shut the door on an option, you miss picking up a signal that could be vital.
Enter Multi-Armed Bandits
Multi-armed bandits are fascinating. The name originates from a Casino-beating strategy wherein you put some tokens in multiple slot machines and crank their arms (a Slot Machine = a one-armed bandit, stealer of your tokens. Many slot machines = Multi-armed bandits… you get the drift). You feed more tokens in the winning machine(s) and less in the losing ones, without diverting all your money to the winning one. This is where ML takes traditional A/B testing and amps it up – exploit winning campaign mechanics by diverting more comms there, and suppress losing campaign mechanics but don’t shut the door on them.
You might like to know more about our prompt generator & how it differs from the rest.
Sometimes, all you want is to know what works
This is where we started, right? Multi-armed bandits can solve a lot, but they’re not great at actually spelling out what works and what doesn’t to you. Good old descriptive analytics does that just fine. In SOLUS we just released a new module to help analyze what campaign attributes work. Here’s how we’ve done it:
Campaign Intelligence in SOLUS AI
- We’ve broken down all campaigns and triggers into attributes like Type/ Segment/ Timing/ Size/ Channel/ use of Personalization/ Offer type/ use of Recommendations etc
- We measure them on Metrics of Yield (Incremental Rev per Outreach), Conversion %, Incremental Revenue $
- We give you the ability to look at variables in isolation or in combination
A couple of no-context screen grabs below: You have to see these on your data, hence is no attempt to describe these charts:
One more where you see campaign attributes in combination:
This personalization engine from Solus AI is incredibly useful. Give it a look (or try), and if you’d like to know more about how we work with Multi-armed bandits to auto-optimize campaigns, contact us today!