Your boss has tasked you with the ultimate goal of increasing the response rate on your next marketing campaign.
One of the ways you have chosen to do this is to engage a modelling company. Their proposed solution is to build a model that will help you drive more responses. Your many years of experience has led you down this path before - it hasn't worked and it burned you.
Recently, you hear that there have been breakthroughs in machine learning and artificial intelligence and you decide to give it another shot. But you're more cautious this time and you ask yourself:
How can I ensure the model a company builds me will be successful?
What are steps I can you take to ensure a model will perform in your next campaign?
The typical scenario would have gone like this:
The modelling company you engage (Company-XYZ) will ask you for your historical data on previous campaigns.
You provide them with this data, either through a FTP or giving them access to your CRM system.
And they tell you that they think they can get you 20% more responses.
However you find, after using their model, you're responses are not up by 20%. They aren’t even close. How you can prevent this?
What method can you employ to ensure that the model Company-XYZ has created will be successful in your next campaign?
The answer is simple. A blinded back test.
A blinded back test is a method employed by data scientists to ensure that their newly built model is stable, not overfit and will also mimic performance in real life. This is the best indicator of how successful the model will be.
This kind of testing is used in data science competitions. It is the best method and industry standard for checking the performance of a model. Large amounts of prize money is given to the model with the most accurate blinded back-test.
The benefits of a blinded back test are:
Ensures Company-XYZ is not scamming you.
Gives you an estimate of how well Company-XYZ's model will perform
You should hold every modelling company you engage to this standard. You will find quickly that some companies will no longer be able to do the job, or you will be able to weed them out if they fail this test.
How to apply the blinded back test:
When Company-XYZ requests data from you, only give them half of your data. Specifically, give them a randomized half.
Company-XYZ will then use the data you provided them with to train their model. After model training is complete, they will come back to you and say something like: "Our super-secret modelling process TM is telling us we can provide you with 20% more responses then your current process".
This is the point where you might have failed in the past, but the application of the blinded back-test will ensure your success. Tell Company-XYZ that you're going to give them a blinded dataset that you would like them to score. After they scored the blinded dataset, they will give you back the same dataset, but with their predictions on responses.
Here is where you have to do a little work to confirm their success. You append a field with the actual responses and compare it to their prediction. Count up all of the instances where, they predicted a sale, and there was actually a sale. Divide that by their total number of predictions. (If you are statistically savvy, then you will know that this is a confusion matrix).
Interpreting the results compared to the 20% originally stated:
Here are the possible scenarios:
Back-test results < the 20% expected - If this is the case, the model that was created is overfit. This basically means, the model has no predictive power and will fail in the real world. This model is garbage, have them throw it out and try again.
Back-test results > the 20% expected - If this is the case, the model is very unstable. This means, that the results they were expecting were not the results that happened. Performance could have easily gone in the other direction. This model is garbage, throw it out and have them try again.
Back-test results = with 20% expected - WINNER!! The model is stable, and did exactly what the modelers have predicted. Move forward with this model.
The blinded back-test can be used to ensure the honesty of a company's numbers, help to avoid the model from overfitting and can give you an estimate of the response rate performance in your next campaign.
A good strategy might be to engage multiple companies and give them the same data, and hold them to the same standard. The company with the highest stable back-test, should be who you go with.
WE’D LOVE TO TALK! Contact us at firstname.lastname@example.org.
Dive deeper into the intricacies of back testing in part II of this article.