Prediction helps to prompt the customer to buy at the right time, or, conversely, not to touch him if he buys and so. As part of the AB test, predictive-based site personalization mechanics helped to increase the conversion rate of the online store by 16.5% and ARPU by 35.7% relative to the control group.
Azamat Tibilov, Chief Marketing Officer for Mario Berluchi, talks about predictive mechanics, measurement of results, the history of the Data Science business and shares tips for online retailers who also want to grow revenue through useful and data-driven marketing.
Mario Berluchi is a Russian manufacturer of shoes, bags and accessories with five offline stores in Moscow and an online store.
Scale. 200 thousand site visitors per month.
IT. Site on Bitrix, back office on 1C, client data platform Mindbox.
Task. Increase revenue by working with accumulated data.
Result. An increase in website conversion by 16.5% within the AB test, an increase in ARPU by 35.7%, a decrease in the share of abandoned carts by 17.2%.
How Prediction-Based Website Personalization Mechanics Work
When a client visits the site, we record his actions and run it through the prediction algorithm: “buy in the current session or not buy” and “will return within 7 days or not return”. The prediction is recalculated every 10 seconds for each client.
Mechanics triggering conditions:
- if there are items in the cart,
- if no discount coupon has been applied,
- if the predicted purchase probability is less than 30%,
- if the client is predicted not to return within 7 days.
If the conditions are met, the client sees a pop-up in the basket and decides whether to buy the product in the current session or not: The
pop-up pops up in the basket if, according to the algorithm's prediction, the client does not place an order in the current session and does not return later
Predictive Mechanics Results
AB tests with 95% confidence
Part of the clients within the test were in the control group and did not see the pop-up - for her, the mechanics were turned off, and the other part saw. We compared conversion, ARPU, and cart abandonment in these groups - we got statistically significant results with 95% confidence:
↑ 16.5%
Increase in website conversion relative to the control group using the t-test method
↑ 35.7%
ARPU growth using the bootstrap method
↓ 17.2%
Decrease in the share of abandoned carts using the z-test method
Comparison of conversions and ARPU: in May 2019 and May 2020 - after the introduction of predictive mechanics
Conversion before and after implementation of predictive mechanics
ARPU before and after implementation of predictive mechanics
Why did you launch the Data Science direction
Initially, we wanted to build end-to-end analytics to assess the quality of advertising channels in the context of real purchases, because 50% of orders “fall off” at the confirmation stage.
For end-to-end analytics, it was necessary to collect user behavior data into the Google BigQuery database. In addition to the standard user actions - adding an item to the cart, visiting a product card, making a purchase - we collected many more actions with the site's content - hits. More than 20 thousand lines of hits were accumulated daily, and this data was stored in our database, for which we, of course, paid.
With our traffic - more than 200 thousand users a month - there was enough data, and we carried out standard analytics, for example, user actions with content after any changes, purchases after promotions. Then we conducted a brainstorming session with the owner of the company and decided to try, in addition to simple analytics and AB tests, to build something more interesting: try to predict customer behavior on the site using machine learning algorithms based on our historical data. We treat such ideas as an internal product in which we are ready to invest money and time in order to get results later - the growth of business metrics.
As a result, the Data Science department was assembled and in six months they implemented a mechanic with predicting user actions, which increased revenue. Thus, we discovered a new line of business, which brings us over 30% of our revenue and pays off well.
What specialists were needed for Data Science
Each stage of the launch of a predictive mechanic involves the work of specialists of different functionality, but from related areas. Our staff:
Analyst. Analyzes data, finds anomalies and performs AB tests.
Two Data Scientists. They write algorithms that return predictive answers in the form of the probability of a particular user action on the site.
Marketer. Develops and launches mechanics using algorithms.
Developer. Implements mechanics and algorithms on the site.
How Predictive Mechanics Works
1. We mark up the initial Google Analytics data using Google Tag Manager and use OWOX BI streaming to collect data in the Google BigQuery database. These steps take little time - from the first minute you can see how the data fits into the database.
2. The analyst looks at how the data matches user behavior. If necessary, it builds distribution graphs and looks at how high-quality they are, whether there are tails. If errors are found, we change the streaming setting or clean up the data, because it is impossible to work with dirty data in machine learning.
3. Data Scientists create features (feature engineering) from visits and content, for example, the number of products viewed, the number of items added to favorites, the number of items added per session to the cart.
Distribution of the weights of the algorithm features - on their basis we predict the client's behavior
4. Train the model on historical data. Let's say we want to predict if the user will have the next session or will return to us within 7 days. To do this, we take historical data, signs - and implement the algorithm. For prediction, we use the classification - the binomial answer in the form of 1 or 0.
5. We validate the model on historical data: forecast accuracy, business metrics.
First of all, we look at the proportion of accuracy (proportion of correct answers) and ROC-AUC (area under the error curve):
Accuracy 0.88 means that 88% of the time we accurately predict that the user will come back or not. Precision - How much of the prediction that the user will return was correct. Recall (completeness) - about what proportion of real returns we predicted.
We use AUC ROC (area under the error curve) to assess the performance of the algorithm on a sample of data
In addition to the answers of the algorithm 1 and 0, there is also a probability of action in percent. And here we set a threshold: if the probability of a user's return is more than 30% and such users most often do return, then the answer is 1.
6. We predict the user's actions.
7. The marketer develops the mechanics for applying the forecast.
8. We launch the AB-test - only on new users who have got acquainted with our site right now. The test lasts about three weeks, and during this time we watch how the cumulative p-value changes. At some point, the difference between the groups becomes significant, we understand that soon the test can be completed and the mechanics can be rolled out into production.
9. The analyst measures the results of mechanics.
Based on what customer data the prediction works
Visit-based. Based on actions on the site: viewing product cards, adding products to the cart, shopping.
Content-based. Based on actions with site content. First, we collect data on user actions: opening a size table, adding a product to favorites, reading reviews. Then we look at how these actions affect proxy metrics (intermediate conversions before ordering) - this is necessary because there is more data on these metrics than orders. Next, we look at the correlation of proxy metrics with purchase conversion and return rates.
The visit based and content based approaches overlap. But in visit based we evaluate user behavior, and in content based - the content itself.
CRM-based. Enrichment of data from the CRM online store, accounting of purchase history.
Online shopping tips
1. Analyze the data, even if you are a small online store. There are hidden growth points in the data that allow you to take your business to the next level. In today's world with huge competition in digital, it is impossible to solve the problem of business growth with a banal injection of money.
2. The growth of conversion, a key metric of an online store, is the most powerful factor in the development of your business.
3. Don't be afraid to build infrastructure and introduce new technologies into your business. The introduction of machine learning allows you to take a step forward and break away from competitors.
4. Learn to calculate ROI from investments in new technologies. Most companies are afraid to allocate a budget for new tools because they do not fully understand how they will benefit.
Further plans for marketing development
Now we have dynamic pricing in our work - we will evaluate when to give a discount on which product or, conversely, not to give it. It all sounds simple: the product is often bought - we do not give a discount, seldom bought - we give it. But we go a little deeper and wider - we look where this product was in the catalog, what marketing mechanics it participated in, how many times this product was viewed, how many times it was added to the cart.
And the next step is dynamic pricing for each user.
How to replicate predictive mechanics in your online store
We are developing cooperation with Mindbox and offering platform clients to implement our predictive mechanics. If you want to repeat it in your online store, write to your colleagues.
***
Authors:
Azamat Tibilov, Mario Berluchi Marketing Director
Maria Baikauskas, Mindbox Manager
Sema Syomochkin , Mindbox Editor