Not sure whether the AI Sales Agent will increase sales in your store? Check it with an A/B test. For 7 days, half of your visitors chat with the AI chat and the other half get what you have today without AI. Then you compare the conversion rate and revenue of both groups on the same traffic. The test is free.
What does the test tell you?
Do customers who can chat with AI order more often?
Does your store earn more per visitor with AI?
How many chat conversations does each group of customers have?
Who can run the test?
The A/B test is available if:
your store has no active AI Sales Agent package and no trial in progress,
your store has no custom pricing for the AI Sales Agent,
you have billing permissions in your edrone account.
A previous trial or package does not block the test.
Remember: each store can run the free test only once.
How to start the A/B test
Go to AI Sales Agent → Chat → Settings.
In the AI availability section, turn on Test AI on half of your customers.
Publish the chat. The test starts when you publish and lasts 7 days.
📌 Tip: when you turn on Regular chat (that is, switch AI off), you will see the Better with AI or without? window. Instead of turning AI off, you can click Start A/B test there.
⚠️ Important: if after publishing you see a message that the chat is published but the A/B test did not start, turn on the test again and publish the chat once more.
What do customers see during the test?
Half of your visitors see the AI chat, which answers 24/7.
The other half get what your store has without AI:
with Regular chat on: the regular chat, where you reply in Inbox,
with Regular chat off: no chat at all.
The split is per visitor, so the same person sees the same chat for the whole test.
How to check the results
While the test is running, click See results. You will see which day of the test it is (for example Day 3 of 7) and preliminary results. After the test ends, the button changes to See summary.
For the With AI and Without AI groups you will see:
Visitors,
Visitors who ordered,
Conversion rate,
Orders,
Revenue,
Revenue per visitor,
Chat conversations.
At the top of the results you will see a verdict:
Verdict | What it means |
Visitors with AI order more often | The conversion rate with AI is higher and the difference is statistically significant. |
Visitors without AI order more often | The conversion rate without AI is higher and the difference is statistically significant. |
No clear winner | The difference in conversion rate between the groups may be due to chance. |
Not enough data yet | The groups can be compared once each has at least 5 visitors who ordered. |
A winner is shown only when the difference is statistically significant.
Orders that cannot be linked to a store visitor (for example phone orders) are shown separately as Orders without a group.
When does the test end?
automatically after 7 days,
earlier, when you click End test,
right away, when you buy an AI Sales Agent package.
After the test, all customers get the chat set in AI availability again. To change it, turn Regular chat on or off and publish the chat.
Packages and pricing are described in AI Sales Agent — How It Works and How to Enable It.
Need more help?
If you need more help, message us in the chat 💬 or by email at hello@edrone.me