A/B Testing Your Game Covers

Only available for published games

Artwork experiments are only available for published games (Full Launch). If you're still in Basic Launch, you can run them as soon as your game goes to Full Launch!

Your cover is the first thing players see on CrazyGames, and even a small change can make a big difference to how many of them click on your game. With artwork experiments, you no longer have to swap your covers and hope for the best - half of your players see your current covers, the other half see the new ones, and you pick the winner.

You can test your covers, your preview videos, or both at once. This page focuses on covers, but the same advice applies to videos.

What is an A/B test?

An A/B test shows two versions of something to two groups of players at the same time, then compares how each group behaves.

The baseline is your current, published artwork.
The variant is the new artwork you want to try.

Both run side by side, over the same days, on the same kind of traffic. So if the variant gets more clicks, it's because of the artwork - not because it was a weekend, or a holiday, or your game got featured that week. That's what makes it much more reliable than swapping your covers and comparing this week to last week.

When to run an experiment

Test before you commit: Try a new art style, character or title treatment without replacing artwork that already works.
Settle a debate: Can't agree on two covers as a team? Test them and let the players decide!
Refresh an older game: New covers can bring players back to an older game - an experiment tells you whether they actually do better.
Your game has changed: Added a new world, a new character or a new mode? Test a cover that shows it off against the one you have now.

Seasonal covers

Most seasonal art doesn't need an experiment. If you just want Christmas covers for December, submit them as a regular artwork update in the Art tab, and switch back when the season is over. Only run an experiment if you want to find out whether your seasonal art actually beats your current art. Keep the timing in mind too - an experiment can run for up to 14 days, and a winning variant still needs to be reviewed, so start early or the season could be over before it goes live.

Requirements

Before you can start an experiment, your game needs:

Full Launch, and published: The Experiments tab only shows up for published Full Launch games.
Complete published artwork: All 3 covers (landscape, portrait and square) and both preview videos (landscape and portrait) need to be published. An experiment compares the variant against your published artwork, so there has to be a complete baseline to compare against. If anything's missing, you'll see a "Complete your artwork to run experiments" message in the Experiments tab - upload the missing covers or videos in the Art tab and wait for them to be approved.
No artwork waiting for review: If you've submitted an artwork update that hasn't been reviewed yet, wait until it's published or rejected before you start an experiment.
A complete variant: For a cover experiment, the variant needs all 3 sizes, so players see a consistent look wherever your game appears. For a video experiment, it needs both the landscape and portrait videos. For both, it needs all 5.

Limits

These limits apply per game. Running an experiment on one game doesn't use up anything for your other games.

One experiment at a time, per game: You can't create or start another experiment for that game until the current one is over (including the review of a winning variant).
Up to 2 experiments per game, per calendar month: An experiment counts towards the month its data collection starts in, so one you start late on 31 October counts towards November. If you cancel an experiment before data collection starts, it doesn't count.
Up to 14 days: Each experiment collects data for 14 days at most, then it stops automatically.

Your artwork is locked while an experiment runs

You can't submit a regular artwork update for that game until the experiment is over, including the review of a winning variant. A new upload would change the baseline you're measuring against, and the results would no longer mean anything. If you need to change your artwork urgently, stop the experiment and pick a winner first.

Setting up an experiment

Open your game in the Developer Portal and go to the Experiments tab.
Click New experiment.
Details: Give it an Experiment name (up to 30 characters), pick a Type (Covers, Videos or Both), and add a Description of what you're testing (optional, up to 500 characters).
Artwork: Upload your variant next to your baseline. Anything the experiment doesn't test is marked "Not tested, same as baseline", and players keep seeing your published version of it. Tick Show label zone to see the top-left area of each cover where labels can appear, and keep anything important out of it.
Review: Check everything over, then click Start experiment. Not ready yet? Click Save as draft and start it later.

Data collection starts at the next midnight UTC after you start the experiment, so every day in your results is a full day. Until then, your players haven't seen anything, and you can still cancel it: Move back to draft to start it later, or Archive experiment to get rid of it for good. Either way, it doesn't count towards your monthly limit.

We'll email you when your experiment starts, after it's been running for 7 days, and when it stops.

Your variant still needs to meet our requirements

The variant must follow the same cover requirements as any published artwork. If it wins, our team reviews it before it goes live (this takes up to 5 business days), and a variant that doesn't meet the requirements is rejected. The time to check is before you start, not after you've spent 2 weeks testing it!

How players are split

50/50 split: Half of your players see the baseline and half see the variant.
Each device keeps its version: Once a device is shown a version, it keeps seeing that one, so nobody sees your covers switching back and forth.
It's per device, not per account: The version is tied to the device (the browser, to be precise), not the player's account. So a logged-in player could see the baseline on their computer and the variant on their phone. Logging out or clearing their browser data can also move a player to the other version.
Only the tested artwork changes: In a cover experiment, everyone still sees your published videos, and the other way around.

Designing a good variant

The golden rule: change one big thing. You want a result that teaches you something about your players.

Test one big idea at a time: A different main character, a new colour scheme or a different composition, for example. If you change everything at once, you won't know which change made the difference.
Make the difference noticeable: Tiny tweaks are really hard to measure - a slightly brighter sky or a slightly bigger title won't move the numbers enough to tell. Players scroll past dozens of covers in a few seconds, so only bigger changes register. Bolder changes give you a much clearer answer.
Check it at small sizes: Most players see your cover as a small thumbnail, so make sure it still reads well when it's small.
Keep your covers consistent: All 3 sizes should look like the same game, just like your published covers.
Make sure it represents your game: A cover that promises something your game doesn't deliver might win on clicks, then lose players the moment the game loads.
Write down what you're testing: Use the experiment's description to note what you changed and why. Future you will thank you when you look back at the results!

Some ideas worth testing:

Your main character up close vs. a wider scene with more of the gameplay
A dark, moody palette vs. a bright, colourful one
An action moment vs. a calm, inviting shot
A different hero character, vehicle or enemy as the focus

For more on what makes a great cover, check out our game covers guide.

How long to run it

There's no minimum, but more data means a more reliable answer.

We'd suggest at least 7 full days: Players behave differently at the weekend, so a full week gives both versions a fair go on every day.
Low traffic? Let it run the full 14 days: If your game doesn't get many visits, a few extra clicks either way can swing the results.
Don't stop early because one version is ahead: Results bounce around a lot in the first few days. Wait until the lines settle before you call it.

Reading your results

Open your experiment from the Experiments tab to see its results. They show up once the experiment has collected its first full day of data, and they're updated daily.

The Results table compares the Baseline with the variant (labelled Experiment in the table) for the whole experiment so far, along with the Delta between them. You also get a daily chart for each metric:

CTR: The share of players who clicked on your game after seeing its cover somewhere on CrazyGames. This is the one your cover affects most directly.
Conversion to Gameplay 1m: The share of visits to your game page that turn into a play of at least 1 minute. This tells you whether the players you attracted actually stuck around.
Plays per User: The average number of plays per player, on the days they played your game.
Avg Playtime: The average time played per play, not counting time away from the game.
D1 Retention: The share of players who came back to play again the day after they first played. This one lags behind the others by about 2 days, as we need to wait and see who came back.
Average DAU Reached: The average number of players per day who were shown your game with either the baseline or the variant. It tells you how much traffic your results are based on.

You can filter the results by player type (New players were acquired on or after the day the experiment started, Returning players before that) and by device (Desktop or Mobile).

Look beyond clicks

A higher CTR isn't always better! A flashy cover can attract players who leave straight away. If the variant wins on CTR but loses on conversion, playtime or retention, it's bringing in the wrong players. The best variant gets more clicks and keeps those players playing.

Stopping an experiment and picking a winner

Click Stop experiment whenever you're happy with the data. Otherwise, it stops automatically after 14 days.

Data collection ends at the next midnight UTC after you stop it. From then on, the charts no longer change, and you can pick a winner. Until you do, your players keep seeing the version they were shown (and new players are still split 50/50), but that traffic no longer counts in the results. So try not to leave it too long!

The baseline wins: Your published artwork stays live for everyone. The variant isn't published and doesn't need a review.
The variant wins: It goes to our team for review, and replaces your current artwork once it's approved. While it's being reviewed, everyone sees your published artwork. If it's rejected, your published artwork stays live and you get feedback.

Don't be afraid to keep the baseline

If your current covers win, that's still a useful result! You've learned what your players respond to, and you can try a different idea next time.

Gotchas

Midnight UTC, not your midnight: Data collection starts and ends at midnight UTC. Depending on where you are, that might be the middle of your afternoon or the early hours of the morning.
Starting late in the month: An experiment started on the last day of a month counts towards the next month's limit, as that's when its data collection starts.
Stopped experiments count: Once data collection has started, the experiment counts towards your monthly limit, even if you stop it after a day. Only experiments cancelled before midnight UTC don't count.
Your game can't change artwork mid-test: Plan experiments around your updates. If you've got a big update with new covers coming, finish the experiment first.
One result is one result: A variant that wins once isn't a rule for every game. Use what you learn as your next idea to test.