Headline Testing Orientation and User Guide Table of Contents 01 INTRODUCTION 02 GETTING STARTED 03 CREATING A HEADLINE EXPERIMENT 04 IN-PROGRESS TESTS 05 TEST CONCLUSION AND ARCHIVE 06 TIPS AND BEST PRACTICES 07 FAQs 08 METRICS GLOSSARY Introduction What is “MAB” Headline Testing? “MAB” headline testing is a multi-variate approach to running live tests to determine if one headline is more successful at attracting visitors’ interest (clicks) and holding their attention on the page (engaged time). This multi-armed-bandit approach, also known as Thompson Sampling, is different from other multivariate testing methodologies. It exploits the success of winning headlines by ‘playing’ them more frequently as they prove to outperform other headlines during the test. To learn more about Thompson Sampling, check out the methodology report from our Data Science team. What does a test look like? Simply select an article on your homepage (or another high traffic landing page or section front) and enter the various headlines that you want to test. Chartbeat will then dynamically serve those different headlines to your audience equally, and as certain headlines begin to receive higher click-through-rates and engagement, they will be served more frequently to your audience while unsuccessful headlines are served less often. The test ends when the winning headline has been determined with 95% confidence and Chartbeat begins to automatically serve that headline 100% of the time. To make sure that tests come to a conclusion as efficiently as possible, we have an alternate way to determine a winner, called ‘soft converge’. If 20 minutes have passed and we're 95% confident that no headline is better by a margin of 25%, the leading headline will win. How do I get started with Chartbeat Headline Testing? Your Customer Success Manager is the best person to contact if you’re interested in performing headline tests on your site. They’ll be your best point of contact to discuss pricing, permissions, and other details. If you’re unsure who your Customer Success Manager is, just reach out to the Chartcorps at support@chartbeat.com and they’ll be happy to point you in the right direction. 01 Getting Started Install the Headline Testing Bookmarklet As the interface for the Headline Testing tool is built into the Heads Up Display, you’ll need to install a specific version of the browser overlay. Note that the Headline Testing product is optimized for Google Chrome. Click here to get the installer. Select Headline Testing Mode Once the “MAB and HUD” bookmarklet has been installed, simply refresh the page, open the Heads Up Display bar, and select the “MAB Beta” option on the right hand side. 02 Creating a Headline Experiment Select a Headline to Test When entering the Headline Testing mode, the colored pins on your homepage will disappear. As you hover your mouse over headline links, they will become highlighted with an edit pin. Click on the headline or pin to initiate a test. Note: If the headline you want to test is nested together with an image inside a single anchor tag, you must click directly on the headline. This will only work if the headline is in a <div> tag nested in the larger anchor tag. If you click on the image instead of the actual headline, you will see this error message. Additionally experiments cannot be created on headlines that link off site. Entering Alternative Headlines Variant A will be auto-filled with the headline that appears by default in the page and is not editable. As additional headlines are entered they appear inline on the page so you can preview their appearance before testing. While there’s no limit to the number of headline variants that can be tested, remember that an experiment’s progress will be slower the more headlines are added. Users can delete or edit any variant before starting the test. Select ‘start’ to begin testing the desired headline variants. 03 In-Progress Tests Following a Test’s Progress To monitor the progress of a test, select the pin next to the headline currently being tested. Active deadlines experiments will be indicated with a pie chart representing the number of headlines being tested and the play percentage of each headline. Headlines with a higher play percentage are ones that are performing better. Note: Editing tests, adding variants, removing variants, and editing the copy of headline variants will automatically stop and restart the test. A trial is the number of visitors who have loaded the page with a headline currently being tested. While we can’t guarantee that a visitor saw one of the headline variants, we do know that a variant was loaded into their browser. Play percentage represents how often that particular headline variant is being shown to visitors. A higher play percentage indicates that the a headline is performing “better”. The CTR chart displays the percentage of visitors who clicked on each headline out of the total instances that it was served. The Engagement chart shows the percentage of visitors who clicked on a headline variant and stayed engaged on the page for at least 15 seconds. The center of the boxes (thick line) represents the estimated CTR & engagement of a particular headline. There is a 50% probability that the headline’s performance rate is somewhere between the right and left edge of the box. There’s a 100% probability that the true performance of a headline is somewhere between the right and left edge of the whiskers (the thin lines). 04 Test Conclusion and Archive A Completed Test A green check mark next to a headline indicates that the experiment is complete and a winner has been determined. Select the pin to see the the winning headline. Experiment Archive To see a full report of all currently running, completed and stopped tests – select the menu icon in the bottom right of the screen. The Test Archive displays a list of all current and completed headline experiments, the engagement and click-through-rate performance graphs, and the number of trials for each headline. Use the Test Archive as a resource to analyze patterns in your audience’s behavior and preferences. 05 Tips and Best Practices Test from a Hypothesis. Insights and best practices will develop faster if you’re systematic about the experiments. Develop a hypothesis, such as using a ‘casual tone’, and compare and analyze results. Test Contrasting Headlines. The more differentiation in your headline styles, the more likely you are to see distinct audience preferences. Headlines that simply reorder words are less likely to have significantly different click-through. Write Daily Wrap Reports. Create a daily summary of the insights you gained from previous experiments and share them with the team. This way all editors and content producers can benefit and learn from all headline tests. Know your Audiences. Remember that homepage audiences are primarily your site’s most loyal readers and will generally behave differently than your social media audience. Don't assume that a headline that does well on your homepage will also do well at attracting visitors from Twitter or Facebook. Test in Higher Traffic Positions. Once comfortable with the tools, don’t be shy about trying headlines tests in your site’s most prominent positions. The higher traffic will help a winning headline to emerge more quickly. Don’t Stop Tests Prematurely. It might be tempting to stop a test when the play percentage for a headline goes up quickly to 70% or 80%. However, there will likely be instances where headlines jump to an early lead but end up being inferior. Don’t get Discouraged by Ties. It’s possible that there are two great headlines that are attracting similar levels of engagement. If you’re frequently seeing similar results, it might be that the headlines being tested are too similar. 06 FAQs How long do headline tests last? As a headline test is actively measuring the click-through success of each headline as compared to the others, there is no set length of time for a headline test. Generally, the length of an experiment depends on three factors: the volume of traffic to your homepage, the number of headlines in the experiment, and the extent to which one headline is actually better than the other(s). On a site where there are hundreds of trials per minute tests will likely take 10 to 30 minutes, perhaps longer if the two headlines aren’t that different. Since clicks and engaged clicks are counted separately, how much does each contribute to the success value? Clicks to engaged clicks are weighted at a ratio of 1:2. One way of thinking about it is that a click on an article is worth ‘one point’, while fifteen seconds of subsequent engagement is worth ‘two points’. What happens to a test when the headline moves down or off the page? When a headline moves from one location to another on your page the test will stop. To see how the test did you can go to the test archive page to see what the results were, and you can also restart the test at the new position. Is there a max number of tests I can run simultaneously? Currently only 5 headline experiments may be run simultaneously, but as we continue to rollout this product and evaluate how our backend systems can scale to accommodate the data, this number may increase. What browsers will Headline Testing work in? Tests can only be created in Google Chrome Will visitors who come to my homepage multiple times see different headlines for the same story? No — we will always show visitors the same headline they saw initially. 07 Metrics Glossary Trial A trial is every time that a test headline is loaded in a page and served to a visitor. For example a headline that ran a total of 2,000 trials was served to 2,000 unique cookies. On the other hand, if an experiment had 5,000 trials it means that headlines were served a total of 5,000 times. Click Through Rate (CTR) Percentage of visitors who click on a given trial headline (can be lower than 1%). Play Percentage The percentage of people being served a given headline, as well as the confidence that headline is “better”. For example at 65%, a headline is served to 65% of people and has a 65% certainty that it is "better". Note that at 95% play percentage a headline "wins" and is played 100% of the time. Quality Clicks When a visitor clicks onto a story and subsequently spends at least 15 seconds of engaged time, it is considered a quality click. Note whenever a visitor reaches the threshold of 15 engaged seconds it will be retroactively be counted as a quality click. 08
© Copyright 2024