Learning to love your losing tests: Embracing a culture of continuous experimentation for airlines

Author

Tim Axon

Published

26 May 2023

Last year there were 9.3 million flights worldwide, an average of 25,500 per day. This year there is expected to be a 19 per cent rise, bringing the industry back to numbers close to 2019, the last normal period prior to the impact of Covid.

82% of flights booked online

With customers returning to the skies, for airlines there is everything to play for. And given 82 per cent of all travel is now booked online via a personal device, the digital battlefield is where it’s at.

Arming yourself for the digital battlefield

It stands to reason, therefore, that experimentation and web optimisation will be crucial tools within marketing armoury in this growth period because to drive bookings airlines need to continuously test and refine their online presence to deliver the best possible experience to their customers.

JetBlue Airways recently conducted a series of A/B tests on its website to determine the impact of different page layouts on its booking rates. The airline found that a simpler, more streamlined page layout increased its booking rates by 4.3 per cent. Similarly, Delta Air Lines optimised its mobile app by testing various features such as trip planning, seat selection, and baggage tracking. The results showed that customers who used the optimised app had a higher satisfaction rate and were more likely to book again with the airline. And Singapore Airways is optimising its website for UX by providing a seamless user experience with fast page load times, easy navigation, and personalised content. The airline is also bolstering conversions by adding social proof, urgency, and scarcity to its website to encourage customers to book immediately.

Experimentation also provides valuable insights into customer behaviour such as what customers are searching for, what they click on, and what they ignore. This is rich data which can be analysed and used to inform future tests leading to continuous improvement of the customer experience.

Winning with culture

Experimentation is important because it delivers results. But fostering a culture of continuous website experimentation is more important. The most successful optimisation strategies are those that promote ongoing improvement and innovation. By embedding experimentation into the business employees are not fearful to test new ideas and learn from their mistakes.

This is why we believe that losing and flat tests are just as valuable (if not more so) that winning tests. Winning, flat, and losing tests refer to the performance of the variations being tested compared to each other.

A winning test is when one variation significantly outperforms the other variation in terms of the desired outcome. For example, if the desired outcome is clicks, a winning test would be when one variation results in significantly more clicks than the other. A winning test can provide clear evidence that one variation is more effective than the other and should be implemented.
Flat test: A flat test is when there is no significant difference in performance between the two variations being tested. This means that both variations perform similarly and there is no clear winner. While a flat test may not provide a clear winner, it can still provide valuable insights into customer behaviour and guide future testing.
Losing test: A losing test is when one variation significantly underperforms the other variation in terms of the desired outcome. For example, if the desired outcome is conversions, a losing test would be when one variation results in significantly fewer conversions than the other. A losing test can provide clear evidence that one variation is less effective than the other and should be avoided.

Learning to love your losing tests

Flat and Losing A/B tests provide valuable insight into what doesn’t work. While winning A/B tests provide clear evidence of what works best, losing tests can be just as informative as they demonstrate what strategies or design elements are less effective.

By celebrating Losing A/B tests it is possible for airlines to avoid costly mistakes and guide future testing. Understanding the reasons why a variation performed poorly, means businesses can learn what changes to make in future iterations to improve performance.

Moreover, Losing tests can also help airlines avoid making assumptions about what customers want. By testing multiple variations, businesses can make data-driven decisions based on real-world evidence rather than gut feel or accepted standards. The results of Losing tests often provide feedback that contradicts what the business thought would work best – and this can be hard to take! However, those airlines that embrace these uncomfortable truths can adjust their strategies to ensure that they are delivering the best experience for their customers.

Making the shift

Experimentation and web optimisation in the short term undoubtedly can produce good results, but these will be in the short term. The key in 2023 is to shift to a culture of experimentation enabling testing at scale to deliver transformative business results in the long term. To misquote Aaron Levenstein: “Optimization is like a bikini. What it reveals is suggestive, but what it conceals is vital.” Without continuous improvement all you have is suggestion.

Share This Article