The brief is ‘everything’.

“A brief is not a vending machine – you don’t just drop in a few coins and expect something perfect to drop out. If it’s worth doing, it’s worth briefing properly”.
One really key reminder for me, and this should be obvious: the brief truly is everything. Across the day the concept of a strong brief came up in many formats.
As an example, during Madison Hajeb’s (Senior Director, Testing and Innovation at Tapestry) talk about experimentation, she mentioned a core part of her process is a Test Intake Form. This essential briefing tool:
• collated all the valuable and vital information to formulate an experiment from; less chasing, less Q&A, more doing.
• provided a format for her testing team to push back, validate the brief, tie it properly into their strategy and prevent HiPPO effects from the business.
• created a clear process for her team to prioritise tests and build a roadmap of activity across the year.
• allowed everyone within the business to feed into testing and experimentation, creating a culture that could see value and share success in experimentation.
o Madison mentioned that one of the most successful tests ever created for Tapesty came from a completely different, and unlikely, department within the business.

"The median company runs 34 experiments a year”.

If you’re serious about experimentation, it needs to start somewhere. Anywhere. And then scale. Many talks at Opticon centered around the common issue; experimentation starts as a small, infrequent and sometimes uninspiring programme of work. There’s a lot of pushback, questions and scepticism if your company culture is not already set up to fully back experimentation as a concept.
A lack of positive results can be seen as failure rather than a chance to learn. But if you stick with it, there’s the opportunity to build momentum and optimise the process to create a lot of reward. Some top tips to scale out experimentation programmes included:
- Involving developers into your programme to code larger changes that impact the customer more significantly, providing bigger potential reward (beyond simpler copy, colour or content ordering tests).
- Running more tests concurrently and being flexible in your statistical significance levels (being mathematically ‘Pure’ is not as vital as producing business value).
- Testing a higher number of variations concurrently (i.e. moving beyond A/B tests).
- Leverage experimentation to incorporate a personalisation strategy; use tests to try out personalisation tactics and measure their effect before committing to a complex personalisation implementation.
Optimizely Web and Feature Experimentation offer multi-armed bandit algorithms; which allow you to test multiple variations at the same time, whilst gradually moving traffic towards winning variations to achieve the test goal, instead of forcing you to wait for an experiment to end to learn the result. This process maximises reward by getting to a positive outcome, faster, although this can be at the expense of a definitive, statistically significant result, so should be chosen with care.

Opticon 2024 - What I took home

Test planning – the quest for success