Change Experiment Playbook

Design and test assumptions so as to learn early

to maximise chances of ultimate outcome success.

Go to Instructions

60 min

3+ People

What you’ll need

REMOTE

Video conferencing with screen sharing

Digital collaboration tool (Zoom, Hangout)

IN-PERSON

Whiteboard

Markers

Post-it notes

Timer

Instructions for running this Playbook

1. Background

Experimenting is a core part of agile and an integral part of continuous improvement. Rather than leaving the ultimate change outcome to chance, this approach is replaced by continuous experimenting. A lot of change practitioners leverage their previous experience.

However, this risk could be that different changes, audience, timing, and impact of other changes mean that change approaches that have worked before may not work in this case. This is a big risk. As organisations continue to evolve and change, continuous change testing becomes ever more critical. Learnings from change tests can provide significant value for future projects.

2. Prep

Scoping:

Ideally the experiment should be small and discrete. It should be simple and fast to test without too much effort or complexity. Some examples include:

Test the change vision message with one team of call centre agents through a short survey
Test one part of the learning module and look at learning test results, time it takes to go through the module and participant interview results
Test a formal announcement message to change champions and assess open rate, click-through-rate and perception of the content through a short survey

Scheduling:

Schedule the meeting when you have sufficient assumptions about the change approach. So, this would likely be in the initial planning stage of the project and well before delivery starts.

Who:

Include project team members such as the project manager, other change team members and business analyst. Also include business representatives.

Organisation:

Prior to the session undertake the following preparation:

Research previous change approaches used within the same audience group by interviewing previous project team members and business contacts. Find out about the details of the nature of the project, change approach used, and outcomes
Study retros of previous projects impacting the same audience group
Talk to business representatives to understand the history of changes that they have been through, how the group has evolved over time through changes. Note any potential stakeholder biases on particular change approaches that have or have not worked (stakeholder opinions often may not equate to actual facts)
Come up with one or two ideas of change experiments you would like to use

TIP: SCHEDULING

Conduct several small change experiments early on in the project especially during the planning phase as well as the earlier part of the deployment phase.

TIP: DESIGN

After initial change experiments, later experiments can be larger and more complex. This is after there is already learning and confidence of what will work or not work from previous experiments.

Don’t be afraid of failed experiments. This is good! With every failure is a potential lesson learnt!

3. Run the session

Introduction – 5-10 min

Introduce participants, the overall session design and flow. Go through session flow, objectives, and outcome anticipated.

Go through each of the 4 segments – 45 min

Run through the core parts of the change experiment design, using the Test Card from Strategyzer which can be downloaded here.

The hypothesis is the assumption you are going to use in the experiment, “we believe that ……”. For example:

We believe that impacted employees would prefer to find out about the change through their managers
We believe that sending updates through a project Teams channel is more effective than using email
We believe that team leaders would prefer to know about the change as early as possible, even when we have not defined all the change details
We believe that breaking down each learning module to 10 minute chunks is more effective than the previous 15 minute chunks that the impacted teams were used to with previous modules

Define the method to execute on the hypothesis. “To verify that, we will …..”. For example:

To verify that we will send the change upate for half of the test team through their managers and the other half by email
To verify that we will project updates through a project Teams channel for half of the test group and the other half through emails
To verify that we will update one group of team leaders in their team meeting very early on in the project planning stage and the other group later on when we have defined the overall plan
To verify that we will roll out a test module for 1 team with the 10 minute module and the other team with the 15 minute module

Measure the outcome of the experiment. It is very important to define a clear demonstrable way of measuring the outcome. You won’t want to be in the situation of arguing with business stakeholders about what the outcome is. There should not be different ways of interpreting the outcome through the measures put in place. Examples might include:

We will conducted a survey to both groups
We will conduct a focus group to ask for participant feedback
We will analyse session test results and time taken to complete the module
We will look at click-rate of emails and article viewership rates
We will examine test results of change initiative understanding

Conclusion of experiment outcome. “We are right if …….”. This is basically saying that the hypothesis or assumption is correct if the following criteria is met. This is another critical part of the experiment as it determines the outcome success criteria. For example:

We are right if the survey results of the test group is 20%+ more effective than the other group
We are right if ratings assigned to feedback items for test group is 30%+ more effective
We are right if test results in test group is at least 15% higher than the other group
We are right if the click-rate of emails and article viewership rates of test group are 10%+ higher

Capturing outcomes – 5 min

During the session scribe the contribution from the group to fill in the Card. Don’t worry about trying to perfect any wording since this can be done after the session.

Also agree with the team on next steps including:

Ownership in executing the experiment, i.e. who owns which part
Timeline for executing the experiment
Milestone for reviewing results and deciding on the outcome of the experiment
Build-in the project plan any linkages with other delivery activities based on the experiment outcome (e.g. confirming broader change approach)

What to do with the output?

After you’ve written up the Test Card forward this to participants to confirm.

Place all change experiments in a central folder that may be accessed by all project teams to share the findings.

TIP: INTRO EXPERIMENTATION

If the team is not familiar with the concept of agile experiments (in particular business representatives), talk about why a change experiment is needed to set the context. Some of the reasons include: