How to write a change management survey that is valid

How to write a change management survey that is valid

An important part of measuring change is to be able to design change management surveys that measure what it has set out to measure.  Designing and rolling out change management surveys is a core part of what a change practitioner’s role is.  However, there is often little attention paid to how valid and how well designed the survey is.  A survey that is not well-designed can be meaningless, or worse, misleading.  Without the right understanding from survey results, a project can easily go down the wrong path.

Why do change management surveys need to be valid?

A survey’s validity is the extent to which it measures what it is supposed to measure.  Validity is an assessment of its accuracy.  This applies whether we are talking about a change readiness survey, a change adoption survey, employee sentiment pulse survey, or a stakeholder opinion survey.

What are the different ways to ensure that a change management survey can maximise its validity?

Face validity.  The first way in which a survey’s validity can be assessed is its face validity.  Having good face validity is that in the view of your targeted respondents the questions measure what they aimed to measure.  If your survey is measuring stakeholder readiness, then it’s about these stakeholders agreeing that your survey questions measure what they are intended to measure.

Predictive validity.  If you really want to ensure that your survey questions are scientifically proven to have high validity, then you may want to search and leverage survey questionnaires that have gone through statistical validation.  Predictive validity means that your survey is correlated with those surveys that have high statistical validity.   This may not be the most practical for most change management professionals.

Construct validity.  This is about to what extent your change survey measures the underlying attitudes and behaviours it is intended to measure.  Again, this may require statistical analysis to ensure there is construct validity.

At the most basic level, it is recommended that face validity is tested prior to finalising the survey design.

How do we do this?  A simple way to test the face validity is to run your survey by a select number of ‘friendly’ respondents (potentially your change champions) and ask them to rate this, followed by a meeting to review how they interpreted the meaning of the survey questions.

Alternatively, you can also design a smaller pilot group of respondents before rolling the survey out to a larger group.  In any case, the outcome is to test that your survey is coming across with the same intent as to how your respondents interpret them.

Techniques to increase survey validity

1. Clarity of question-wording.

This is the most important part of designing an effective and valid survey.  The question wording should be that any person in your target audience can read it and interpret the question in exactly the same way.  

  • Use simple words that anyone can understand, and avoid jargon where possible unless the term is commonly used by all of your target respondents
  • Use short questions where possible to avoid any interpretation complexities, and also to avoid the typical short attention spans of respondents.  This is also particularly important if your respondents will be completing the survey on mobile phones
  • Avoid using double-negatives, such as “If the project sponsor can’t improve how she engages with the team, what should she avoid doing?”

2. Avoiding question biases

A common mistake in writing survey questions is to word them in a way that is biased toward one particular opinion.  This assumes that the respondents already have a particular point of view and therefore the question may not allow them to select answers that they would like to select.

Some examples of potentially biased survey questions (if these are not follow-on questions from previous questions):

  • Is the information you received helping you to communicate to your team members
  • How do you adequately support the objectives of the project
  • From what communication mediums do your employees give you feedback about the project 

3. Providing all available answer options

Writing an effective survey question means thinking through all the options that the respondent may come up with.  After doing this, incorporate these options into the answer design.  Avoid answer options that are overly simple and may not meet respondent needs in terms of choice options.

4. Ensure your chosen response options are appropriate for the question.

Choosing appropriate response options may not always be straightforward.  There are often several considerations, including:

  • What is the easiest response format for the respondents?
  • What is the fastest way for respondents to answer, and therefore increase my response rate?
  • Does the response format make sense for every question in the survey?

For example, if you choose a Likert scale, choosing the number of points in the Likert scale to use is critical.  

  • If you use a 10-point Likert scale, is this going to make it too complicated for the respondent to interpret between 7 and 8 for example?  
  • If you use a 5-point Likert scale, will respondents likely resort to the middle, i.e. 3 out of 5, out of laziness or not wanting to be too controversial?  Is it better to use a 6-point scale and force the user not to sit in the middle of the fence with their responses?
  • If you are using a 3-point Likert scale, for example, High/Medium/Low, is this going to provide sufficient granularity that is required in case there are too many items where users are rating medium, therefore making it hard for you to extract answer comparisons across items?

Example of survey design interface and reporting from Change Automator

5. If in doubt leave it out

There is a tendency to cram as many questions in the survey as possible because change practitioners would like to find out as much as possible from the respondents.  However, this typically leads to poor outcomes including poor completion rates.  So, when in doubt leave the question out and only focus on those questions that are absolutely critical to measure what you are aiming to measure.

6.Open-ended vs close-ended questions

To increase the response rate, it is common practice to use closed-ended questions where the user selects from a prescribed set of answers.  This is particularly the case when you are conducting quick pulse surveys to sense-check the sentiments of key stakeholder groups.  Whilst this is great to ensure a quick, and painless survey experience for users, relying purely on closed-ended questions may not always give us what we need.

It is always good practice to have at least one open-ended question to allow the respondent to provide other feedback outside of the answer options that are predetermined.  This gives your stakeholders the opportunity to provide qualitative feedback in ways you may not have thought of. 

To read more about how to measure change visit our Knowledge page under Change Analytics & Reporting.

Writing an effective and valid change management survey is often glanced over as a critical skill.  Being aware of the above 6 points will get you a long way in ensuring that your survey is designed in a way that will measure what it is intended to measure.  As a result, the survey results will be more bullet-proof to potential criticisms and ensure the results are valid, and provide information that can be trusted by your stakeholders. 

The #1 success factor in driving agile changes

The #1 success factor in driving agile changes

There are many facets of driving agile changes.  Agile changes are featured by such as developing minimum viable product and not investing too much initially, developing a series of iterations to gradually improve the product, engaging stakeholders early and frequently to ensure the outcomes meet business needs, developing working product/solutions from which feedback may be sought to feed iterative improvements prior to final release.

With so many facets of implementing agile changes, what is the most important part of driving agile changes?  What is the core concept that must be done right without which the change would not be considered ‘agile’?

One of the most critical parts of agile change is the concept of developing a hypothesis that can be tested.  The outcome must be clear in terms of whether the solution developed meets the business needs or not.

Why hypothesis?

In waterfall methods of delivering projects, the focus is on spending significant focus understanding and detailing features and ‘requirements’ from the business.  From these, the solution is then designed and developed.

The problem with this approach is:

  • Significant resources and investment may be required to sufficiently develop the solution depending on the complexity involved
  • It may also take a long period of time to involve various stakeholders and investigate solution design options before a final product can be developed.  A series of design decisions also need to be made, each step taking time to undergo
  • The business may not know what they want and they would need to provide ‘requirement’s that may or may not meet their needs.  For example, prior to the launch of iphones, touch screen phones were not popular and were not seen as the design of future phones
  • The risk can be significant if the solution developed does not meet business needs.  Millions of dollars of project investment could have been wasted if this is the case.   

On the other hand, what is the advantage of a hypothesis based approach?

  • Does not spend a lot of time creating a sophisticated solution or product.  Instead, a simplified version is developed which captures the core of business need.  This is then tested, and then the results can then feed into further improvements required.  In this way, the process allows organisations to fail early and cheaply in order to eventually come up with the winning solution
  • Instead of focusing on detailed planning which is based on a series of assumptions which may not have been tested to be valid, the focus is on deriving a solution that CAN be tested and validated or invalidated.  This is especially important when the solution is new and has not been implemented previously in the organisation
  • The hypothesis approach is a scientific approach where the focus is on proven results based on data.  In the same way a laboratory technician would conduct a series of experiments to test the properties of a chemical solution to further understand it, in the same way the project team would conduct a series of ‘experiments’ (or iterations) to gradually test and from testing results, improve the solution
  • Tests are always based on ‘real’ data and real scenarios therefore there is a much greater chance that the final solution will meet business needs

The importance of a hypothesis-approach for organisational agility

The survival and growth of a company are dependent on its ability to go into different products, different territories or different customer groups to expand its offering.  In order to do this, the company needs to ultimately launch various products or services that do not exist currently or that have not been launched in certain new areas/segments. 

Therefore, the ability of the organisation to continuously develop, launch and learn from new products and services is critical for its success.  Each product launch is a new hypothesis that is to be tested.  And with each testing, a set of learning is achieved which will improve its next product launch.  In this way, this is how companies become agile and develop the ability to flex and change based on its ability to generate hypotheses.

For digital businesses developing hypothesis is a core way of operating.  A hypothesis can be as small as testing the wording of the website using A/B Testing to see which wording is more engaging for website visitors.  A/B Testing is where a certain number of visitor traffic is channeled into one version of the website versus another version.  And the results of visitor interactions can be used to validate which version is more engaging.

Change management hypothesis testing

To truly adopt a hypothesis-based approach to change management one needs to adopt change hypothesis testing.  What is change hypothesis testing I hear you ask?  It is basically developing a series of small change experiments to test assumptions.  Change experiments are important because they help to inform what change tactics or approaches work or do not work.

Some examples of change experiments include:

  • Wording of campaign phrases or positioning
  • Email click-through rate based on details such as who email is from, time of delivery, etc.
  • Effectiveness of training exercises
  • Employee awareness after town hall messages
  • Website effectiveness
  • Impact assessment approach effectiveness
  • Campaign medium effectiveness such as freebies, posters, etc.

However, it is critical to ensure that hypothesis to be tested is not time nor resource intensive.  The experiment must also be tested using feedback data.  The hypothesis cannot be proven or disproved unless it is backed by hard data and not just opinions.

4 change leadership lessons from these 2 prime ministers

4 change leadership lessons from these 2 prime ministers

Australia and New Zealand are like 2 brothers. One big brother, Australia,
and the smaller brother New Zealand. We are culturally similar and speak
with almost the same accent (almost but not quite the same). Both
countries have experienced recent tragedies and challenges. However,
there are 2 very different prime ministers. Let’s explore what we can learn from
these two leaders within significant change events.

New Zealand

On 15 March in Christchurch New Zealand, there was a mass shooting at 2
mosques resulting in 51 killed and 49 injured. This has cut through the
psyche of New Zealand quite deeply as it was the first time the country had
experienced mass shooting at this scale. Being a small country with a
relatively liberal and tolerant culture this came a shock for most.

Jacinda Arden, the Prime Minister of New Zealand, reacted swiftly. Within
a few hours of the event she addressed the terrorist directly
demonstrating strength and determination. She quickly flew into
Christchurch to visit survivors and their relatives. Dressed in black head
scarf, she visited mosques and asked how she could support the mosques
and the victims. Within a few days of the event she also called out
blatantly the responsibility of social media platforms in hosting hate
messages which was the case for this incident as the attacker posted
Facebook messages prior to the attack.

She then made sweeping changes to gun laws in New Zealand banning all
assault rifles and military-style semi-automatics. This happened within a
few days of the event and though some may argue that this is much easier
to achieve in New Zealand than the US but the point is that she acted
swiftly and had even convinced the conservative opposition party to enact
on this law.

4 key lessons we can learn from her example as a change leader include:

1. Displaying agile leadership. She proactively faced into a catastrophic
situation and worked with others to address the situation head-on.
She made fast and clear decisions to resolve and contain the

2. Authenticity. She spent time with those affected by the tragedy and
showed empathy and care. This wasn’t about the photo
opportunity as it was more about spending time to listen and show
care for those impacted by change. She didn’t try to be someone
she is not. Instead of the antagonistic and hostile speeches that one
might expect from leaders like Trump, her words were empathic,
strong and unwavering.

3. Displaying emotional connection . She also placed herself in the
shoes of those affected by the tragedy with her cultural sensitivity
and emotional connection to those impacted. The grieving was not
only felt by those involved in the tragedy, the whole nation was
grieving. Her visibility was critical to speak for the nation but also to
acknowledge everyone’s emotional state and concerns. The critical
word here is ‘visibility’. Felt emotional connection wont garner
groups of people if they are not displayed.

4. Collaborating with others to drive change. A series of changes
ensued not just gun law changes, but also driving security, and
social media regulation changes. In an interview she used the words
“duty of care as a leader” to safeguard her people and address their
concerns. She is not just speaking for herself, but also for other
leaders, including business leaders, to step up and take action. She
also influenced various world leaders on the same agenda to rally


Right now in Australia, at the time of writing, we are still in the middle of a
catastrophic set of fires raging across most states of Australia. More than
1300 homes have been burnt down and 18 people have died so far. In
Sydney, we have had more than 2 months of smoke haze in our air
resulting from bushfires, and sometimes the air quality can be 11 times
more than ‘hazardous’ level. This is absolutely the worst I have ever
experienced in Australia. This morning, I received the message that at the
southern highlands where I spend Christmas, the area is surrounded by
bushfires and residents have all been evacuated.

Let’s have a look at how our Prime Minister has lead the country during
this period of environmental change. Unlike the leadership we’ve seen
from Jacinda Ardern, Scott Morrison our Prime Minister flew out with his
family to Hawaii to spend holidays by the water. Whilst the country is
burning and people are suffering, even under intense criticism, our prime
minister was absent and away. When prompted to address serious
climate change issues, he responded by saying that it was not the time to
talk about climate change.

Eventually after continued public pressures, after Scott Morrison came
back from holidays he proceeded to visit some of the towns completely
destroyed by bushfires. Many of the victims refused to shake his hand. In
the business world we have also seen this type of reaction from those
who felt they have been deserted and have not received any leadership
support. There have even been incidents where the victims have asked
Scott questions and he had ignored them and moved away, then later on
quoting how he had promised help for them.

Whilst fires continue to burn through our states, the Prime Minister’s
party released a party propaganda social media tweet proclaiming the
party’s prowess in helping Australians through supporting firefighters,
listing the financial assistance offered as a part of the package. An
Australian TV panellist said this was like “being ‘sold to’ at a funeral”. It
was completely inappropriate and badly timed.

In terms of the same change leadership lessons we had captured from
Jacinda Ardern, what can we also learn from Scott Morrison’s change
leadership example?

1. Displaying agile leadership. Lack of action and decision at the
commencement of the change is almost unforgivable. It is very hard
to salvage from the lack of leadership support when at this pivotal
moment when there is no leadership action or response.

2. Authenticity. Unfortunately, authenticity by definition cannot be
faked nor acted. People see through the actions and inactions of a
leader. There is no amount of corporate communications packaging
nor word-smithing that can change how others experience through
change leadership, or the lack of. Being open and transparent
remains the best approach for any change leader.

3. Displaying emotional connection. It is difficult to fake emotional
reaction. Through overall body language as well as tonal cues
people can easily pick up on a leader’s ability to connect
emotionally. When people are in distress and in suffering, the best
approach is to simply listen and show that you have heard them.
Ideally, you are also able to address at least some of their core
concerns. But the critical must-have remains how a leaders
displayed active listening and showing that he or she cares.

4. Collaborating with others to drive change. What Australia needs is
global leadership to drive climate change and to work with various
agencies and leaders, the same way that Jacinda Ardern has been
doing with New Zealand‘s agenda. Several countries have proactive
offered support in fighting bushfires even without Scott Morrison
reaching out to tap on others.

Change is all around us, not just in the organizations that we work in.

In the same way, change leaders are also all around us.

Leading change is an absolutely critical skill to master and will well into the future.