“Not everything that can be counted counts, and not everything that counts can be counted.”

When Einstein uttered these words little did he know that he was stating the case for techniques like Most Significant Change (MSC).

MSC is a simple process for helping senior decision-makers develop a gut feel for what an initiative has achieved. It’s not a replacement for gathering and analysing the numbers. Rather is a supplemental evaluation that helps to systematically develop decision-makers' intuitive knowledge. And research shows that many of the decisions we make are based on our judgements and intuitive, so it’s a part of our knowledge we mustn’t ignore.1 2

Here's how MSC works. It can be done in 4 steps.

STEP 1 - COLLECT STORIES OF SIGNIFICANT CHANGE

The process starts by asking two simple questions of the people affected by the initiative of interest.

1. What is the most significant change that happened since the initiative started?
2. Why is this change significant for you?

STEP 2 – IDENTIFY AND ASSEMBLE THE DECISION-MAKERS WHO NEED TO KNOW WHAT REALLY HAPPENED AS A RESULT OF THE INITIATIVE.

This step is crucial to the success of the evaluation and consists of the evaluation designers asking the question, “Who needs to know, in their gut, the impact this initiative is having?”. These decision-makers could be at any level in the organisation, in any location. The evaluation designer then arranges the decision-makers into groups of 6-8 people and arranges for these groups to meet for 90 minutes or so to consider the significant change stories collected in Step 1.

STEP 3 – SELECTING THE MOST SIGNIFICANT CHANGE STORY

When you have your decision-makers in a room the facilitator guides the group in a discussion about 4 to 6 of the stories that where selected. In fact, we encourage the group to read each story then argue why they think a story is most significant. This discussion helps embed the stories in the minds of the participants while raising issues of strategies and implementation. The participants experience a lively debate and get to know one another and the issues affecting people in the field. Most importantly they develop an intuitive understanding of the impact the initiative is having. At the end of the session the group agrees on a most significant story and describes why they selected it. They also identify actions they will take to reinforce the good things that are happening and disrupt the undesirable outcomes.

The result is communicated to the original storytellers. The most significant change story from each group is then made available to the next level in the organisation, such as an executive group, who repeats the process with the subset of stories.

STEP 4 – MAKING THE STORIES AND WHAT WAS SELECTED AVAILABLE

The evaluation concludes by collating all the stories and creating a document that includes which stories were selected and why.
Invariably lessons are learned during the process and these ideas can be then fed into a continuous improvement process.
The selection process is frequently scheduled to occur on a regular cycle. Organisations that use MSC often select a period of between selections of 3-6 months to evaluate ongoing change.

Additional resources

How to conduct an MSC selection workshop

Dart, J. and R. Davies (2003). “A Dialogical, Story-Based Evaluation Tool: The Most Significant Change Technique.” The American Journal of Evaluation 24(2): 137.

A short history of MSC

Zahmoo - software for supporting MSC projects

References

1. Klein, G. (2003). Intuition at Work. New York, Currency Doubleday.

2. Westen, D. (2007). The Political Brain: The Role of Emotion in Deciding the Fate of the Nation. New York, PublicAffairs.

Bob Sutton is on a campaign against workplace arseholes. In yesterday’s post he describes Rob Cross’ work on social network analysis. In particular he looks at how to identify people who energise and de-energise.

Bob’s interesting in ways to measure the impact of arseholes.

I am trying to figure out some ways and places to measure this stuff, and am hoping to recruit Rob to help as has some really cool software that he uses with the companies that he works with and that are partners in his network.

One technique he might consider is Most Significant Change. While this technique wont create a measure of arseholeness, it will give people in the organisation a very good understanding of what’s happening and provides a forum to address some of the issues. Very soon he will be able to use the Zahmoo software to support the technique.

People have heard that storytelling is great for dealing with tacit knowledge. They say things like, “If we could only capture our stories we could then capture our organisation’s tacit knowledge.”

This is the big mistake! Stories only have meaning in the context of their telling. That is, you need to tell and listen to stories to transfer (not capture) tacitly held knowledge. It’s a social process. You need to be part of the conversation.

In practice, this means creating spaces for stories to be told and listened to. We do it in a bunch of different ways depending on the needs and objectives of our clients.

For example, if we are helping tackle complex issues such as trust, leadership, culture change, we would create the space in sensemaking workshops.

If we need to evaluate the impact of difficult to measure initiatives we create the space using Most Significant Change and the selection workshops.

NASA creates this space for staff to listen to and tell stories in their monthly project management seminars where PMs discuss the stories collected in the their monthly newsletter, ASK.

Everyone is busy and no one will give up their valuable time to listen and tell stories. But they will allocate time to evaluate a project, tackle a complex problem or learn lessons from their colleagues.

The stories don’t contain magical solutions that we can capture, dissect and unleash. Rather they provide a language of engagement, of learning and a way to transfer what is impossible to write down and store in any database.

An article titled ‘The power of ordinary practices’ was the seventh ‘most read’ of Harvard Business School’s Working Knowledge articles for 2006. The articles includes the following:

One of our projects has involved collecting about 250 anecdotes from within a large multinational on the theme ‘values in action’. The anecdotes were used as part of a management development program. After short-listing the anecdotes, teams went through the most significant change process to identify anecdotes that provided the best examples of behaviours they should model. The following anecdote was selected as the most significant by one of the teams.

That something so innocuous has such impact reinforces the ‘impact of ordinary practices’. As we regularly comment – little things can make a big difference. But, you can tell managers this sort of thing a hundred (bazillion) times without it really sinking in. So, here we see some of the power of narrative – a simple anecdote has had a major impact upon a group of senior managers by giving them a powerful example of the effect their behaviour has on others. 

I have just posted an interview with Jess Dart, co-developer of the Most Significant Change technique, over at the Zahmoo blog.

I met with Rick Davies this week. He’s in Melbourne visiting his family and doing some work for Oxfam. We talked about the Zahmoo project and he made some very helpful suggestions. Rick asked me to make a link from the Zahmoo home page to the MSC guidebook that he and Jess Dart put together, which is now there. If you want to know how to do MSC this is the resource.

Rick made an interesting observation. He asked why we make so many of our ideas available noting that this behaviour was very unlike most consultants. Before I could answer he said, “would you like to be remembered for one idea or would you like to known as someone who can create and implement many ideas? It seems your website demonstrates the latter.”

Most Significant Change is a monitoring technique based on the collection and selection of stories. The technique involves collecting stories, gathering people together to talk about them and then selecting the stories they believe are the most significant. This selection process creates new conversations in an organisation while systematically developing an intuitive understanding among staff of a program’s impact. Here is a short history of the technique.

Today we are announcing that we will soon launch a new web 2.0 innovation that will help you run your most significant change projects. The project is called Zahmoo and if you want to get an early look at the application, sign up as a beta user here.

Decision-makers are under increasing pressure to justify their decisions and then account for their success (or otherwise) to a variety of stakeholders. Evidence-based management (1) is further increasing this pressure. While we know intuition plays a significant role in decision-making (2-4), large decisions (Will we merge and how? How will we change our culture?) will require thoughtful deliberation as well as experimentation. A conundrum emerges, however, when dealing with programs designed to change behaviours: how do we know that the program of activities is responsible for the change?

Initiatives designed to change behaviour are notoriously difficult to assess using traditional techniques. Let's take a learning initiative called After Action Reviews (AAR) as an example. This intervention is designed to create knowledge and new behaviours through personal and group reflection. A group is facilitated to answer and discuss 3 questions in relation to a current or recently completed project: What was supposed to happen? What happened? What accounts for the difference?

Once a program of after action reviews is in place, is it the AARs or something else creating new knowledge? This knowledge, the argument goes, should create new behaviours. But it is the knowledge gained from the AAR, or something else, creating the behaviours? Finally these new behaviours should impact organisational outcomes. Again, are the new behaviours creating the impact or something else? There are two many causal links in this complex system to know for sure (5).

Assessing hard facts alone is insufficient in helping stakeholders appreciate the impact of a program designed to change behaviours. Qualitative perspectives are essential. The need for a balance to hard facts is heightened by the increasing number and variety of stakeholders involved, each one having their own criteria and needs. A new method of evaluation was required and its development occurred in the most unlikely place.

To date the application of Most Significant Change has been mostly confined to NGO programs and other not for profit organisations. But this is changing. Corporations are also recognising that issues such as culture change, communities of practice, learning initiatives generally and leadership development could benefit from an MSC approach. Anecdote is currently assisting one large IT and consulting company implement MSC to evaluate the impact of its culture change program.

Jessica Dart (an Anecdote Associate) and Rick Davies have published the technique in the prestigious American Journal of Evaluation (7) and have made a guide freely available to anyone interested in implementing the technique. Anecdote and Jess Dart have teamed up to provide support services to corporations and public sector agencies to help them get the most out of Most Significant Change.

References

1. Pfeffer, J.; Sutton, R. I. Hard Facts: Dangerous Half-thruths & Total Nonsense: Profiting from Evidence-based Management. Harvard Business School Press: Boston, MA, 2006.

2. Gladwell, M. Blink: The Power of Thinking Without Thinking. Little, Brown & Company: New York, 2005.

3. Klein, G. Intuition at Work. Currency Doubleday: New York, 2003.

4. Gigerenzer, G.; Todd, P. M.; ABC Research Group. Simple Heuristics That Make Us Smart. Oxford University Press: Oxford, UK, 1999.

5. Dixon, N. The Organizational Learning Cycle: How We Can Learn Collectively. Gower Publishing Company, 1999.

6. Davies, R. An evolutionary approach to facilitating organisational learning: An experiment by the Christian Commission for Development in Bangladesh. Centre for Development Studies: Swansea. UK, 1996.

7. Dart, J.; Davies, R. A Dialogical, Story-Based Evaluation Tool: The Most Significant Change Technique. The American Journal of Evaluation 2003, 24, 137.

Would you like to streamline how you do Most Significant Change? Check out zahmoo.