In the world of international development and social enterprise, Impact Assessment is a big topic these days. How do you answer the ‘so what’ question – the idea that performance should ultimate be measured in terms of changes to people’s lives, not just our ability to deliver activities.
In the world of business and technology, it is Big Data that is grabbing all the headlines. Essentially, Big Data is about spotting previously hidden patterns in large amounts of structured and unstructured information. It promises to connect data from lots of different sources (for example, weather data with sales data, or voting behaviour with celebrity endorsements), and relies on lots of computing power and smart physicists.
These two worlds may appear to be separate, but in our work we often find the ideas from business and technology filtering through into the way people think about Impact Assessment. We came across a prime example of this in a recent piece of work that looked at Impact Assessment for a capacity-building charity.
As we did our research about existing Impact Assessment frameworks in this area, we continuously came across the same issue. At the highest level, frameworks would reflect a clear and coherent vision of the key components of a capacity development programme. But once it got down to measurement, they all resorted to enormous lists of indicators – data about the things that it is thought can feasibly be counted.
The use of hundreds of indicators was not an approach that suited our client, and this got us thinking: it probably suits very few social impact organisations in general.
The push to managing by results has led many organisations to start collecting and storing large amounts of data. Ironically we see very little of this successfully used to better manage projects in pursuit of results. At the level of Impact, we are often presented with large data sets, in the hope that we can work some evaluative magic and reveal the insights that all this data is supposed to contain.
This is the Big Data approach – throw enough information into the pot and something valuable must come out, right? In our experience – not really.
Part of the reason this is so challenging is that very few non-profit organisations and social enterprises are able to collect large, consistent, and high quality data over a long period of time. Each project often collects its own data, in its own way – even at the organisation level integrating these data sets becomes a challenge.
This situation is compounded by the nature social impact organisations: few have the budgets to support the teams of computer programmers and physicists needed to make Big Data work. Few donors are willing to support large investments in data analysis at the expense of direct delivery of programmes.
The idea of Small Data
Small Data is an idea that we first heard spoken about by the CEO of Evernote. Evernote is a way of storing all your notes, web-clippings, documents, and even drawings in a single place. These are all made available on every computer, smart phone and tablet on which you have installed the free Evernote application.
Evernote is a darling of the tech industry – the same industry that is obsessed with Big Data. When he was asked what Evernote will do about Big Data, Phil Libin – the CEO – said that they were more interested in Small Data: getting information and lessons out of all the material that people and organisations already have.
When you think about it, most social impact organisations already have enormous amounts of data, information and knowledge stored away in Word documents, Excel sheets, field reports, meeting minutes, presentations, emails, and – most importantly – people. The biggest problem they face: mining this information, making sense of it, and learning about the real impacts it reveals.
A practical way of realising Small Data is a tool that Evernote has based on ‘gravitational search’. When you add a new note to your Evernote account, it searches all your existing notes and presents you with related information that you have previously stored. So, if you save a webpage about protection in emergencies from the Global Protection Cluster, then Evernote might also show you a PDF guideline from Save the Children that you had previously saved.
This is powerful for an individual, but can be transformative for a team. Evernote for Business lets whole organisations share notebooks. So if a programme officer is drafting a note about strengthening the governance of partners, Evernote can track down field reports saved by other members of staff in which governance was a key theme. Suddenly, learning from the wealth of knowledge that an organisation already has can become a part of normal working habits.
Impact Assessment and Small Data
If one practical tool for utilising Small Data is Evernote, one approach to systematically capturing knowledge about impact is the Collaborative Outcomes Reporting Technique (CORT) developed by Jess Dart (who also developed Most Significant Change with Rick Davies). You can read more about both CORT and MSC at betterevaluation.org
According to Jess, “CORT is especially useful when a program has emergent or complex outcomes that are not fully defined at the onset of a program. For this reason a theory of change is refreshed at the start of the evaluation process. In addition qualitative inquiry is used to capture unexpected outcomes and deliberative process are used to make sense of the findings.”
So, there we go: Evernote and Collaborative Outcomes Reporting. Two Small Data ways to make a Big Difference to the ability of social organisations to gains insights about impact from the data they already have.