Fiona:
Oh, kia ora and welcome everyone to Making Your Data Work. I'm Fiona Scott-Melton and I'll be chairing today. We have people with us from across New Zealand and Australia.
Brendan is here with me in Wellington and Caroline is joining us from Melbourne. Before we start, I’d like to do an acknowledgement of Country. Our Allen + Clarke office in Melbourne is on the traditional Country of the Wurundjeri Woi-wurrung and Bunurong peoples of the Eastern Kulin Nation.
We acknowledge that they never ceded their land and pay our respects to Elders past, present and emerging. And for those joining from New Zealand – Ngā mihi nui ki a koutou. So before we get started, two bits of housekeeping. This session is being recorded today and a copy of the slides will be shared later this afternoon.
And please pop your questions in the chat whenever they come to you. We'll gather them up and put as many as we can to our presenters in the last 15 minutes. So now over to our presenters.
Brendan.
Brendan:
Tēnā koutou, ko Brendan Stevenson ahau. I'm a senior consultant here at Allen + Clarke. I've been here about six years now. Before that, I worked at Te Hiringa Hauora Health Promotion Agency at Massey University.
But I still have a role as co-leader on a longitudinal study of ageing that ups over a quarter century of statistical and data nudiness. Caroline.
Caroline:
Hi everyone. Thanks, Brendan. I'm Caroline Crothers, a senior consultant here at Allen + Clarke based in Melbourne.
I've been here about five years and previous to that, I've worked in a range of roles in policy and international development and global health, including coordinating a couple of large scale RCTs. So I've seen firsthand how messy data can be.
Fiona:
OK, hey, thanks both. So decisions about funding and programmes get made on data. And data that looks good can still mislead us, right?
So today we're going to use an example that will show how easy it is to overestimate the contribution a programme is making. We'll show that it's important to take into consideration how the data was designed, who recorded it, who made it into the data set, what counted as a record. We'll also look at what good data, quality, hygiene and culture look like, and finish up with how you can get more from the data you already have.
So without further ado, over to you, Caroline.
Caroline:
Thanks, Fiona. So today we'll be using a fictional case study. This is a state funded counselling programme for at risk young people aged 18 to 24 delivered by 16 NGO providers.
And in this example, young people enter the programme through referral at a crisis moment and complete a wellbeing survey at baseline and then again at a six month follow up. On average, you can see on the slide, the results suggest that the scores for wellbeing rose by 20 percentage points for this programme, which looks great. But several things can inflate a result like this.
And this example has four of them. So we'll be concentrating on these data issues throughout the rest of the webinar. And I'll just run through them briefly now.
So the first is a change in measurement. So the wellbeing survey was revised partway through the programme intervention, and a new version of the survey produced higher scores for the same level of wellbeing. So some of that improvement is an artefact of the instrument.
In the second example, we look at attrition. So young people who dropped out are missing from follow up data, and those who stayed tended to be doing better than those that dropped off. We'll also be looking at a third data challenge, which is selective reporting.
So some providers in our case study submitted data and others didn't. And it wasn't just a random sample of the services. In our example, the attrition problem suggests there's some provider level bias going on with submission to.
In our fourth example, we're going to be looking at the challenges around attribution when you haven't got a comparison group. So in our example, young people in crisis are entering a programme at a very low baseline, and they may improve over time, whether or not they received the support. So this is a little bit less about how the data challenges, and more about what we can ask of the data that we have.
Fiona:
OK, so Caroline, once we've accounted for all four, am I right in thinking that about a fifth of the headline improvement in our example can reasonably be linked to the programme?
Caroline:
Yeah, that's exactly right, Fiona. So for the rest of the webinar, we use this case study to take a look at how you can spot things that can inflate results, including how to design your data collection to prevent it, and how to analyse the data you already have when prevention is no longer an option. So this is the getting it right from the start part, and also the what to do if you didn't that comes from our title.
So Brendan, I'm going to hand over to you for the first issue, a change in the survey, what does this mean for your evaluation, and what can you do about it?
Brendan:
Thanks, Caroline. The first issue is a gnarly one. Partway through the programme, a question in the survey changed.
Not a recommended thing to do, and often rather painful. Sometimes it may be necessary because your initial measure was just wrong, or something else has changed in the environment. On the slide is the same young person with the same level of wellbeing, version that puts about one point of the scale, and version that puts them higher up.
The gap is not an improvement of the person, it's an artefact of the measurement. In our case study, this accounts for about three of the 20 points.
Fiona:
So Brendan, if this happens, what should you do?
Brendan:
So in this example, we collected both versions of the question for a period of time, before switching to just using the new version. This gives us the ability to see how the scores differ, see how the categories could be mapped between two versions, or use a regression analysis to map values between the old and the new. If you didn't run an overlap, you can still do some of this, but with less certainty.
There may be published research giving clues about how the two versions map, or if they're both categorical, there might be a common sense way to map the categories. Can't do either, you'll have to report the results independently, and make a more evaluative judgment about the change.
Caroline:
Thanks, Brendan. So the second data issue we're going to cover today is missingness, and what this means for your evaluation and what you can do about it. So there are two types of missingness.
There are people skipping questions, so item level within a survey, and missing from later rounds of survey follow-up. So this is person level missingness or attrition. At the item level, fixes could be covered in a webinar on its own.
So today we're just going to focus on attrition, and this is what drives some of that percentage point gain in our case study. So person or unit level data is both easier to understand and a little bit harder to fix. So on the slide, the top row is everyone who enrolled in the program at intake, and the bottom row is everyone who we still have enrolled in the program, and have follow-up data on.
The greyed out people are the people who left early, so they don't count towards the average. Now, when dropout isn't random, the result can be biased, and in our case study example, the highest need, most vulnerable young people also were the most likely to drop out of the program. So the results describe only those who stayed, not everyone the program set out to help.
And in our example, this accounts for about six of the 20 percentage points in the wellbeing measure.
Fiona:
Okay, this is something that we need to be mindful of, right? So what can we do about it? We can't compel people to stay in a program, obviously.
Caroline:
That's right, we can't. But we can collect good data, and we can get better at understanding who's leaving and why. And that information can be used to reduce dropouts from an implementation level.
You can consider how you can reduce people from disengaging. But it can also be used to assess how much that dropout is likely to affect the measurement of your impact.
Brendan:
So to do this requires collecting good information about who your clients are, starting at the beginning. So perhaps define the population, who is your measuring. Decide who counts as a participant from day one, so you know exactly who should show up at follow up and who is missing.
Get consent to follow up and collect several ways to reach each person, such as phone, email, and a trusted contact. People change numbers and moves, so the more ways of contacting them, the better. Also, plan repeated follow up attempts and set a sort of a response rate target if that's appropriate.
Caroline:
Yeah, or you could seek to collect data at more frequent intervals. So this could be as simple as seeking high level feedback on a regular basis or providing some kind of online feedback tool. And when you can't reach people, it's important to record the date and the reason for the exit as much as possible.
This can tell you whether people left because they were doing really well or because they were struggling.
Fiona:
And what about the data you already have? What can you do then?
Caroline:
With the data you already have, the aim is to understand how much attrition is shaping your result, essentially. So you can do this by comparing completers to non-completers on key characteristics such as age and gender and ethnicity. And if the two groups look different, the dropout probably isn't random.
And if there's significant bias, you can weight the people who remain so they resemble the group who started.
Brendan:
Yeah, so weighing your data is a more advanced technique used when there are large biases in your data. Basically, you weight the responding individuals so they resemble the original or full population. So in our example, say we have 500 high need clients and 500 low need clients at the beginning, but only 300 clients remained at the programme at the end, while all of the low need clients stayed.
So what we need to do is weight the remaining low need clients up so they represent those who have left, and reduce the influence of the low need clients down while still totalling 700 as this is your final sample size or denominator. In this case, a high need client would have a weight of 1.75 peoples and a low need client a weight of 0.7 peoples. So the denominator stays the same, but you've adjusted for this particular bias.
This does require a bit more work, both calculate the weights and a software package that handles this, that can handle this would be quite useful.
Fiona:
So Brendan, to check for bias like this, you need to have collected good information about who the clients are, right?
Brendan:
Yes, yes. You'll need to collect the variables to group your data at the beginning and use standardised response options wherever you can. Visual suspects, age, gender, ethnicity, disability status and some form of location data are also important.
But if you don't want to ask loads of extra questions, some variables can be derived. An address can be linked to government datasets covering like socioeconomic status, disadvantage, rural location and local demographics without asking us. If you do collect addresses, plan how they'll be stored and who can access them and how they'll be used.
Good governance data. Another advantage is to understand your data better. A single number can hide what is happening in subgroups or over time.
For example, a small positive effect for a large group can stomp a large negative effect for a small one. So the tyranny of the majority.
Fiona:
Yeah, for me, what's important there is that if you don't break your findings down, you could have a situation where the programme looks like it's delivering awesome results, but that finding could be misleading for decision makers using the findings because while outcomes are positive for one group, the same programme could be harmful for another group, making things much worse for them. So this could lead to a situation where effectively, decision makers are making poor decisions, totally unaware of the poor outcomes for a group. But I've also come across situations where organisations are trying to break down data, but they don't have a large enough sample.
For example, in one instance, the sample size was less than 10 people for a subgroup. They were using percentages. So when they looked at that subgroup, each person counted for 10%.
Also, I'd hardly say that at 10 people, the sample was representative. What would you say, Brendan?
Brendan:
This is clearly a data crime. To say that such a small number of people are representative is misleading and potentially support a decision based on that insufficient information. The only thing you can do is to either report the actual numbers, not percentages, or collapse the small categories into a broader superordinate category where it makes sense.
Caroline will take us through the last two issues.
Caroline:
Thanks, Brendan. So the third issue sits one level up from the client attrition, and it's at the service level. And here, we're going to be looking at the services that submitted the data.
So of the 16 NGOs that are enrolled in the program, we found in our case study, only 10 gave us data. So on the graphic, each square is a provider. The green ones are the ones who submitted, and the outline ones are the ones who didn't.
So in our case study, who submitted isn't random. In our example, we profiled which providers submitted data. Also happened to be larger ones based in metropolitan areas, and the smaller regional services didn't provide data.
So that means our results are really skewed towards a particular type of service provider. This also means that the types of clients who access those services sit behind that problem, which makes this gap in data submission a confounder. So in our example, this accounts for three of the 20 percentage points.
Fiona:
Yeah, from my experience, this sort of issue isn't uncommon, right? So for example, in one instance, out of around 20 providers, only three or four were regularly providing data or data from the others, really limited and often spasmodic. There was also variance in size and types of program of the service and the types of program that they offered.
So what they were really reporting on was the performance of three or four providers. Have you got some suggestions of what organizations can do who are struggling with this sort of issue, Caroline?
Caroline:
Yeah, thanks, Fiona. It is really common, that's true. So there are a few things you can do depending on who you are in this dynamic.
So for instance, if you're a contract manager or an operational manager, managing the data intake, you may want to think about whether making data submission is a funding condition with clear reporting dates. You may also want to consider whether it's important to resource data capacity, particularly for smaller NGOs. It's also important to monitor submission while the program is running so that gaps like these are caught early and follow-up is done if you need to prompt some people to submit data.
Another angle is that it's really important to make sure that your providers and services understand why good data collection is important and how it ultimately will be used. And then finally, if you're not getting the data in, it's important to have a reason behind why the provider isn't submitting that data rather than leaving it blank.
Fiona:
So Caroline, what's the value in trying to find out the reason a provider or service didn't submit data for data already collected? What's the benefits?
Caroline:
Yeah, so recording that reason turns that blank into a piece of evidence and without it, a missing provider is really just a gap and you can only guess why you're not getting that data. But with it, you can tell whether that gap is random or perhaps tied to something that will also affect your results.
Brendan:
Yeah, such a good point. The reasons point to different fixes and different levels of risk. For example, we had no capacity suggest small or stretched providers.
So you can take a support and recognize that results might underrepresent that type of provider.
Fiona:
That makes sense. From my example, one of the key learnings from identifying which services did and did not collect data regularly was capacity issues. With smaller providers far less able, it seemed to collect data.
Also the providers who did collect good data had a good understanding of the importance of data and how to use it.
Brendan:
Yeah, that can be a thing, especially in very small organizations or services. The other thing can be our clients weren't ready to complete surveys, which is a bit of a warning sign and may indicate that the missing provider serve high needs clients, which means the result is probably flattering as in our example.
Caroline:
Exactly. And if the program is already run, there are two things you can do with the data you have. So each one answers a different question about how far you can trust that result.
So one thing that's important to do is to profile your provider. So that's mapping coverage by provider and comparing those who submitted to those who didn't based on size and location and client mix. And this shows whether the missing providers are different from the ones who did submit data.
The other option is also that you should consider limiting the claims you can make to those particular providers you've actually heard from. And so this keeps your conclusions honest about who the evidence describes.
Brendan:
One caveat I would add to that is that reporting only those who have responded can have the unforeseen consequence of making invisible those who do not respond. Sadly, it's often those who do not respond who most need their voice heard. So explicitly naming who is not represented and why is really important.
Fiona:
So Brendan, the other thing I think quite a lot about is attribution.
Brendan:
So this is the last issue. Even with complete consistent data, before enough to change enough, does not tell us what would happen without the program. On the graph of the solid line is a program group after adjusting for the first three issues, gained about eight points.
The dashed line is what we think would have happened anyway, about four points. The gap is our locally program effect. Of course all the numbers are illustrative.
Fiona:
Sure, this one I often liken to the rabbit foot effect, where coincidentally a person happens to have a rabbit foot in their pocket, as you do, and something positive happens. They attribute causation to the rabbit's foot in their pocket, assuming forever after that having a rabbit's foot in their pocket means things will work out well. But it's got nothing to do with reality.
It was just coincidental. So going back to our example, I think it's really important to consider what are some of the alternative factors or reasons that could contribute to positive outcomes? That could mean in this instance, some young people might get better regardless of the program.
Caroline:
Absolutely. There are several reasons why a young person might get better. So they're entering the program at a moment of acute crisis.
They may have other supports outside the program. There may be maturation effects so that they naturally feel a little bit better at six months post versus that crisis moment. And without a comparison group, all of that gets counted as a program effect.
So the main fix really is to build in one from the start.
Fiona:
Yeah, sure. But I'm wondering, how can you do that ethically?
Brendan:
Well, it needs care. Possible options include using an existing wait list, staggered rollout or matched services that haven't started. You've got to take more than one measure before the program begins.
So the existing trend is visible and perhaps agree up front what size of change counts as meaningful, preferably using sort of published benchmarks. If the program is already running, it's harder. You can sometimes build a comparison from administrative data you have or use existing research on how similarly young people cover over time.
But be honest, without a true comparison group, you can only describe the program as likely contribution, not its effect. Evaluation is a whole methodology developed around this call, not unsurprisingly contribution analysis.
Fiona:
So Caroline, I get a sense that there's a bit of a thread running through all four issues.
Caroline:
There is. I think the important message today is data is a record of the process that made it. And this is a little bit about data culture.
So key to any data collection is keeping the users of that data and those who have collected it involved early to make sure that it will be useful and then collected reliably and accurately over the life of the program.
Fiona:
But I'm curious, where do you start? I'm thinking some people will think the first job with any data set is clearing it. Is that what good data looks like, Brendan?
Brendan:
When you inherit a data set, yes. But today was mostly about getting things right from the start, designing the data collection so the data arrived at the doorstep fit for purpose. The slide shows this happening in three stages, design it right, keep it lean and put it to use.
But reiterating what Caroline said, you would also ask yourself questions like how is it being collected by whom and how does this data get to you? So from the beginning, make sure categories are mutually exclusive and include all the categories you'll need to understand and report on in your data. Use standard classifications, take it from authoritative sources.
Use unique identifiers to remove the risk of duplicates or having multiple names for the one thing. Place a mistake if they have overlapping categories, for example, having 1-5 and 5-10 as two of your categories. So where does the 5 really fit?
Caroline:
And to that end, it's really important to review your data regularly and sense check it against what you would expect and provide reports to users and those collecting the data if you can so that they can see where that data is going and how it's used. And this is all part of building a really strong data culture. Everyone involved in data collection, analysis and reporting needs to understand why good data is important.
So if the person collecting the data knows it could impact funding or staffing or better service delivery, they'll take more care. Likewise, those analysing and reporting data need to know how it was collected to understand why some data may be missing or show unexpected patterns that need to be highlighted and acted upon.
Brendan:
Before we move on, just a quick tip. Report valid but extreme values separately because even though they're real, they can skew an average. For example, client follow-up time, all you need is a few open cases or ones that took a really long time to resolve due to non-programme related factors and the averages blow up.
So reporting these separately ensures you don't lose these important insights about those edge cases.
Fiona:
That sounds pretty important Brent. We don't want those average follow-ups. So once you've got good data, can you repurpose it for other purposes without creating a whole heap of bias or other issues?
Brendan:
You can use data collected for one purpose for something else, but there are a few things to consider. A key one is the primary purpose of that data. For example, contract and data may hold some important information such as how many people have attended a service, for how long and an indication of what difference it made.
However, the outcome measures may be limited to what is required by the contract and may not collect other important information like subgroup indicators or broader social outcomes. Also, contracting data is close linked to continued funding can incentivise providers to showcase standout success stories. While that data may be relevant for contract monitoring, it may increase the risk of positive reporting bias.
Fiona:
I particularly see this when doing outcomes evaluations. Quite often organisations want to use their contract and monitoring data or some other administrative data set for assessing what difference the service or programme has made. But when I look at the data, the usefulness of that data for that purpose is pretty limited.
So what they've got instead is a whole lot of what I'd call output data like how many people applied or were referred or how many attended or how many completed the programme. It doesn't tell me what difference it made, so it doesn't tell me about outcomes. The other thing that I come across is data that is clearly biased, typically or quite often with a positive bias.
So key signs of this sort of thing are where all the reports are really glowing, say contract monitoring data or survey with a very small sample. When I see that, I smell a rat compared to the number of people who participate in a programme or service. Or they may try to use complaints data which will be by definition have a negative bias.
Any thoughts on what you can do Brendan?
Brendan:
It could be that rabbit's foot sort of going off in your pocket.
Fiona:
That could be rabbits and rats.
Brendan:
Yeah, but tricky. That is a really available source of information that will bring all kinds of biases, a bit like having extreme values in the data. These need to be reported separately, explaining the context and referencing the number of responses in these data relative to the overall volume.
This kind of relates to the tyranny of the majority effect mentioned earlier. You have to ensure these smaller group insights are heard and reported on, so that information can be acted on without compromising broader trends and insights. Another thing you can do is power up your data by borrowing statistical power from these other data sources using official government data and combining other data collections as sort of a great start.
Caroline:
Yeah, another powerful source of data are qualitative insights from interviews and clients and stakeholders and open text feedback from your surveys. So this can help you to capture information that you may not have designed for within your data capture within your survey. And it can also help provide some more of that explanatory contextual information to pair with your quantitative data.
Brendan:
Yeah, obviously you need to have a clear way of linking these different data sources at an individual group level, which reminds us of the need to always have those clear unambiguous categories and ideas in your data set.
Fiona:
Okay, thanks for all of that. So here's four things to take away. First, data is a record of the process that generated it.
So before you read a result, think about the process of data collection, such as who's been included? Who hasn't? Who provided the data?
Have there been any changes to data collection tools? Second, good data is designed in, kept clean and put to use. Make sure categories are mutually exclusive and exhaustive.
Use standard classifications, use unique identifiers and fix inconsistent categories. Third, data culture is what holds it all together. Make sure those collecting or entering the data know why it's important that they have a copy of the category definitions and where possible, limit responses to valid values.
And four, you can repurpose good data. But there's a few things you need to consider, such as keeping in mind its original purpose, what its limitations are, and watch for bias creeping in. Right, well that's it from us now.
We'd love to hear from you, so drop your questions in the chat and Brendan and Karen will take us through a few questions that we've already, were submitted before this webinar. So we've got these to kick us off. So first of all, the first question is, designing data collection.
How do you design data collection around what you ultimately want to analyse while balancing real world complexity, effort and cost? For example, we struggle with multi-set categories where an activity spans several areas, which makes later analysis difficult, and with outcomes that are too expensive to measure directly. I'd be interested in approaches to primary, secondary categorisation proxy measures and deciding when additional data complexity is actually worth it.
Brendan, that sounds like a pretty tricky one.
Brendan:
Well, it's a big question. Hopefully the webinar helped to answer some of that. But there is no easy answer, sorry.
You could use a theory driven approach to collapsing the categories. Or perhaps a data-driven approach, so latent cluster analysis is an example of using data to combine groups together. Caroline, have you got any thoughts on this?
Caroline:
I mean, this is such a fabulous question. I think the first answer is to be theory-driven. So what is your theory of change and what data do you need to demonstrate that that theory of change is operating?
So that kind of defines what's in scope for your data capture. And then in terms of delivering multiple programs across multiple areas, it's a question, I think, of what you can harmonise. So what are the key outcomes and indicators that are globally important across all of those programs and make those priorities for each of your programs to be capturing?
But by the same token, you also want to be sensitive to the nuances of those different contexts as well. So I think that's where that secondary data, that administrative data or government or geographic data can do a lot of that work to help perhaps explain some of those country level or regional level differences. And I suspect I haven't answered all parts of that question because it's such a complex question, but I perhaps will pause there.
Fiona:
I've got a question for you two about it. I was wondering while I was listening to you whether one thing they can do is build up data sets over time. So you start off a bit simpler with fewer ones and then as it becomes clearer and as you mature and have got more capability, you could actually add to it.
Would that help at all or not really?
Brendan:
Yes, of course, then you worry that you may have missed really key stuff at the beginning. So sometimes if you go too big, there's too much detail and there's burden on people to collect and the cost that's coming in. If you start too small, then you're missing good point of information at the beginning.
So again, there's no easy answer to this one, but the theory of change is a good place to start in answering your problem correctly. But are we doing another webinar that might help later on?
Fiona:
I think we might be and I think it might be on me, Brandon. Yeah, I think we might be going to do one, so the next one is assessing quality. Yeah, watch out for that.
It'll be in, I think, November assessing quality of existing data. How can a small organisation assess whether it's existing data are good enough for evaluation and identify the minimum practical improvements needed before commissioning one?
Brendan:
Yeah, I'd reckon asking this, but sometimes it's just quicker and more efficient outside of somebody who does this all the time, so perhaps just beginning with a conversation. Fiona, have you experienced this as well?
Fiona:
Yeah, I was just thinking actually today I was talking to someone and it made me think that sometimes one of the benefits of an evaluation is actually that you can actually provide recommendations on data collection and what sort of data would be really helpful. And you can actually kind of work with somebody to actually get them to understand what they have and haven't got. So you can kind of bring that real clarity around what they do and don't have.
So I have seen also findings from evaluation used as kind of a prototype for future data collection to inform it. So it is possible and in evaluation we can use lots of different data sources to work around some of those gaps while highlighting them at the same time. What about you, Caroline?
What are your thoughts?
Caroline:
Yeah, so the question is how small organisations can assess whether they've got good enough data to source, to base an evaluation upon. I think it's worth considering what questions you want to answer and whether that data tells that story. So I would be question driven.
And then also, just remember, there are different levels of analysis that are possible depending on your constraints. So at the top of the pyramid, there's the impact evaluation, which requires some serious data. But at the other end, it might just be about, you know, deciding whether you have the data to tell the story about your programme and to decide kind of where you're at in terms of your programme maturity and the data that you have and pitch it around there.
Fiona:
But even following that up, Caroline, I have done outcomes evaluations where there's no outcomes data and actually that thing we talked about before, contribution analysis, I've got a feeling that could be another webinar, can be used to actually help fill some of those gaps. So there are sometimes ways of filling some of them. It's not ideal, but it can help sometimes.
So, yeah, but it's not easy, really challenging when you're small. So the next one is about sustainable strategies. What sustainable strategies can be used to bridge the gap in granular, disaggregated baseline data across small island developing states with constrained national statistical capacities?
I think this could be useful for some small organisations with constrained capacity too, right? And I suspect this is another tricky question.
Caroline:
Yes, and somewhat similar to, I think, the first one we discussed. So I think at an organisational level, it's important to agree about, you know, your harmonised data structure. So you have common outcomes that you're seeking to measure.
And you may have country level outcomes nested within that. And then it's important also to be really clear on your data governance principles and shared analytic and reporting roles as well. Yeah, Brendan, do you have any thoughts on that one?
Brendan:
I think the answer pretty clearly is this is another huge question. Different ways of trying to approach that. But I think it's largely that they've just got to talk and decide on those shared values and data structure, getting the data already got sort of harmonised so they're measuring the same sorts of things.
But yeah, a tough one.
Fiona:
And here's another one. Changes to data. This one is about organising and using time series data, especially when things have been added or removed.
What's the best approach to addressing this?
Brendan:
Oh, hell, these are tough questions. So regarding organising time series data, apart from making sure you can link individuals and restructuring your data sort of a long and wide format, depending on how you're going to analyse it, I would need to know more. But regarding data measures at different times of data collection, I'll probably do this in the analysis.
There are statistical techniques that estimate missing values for the missing data since drop. So latent growth mixture models, for example. As one form, there's also latent transition analyses, a bunch of different techniques depending on your needs.
Fiona:
Well, I'm afraid we've got some more tricky questions coming. So thank you everyone for popping questions in the chat and please feel free to pop in any more if you've got them. So the first question we've got is if the lack of data capacities is a known and common issue, how could data capacities be mobilised?
Caroline:
Do you want that one, Brendan, or shall I have a crack?
Brendan:
You have a crack because I was just thinking.
Caroline:
OK, so we talked a little bit today about building a data culture and that touches on the different people that are involved in developing your evidence base. So it's important, I think, to have slim data collection that is linked to your research questions or your theory of change. So every item that you're collecting data on should really earn its keep because it is burdensome.
Secondly, it's important to make sure that that data is put to work. So where you are collecting it, it's important to report it back, to also use it within your program and decision making so that it's driving your ongoing program delivery. Those kind of signals to people collecting data on the ground are really important to hear because they're part of what builds the data culture and what mobilises people to participate in your evidence building project.
Brendan, do you have anything to add?
Brendan:
Not really. I mean, the resource thing, building the capacity is always a problem, especially in this constrained financial time. So you have to be more selective and build a data culture that hopefully supports those who end up at the analysis end of things, a team effort.
Fiona:
Hmm. So this is a question from Amanda. She says, an issue we have is data gathering fatigue.
Excuse me. Everyone wants to develop evidence based solutions, which means tapping the same people with surveys, etc. We also notice downturn in survey returns or participation.
So how can we stop survey fatigue, engagement fatigue?
Brendan:
We can't. Everybody's suffering. I mean, 20 years ago when I started the longitudinal survey, we're getting, say, 30% response rates and now we're getting down to the 10%.
So everybody's experiencing it. So the answer is things like much smaller surveys more frequently. It's quite a common technique.
And as you say, there's also a problem of the fatigue at the community level where it's the same groups are getting the same different surveys by different people coming in there and ask these questions. So again, it's perhaps it's engagement at the local level. They need to understand why this is important.
So you might be talking to those at the community level to help you disseminate and help you support the distribution of those surveys. And they're going to get something in return, which is what information is going to be contained in your survey that can help them deliver those services. So it's sort of a you've got to drop down a few levels and involve it at that point.
But it's just a tough one that's really common nowadays.
Caroline:
The other thing I would add to that is that programs are generating a lot of a lot of data in the course of implementing their programs anyway. And so where you're collecting primary data, I would seek to find what data you might already be generating just through the course of delivering your program so that you don't have to collect it from people via a survey and just be really economical about what you do have to collect.
Fiona:
Yeah, I think that's really important, Caroline. I've seen it where we've come to do an evaluation and some of the data collection techniques would normally use whether it be a survey or some other form. When we talk to the organisation, they'll be like, but they were engaged with really recently.
And so we were actually able to use that as a secondary form of data. So we had to use it rather than actually engage in that group all over again. What we agreed was actually that there was very little probably value add in doing that.
So I think it's also getting people to stop and think, do we really need to go and ask them again? And I think back to your point, Brian, I think one thing that's sometimes missing that drives fatigue is that people don't see anything change. They don't see what benefits of engagement really is.
So the more we can shine a light on that might actually help people feel slightly less fatigued, maybe.
Brendan:
One more thing I had, something we've done in the past was multiple ways of them answering the survey. So there was an online option sent through email. There's sort of a social media link.
There was paper-based versions that providers and communities sent out to their clients. And very expensive because there's so many ways of doing it and all had to be entered. But it kind of got the sort of spread and the buy-in more than just sending out an email or hoping people would click a link.
So there's multiple distribution methods that suit the people that you're trying to gather insights from.
Fiona:
OK, and this last one is a bit of a help. It's a good one. Can you all share one tip that has helped you most with getting your data right?
I almost don't know where to start. And I think that could be so true. Where do you start?
Brendan:
Well, we go right back to asking the right questions first and be economical at that point. So I think you've got to be really clear about what it is you're trying to understand, which is kind of that problem definition problem. And then what's the minimum amount of data that we get to answer that, because burden, going back to the previous one, is a problem.
You can't send out 20-page surveys because people don't answer them. So how do you get people to buy in? So for me, you've got to go right back to the beginning and start there.
And then once you've got that, then you can sort of build, OK, questions. What's the question to be asked? OK, and then how are you going to get it out to people?
How are you going to distribute it and then where do you go? Caroline?
Caroline:
So when I analyze any data set, I usually will just eyeball it. So I'm running frequency tables on all the items that are in the in the data set. I eyeball them and I move very, very quickly to visualizing that data.
And there are all sorts of programs now you can use. I use a website called Data Wrapper for non-sensitive data, which is a free website. And it's an online visualization tool.
And it's very, very quick and very, very intuitive. And I often find once you've plugged that summary table into a visualization, it'll immediately kind of start to tell you a bit of a story. And I think that can get you quickly thinking about like what type of analysis you're going to progress with.
So that would be my tip.
Fiona:
My tip is actually often I start with people with back with that thing that Caroline's mentioned, which was a theory of change or getting them to identify what exactly are they trying to achieve and what would good look like? So what difference is it going to make? And then I get people to work back from that.
What data do they already have that could be used for that purpose? Are there some good proxy measures? And then just getting people to think about what are the different ways that you could collect that data.
So just encouraging people to, it comes back to what Brandon was saying, you don't want to go too small because otherwise you could find you don't have the data you need. But just trying to get kind of that sweet spot of just enough data for the data reports, even imagining how are you going to use it? Imagine the reports that you're going to use for it.
How's it going to help inform your, use it as a learning tool for yourselves? And then actually when you put that, so I would say to people, it's better to have a smaller data set, but do it well. So you've got really good quality data.
Then actually try to do too much. And what you've got is kind of higgledy-piggledy mess, you know, which is quite poor quality data. And so that's my thoughts around it.
It's just trying to right size it, which sounds easier than it is. But I find being clear about what you're trying to achieve, what are the key things you need to know about, are great hooks for it, personally.
Brendan:
Yeah, actually on that, I love thinking about the report you're going to write and work backwards. So in some ways, if you've got the data, you sort of, oh my God, what do we do with all this rows and columns of data? So how are you going to write the report?
What sorts of things are you going to be needing to inform that? And then you can sort of work back, sort of meet in the middle between the frequencies and understand what the data tells you. And then what the story you need to tell in the report.
Yeah, forwards and backwards, in the middle and in the middle. You can do everything all at once.
Fiona:
I hope that's helpful. Okay, well, there's no more questions for us. So I just think now's a good time to finish up.
So I want to thank you so much for staying on and for all your questions. We love being able to have these discussions and we hope it has been valuable. And remember, if any of today's webinar has given you ideas or thoughts for your own work, we're always happy to continue the conversation.
So please don't hesitate to reach out. And thanks again for joining us. We really appreciate it.
We hope to see you in the next webinar and have a great rest of your day, everyone.
Brendan:
Thank you. Mā te wā.
Fiona:
Thanks, everyone.
Brendan:
Bye.
