Last week I highlighted a few of the problems with the Nonprofit Social Network Benchmark Reports. Annaliese Hoehling, Publications Director for NTEN, and Jeff Patrick, President of Common Knowledge, were kind enough to answer a few of the questions I had posed about their report. Since I’m still waiting to hear back from Jeff Patrick about a couple things, I’ll share their response to my questions about their methodology today, and the rest of their response later this week
1) What “industry lists” was the survey sent to? Hoehling shared that the survey was sent out to four email lists: Common Knowledge, NTEN, Blackbaud, and Network for Good.
“It’s worth noting that these lists were chosen because they skew in different directions (albeit there is demographic overlap) with Common Knowledge’s list including small, medium and large organizations from a wide variety of verticals and a pretty wide range of technical sophistication and experience with social networking, Blackbaud’s including a really wide range of groups – sizes and sectors, and Network for Good’s list providing especially good reach into smaller organizations who are probably less technically sophisticated although their list is not monolithic along this characteristic either.”
It was great to see the effort they made to try to get a more representative sample. Unfortunately, it’s clear that their efforts were not successful. Hoehling shared that “about 75% of US nonprofits report annual revenue less than $1M” – which is a huge difference from the 40 – 46% of nonprofits with annual budgets less than $1 million that completed their survey . This suggests that nonprofits with large budgets (which are likely more tech-savvy and spend more on social media) were given much more weight in the survey and report. This alone shows that the survey definitely can not be generalized.
2) Is there response rates available?
“We don’t have response rates. We don’t have percentages from each list either (that is, what proportion of responses came from each list) – we created distribution source codes but unfortunately they didn’t get used in all cases, so analysis of this would not be accurate.”
This will be something to watch for 2013. Hopefully they can tweak any issues they experienced with their source code plan so they can provide response rates next year.
3) Who completed the survey? Was it the executive director? The communications director? How did they make sure two or more people from the same organization didn’t complete the survey?
“Unfortunately, we did not ask respondents to indicate their professional role in the survey, which is an oversight and will be added in next year’s report.”
Looks like with this round of the survey there’s no way to know. It’s great that it will be added next year though!
I’m glad they are going to make some changes for next year and hopefully resolve some of these issues. The biggest issue for me is the generalizability. Hopefully they make an attempt to make the survey more representaitve to give us a more accurate picture of how nonprofits are using social media. There are ways to determine whether a survey is representative (and therefore more likely to be generalizable) without having to do a random sample.
“Note that we don’t claim that the survey was conducted rigorously — and therefore we’re careful to refer to “survey respondents” throughout the report, rather than to suggest that the survey results can represent the general practices of the US nonprofit sector.”
Take this survey with a grain of salt. It has some great insights – but they aren’t about the nonprofit sector or nonprofits as a whole. They are about the convenience sample used for the survey.
Enjoy this post? Read about Problem #1, Problem #2, Problem #3, Problem #4, and Part 2 of the official response from NTEN/Common Knowledge.
Photo credit: visual.dichotomy
{ 2 comments… read them below or add one }
As the regular fly in NTEN’s ointment regarding the salary survey, I’ve really been appreciating this commentary. I know for a fact that NTEN is trying to help us all with these surveys, and they are helpful — they provide a lot of value. That said, it’s value that can be diluted if we’re holding up inaccuracies as fact. NTEN’s methodology is always, to my mind, one that suffers from a limited spectrum. You note that the results reported here clearly are influenced by the bandwidth of large NPOs being able to take the time to respond. NTEN’s other surveys, too, suffer from being a reflection of those engaged in the community and resourced to participate, as opposed to the entire sector. There are still a lot of NPOs that have never heard of NTEN, or even Techsoup, and plenty that delete the emails without reading them.
But NTEN does reflect the forward thinking in the sector. It represents the NPOs that “get” technology, the web, social media, etc. The people who are making the right investments are here. So there’s lots of value on reporting what is happening; the report just has to accurately represent the people filling out the surveys.
So I think you’re doing very important work in highlighting the faults in the methodology. But I also think that NTEN is constrained from doing definitive reporting by the inability of the sector to provide definitive, complete statistics. So it’s not a matter of NTEN having to correct every flaw in the methodology – it’s a matter of them correcting what can be corrected and being very, very explicit about both the methodology and the accordant biases of the final reports. They still provide great value, giving CIO’s like me good information to work from and fodder for those executive briefings. But I don’t want the execs to come back at me with charges that my supporting arguments are invalid. the lack of a clear accounting re the strengths and weaknesses of the methodology leaves that possibility open.
I completely agree with your points – I think NTEN’s survey has a lot of very interesting insights and information. The main reason I decided to take this critical look at the report was because I was seeing many blogs, NonprofitTimes, twitter, etc repeating things from the report as if they were facts that were true of all nonprofits – which is not the case. This information cannot be generalized, so I wanted to make sure that was very clear. Particularly since I felt like NTEN was not very explicit in the report about some of these things – hopefully something that will be changed for next year. Thanks for your thoughtful comment!