The last few days I’ve reviewed a few of the problems I’ve discovered with the 2012 Nonprofit Social Network Benchmark Report. Today’s post will wrap up this series and be the last of the problems I feature. Without further adieu, Problem #4:
Seemingly elementary data analysis
The people behind the survey have (more on their response to my inquiry coming Monday!) acknowledged that the Nonprofit Social Network Benchmark Surveys are NOT rigorous. Wondering what rigorous means? ETR.org has a great, concise definition of rigorous research:
Research is considered rigorous if the methods of the research study are designed and implemented so that very strong evidence is produced to substantiate the study’s conclusions.
That’s pretty straightforward right? That is my biggest disappointment with the survey. With such well-known organizations sponsoring the survey, it should have been a no brainer to make sure the survey findings could actually be generalized – or at least have the research conducted rigorously. NTEN staff did point out that they “don’t claim that the survey was conducted rigorously” and are “careful to refer to ‘survey respondents’ throughout the report” because they know that their findings are not generalizable. So it’s true, they did cover themselves – but they shouldn’t need to – they should do the survey right, in a way nonprofits can truly use as something that is generalizable.
Back to the data analysis – or lack thereof. Unfortunately, there is nothing in this report about how they conducted data analysis. I asked NTEN, but they didn’t know and referred me to Common Knowledge (who actually did the data analysis/report writing). I have not yet heard back from them about the type of analysis that was conducted. Reviewing the report suggests a very elementary level of data analysis was done. The main form of analysis that was done was that averages (means) were calculated. It would have been nice to see a bit more here (maybe a chi-square test like the below example did)- including information about what exactly they did.
I do want to note that I’m not holding this survey to extraordinarily high expectations. I’m expecting a basic level of information be provided for a nationally promoted and publicized survey on nonprofits and social media. To see an excellent example of a survey done in the nonprofit sector that is similar to this survey – but is very straight-forward about the generalizability of their survey and using a convenience sample and has a detailed methods section, see the 2010 Nonprofit Fundraising Survey by The Nonprofit Research Collaborative. Page 35 has their methodology information – and information about data analysis completed.
Photo Credit: Auntie P