You might be getting ready to collect data for your study, or perhaps you just like to think ahead! In either case, you’re starting to grapple with the fact that you’ll be looking to share your survey with a great many people. Of course, you’ll need to do this in a way that aligns with what you wrote in your methodology, in terms of both population and sample, keeping in mind all of your inclusion criteria.

At the same time, though, it will still be a great many people. Indeed, because of whom you’ll need to access, you might actually that they are geographically distributed to some degree. As an example, you might be studying teacher self-efficacy related to implementation of a three-tiered RTI model. You work in a school, but your power analysis revealed the need for sample comprised of at least 123 participants. You know some teachers, sure, but not that many. You might need to reach outside the boundaries of your school to your whole district or even your region.

It might actually be that none of your participants are near you—the exploration of work-life balance for offshore oil rig workers that you’re taking on seems to require that you build a bigger boat. Heck, all your participants might be next door, but you quake with fear at the prospect of hand-collecting data and then entering it into spreadsheets to then get those spreadsheets into SPSS or another package for the sake of your statistical analysis.

In every single one of these cases, it’s useful to have technology on your side to take the vast majority of the work out of the process. Thankfully, there are solutions—programs exist that will host your survey or surveys, allowing even geographically-distant or -distributed potential members of your sample to access your questions.

Even more magically, there are services that will recruit participants for you, just in case you don’t know any recently-retired professional and semi-professional Serbian tennis players to ask about their nationalistic pride following the recent resurgence of Novak Djokovic. (Note: This might test the limits of your chosen participant recruitment platform.)

While there are a bunch of them, with more popping up each day, we’ll focus our time and attention for this next bit on what I might call the two main ones: SurveyMonkey and Qualtrics. They have a rather extraordinary amount in common, particularly when it comes to their offerings for doctoral candidates such as yourself. We’ll talk about those many similarities in these solutions, and we’ll also delve very deeply into the differences.

Here we go!

Survey Development and Hosting (“The Fun Stuff”)

Overview

The main service provided by both of these sites is a pretty simple one: They let you take a survey you have in pdf form or on paper, and you can stick it online and share it with a bunch of people. That’s pretty simple, right? Kinda! You’ve likely been able to review surveys as you developed your study and sought out ways to measure those variables that you felt merited more attention. (In another blog post, we talk about the importance of having pre-validated survey instruments for your dissertation—they help to avoid what can be a whole other study’s worth of recruitment, collection, and statistical analysis.)

Anyway, as you reviewed those surveys, it is likely that you found various types of questions: Yes/No questions, multiple choice questions, ranking questions (like multiple choice but with multiple response options), and Likert scale items are only some of the many options.

Both SurveyMonkey and Qualtrics do this just great, I should say. It’s their bread-and-butter. As you dive in, you’ll pretty quickly find the ability to recreate more or less exactly the survey as it appears on whatever document you have, as developed. That’s great—it helps to ensure the validity of your testing, as the survey was certainly validated in the precise form in which it appears. Deviating might not actually be an option for you.

Usability and Features

Here is where they start to differ. I should mention first that both services have both free and paid options, and it’s simply that what you get in the free option and what you have to pay for will differ. Depending on your survey, it might not matter much, but you definitely don’t want to find out too late.

Let’s talk about individual question features first. They both do a bunch of stuff for free: For instance, both services allow you to create more or less any kind of question that you like, including the multiple choice (single- and multiple-response), yes/no, and Likert scale items that I mentioned above. While you likely won’t ever need this, both even allow you to make a slider, with points on a spectrum to represent the different response options.

Highlight! One really important feature that both allow for free is called “forced response”—this means that participants must respond to a question before they’re allowed to move to the next question. This might seem rude—forcing someone to do something like that—but it’s really, really useful for your study. That is, the most common way of dealing any missing values (any skipped questions) is by deleting. the. whole. entry. There goes the whole response! Especially when you are worried about meeting your power analysis-determined minimum sample size, having every response counts. Force a response!

Then there are some features that one or the other doesn’t offer in a free version—for instance, SurveyMonkey doesn’t allow skip logic with their free product, but Qualtrics does. Skip logic is a tool that allows you, depending on a participant’s response to a particular question, have them skip to a later question, past some that necessarily don’t apply to them. For instance, one question in a survey about the experiences of African American university students at Predominantly White Institutions might ask whether the participant has experienced a microaggression while in class. Follow-up questions can vary significantly, but one might be an open-ended item, asking the participant to describe that event. It might also be a question asking the participant to respond about whether that experience changed her perception about that class. If the participant hasn’t had that experience, then the follow-up questions will necessarily not apply.

Another difference is user-friendliness, and Qualtrics seems to have this one, as well. When you work on a question, there is a “permanent” side bar open to the right where you are able to select the question type from a drop-down box that appears in green. Then the type-specific options appear underneath that. For advanced options, such as default answers or the aforementioned skip logic, there is a sprocket off to the left of the question that allows access to those.

SurveyMonkey, by contrast, has a less intuitive interface, at least for me, though again with many of the same options. Here, though, there is no helpful guidance provided by different colors, and options actually appear both on the left and in the question itself. However, the options on the left are for adding new questions and the one in the question itself is for that question. It’s also easy for first-time users to end up with a message that says, “Oops there was an error saving this question.”

This is because SurveyMonkey builds in an additional step where users have to save the question before doing anything else, like moving to another question. That said, you might find this helpful in case you make the wrong move!

Plans and Pricing

So both do the job when it comes to individual question construction. For both of them, though, there are limits to the ability to build whole surveys with just the free option. Let’s talk about that just a little bit more as we get into plans and pricing.

In this case, too, Qualtrics offers a little bit more for free than does SurveyMonkey, and they do so in some important ways:

* Note. You can collect more than 100 responses, but you can view only first 100 without upgrading to a paid plan.

It seems like kind of a mixed bag, though for dissertation work, there are some key differences. First, for the number of surveys, having as many as you’d like at once can be really nice—you might be gathering data in the form of different surveys for teachers and administrators, and having both of these live at the same time can make that process a shorter one. If you’re using a pre-validated instrument, though, you might not be in need of more than one instrument at a time.

Highlight! If you’re measuring more than one variable, you might have more than one instrument. It might feel right to put these in different surveys. However, that can be dangerous, in that it’s often necessary to link participant responses from one instrument to the next. If you have them in different surveys, then it will likely be impossible without tracking IP addresses to match up responses. Even if you can do this, then some complicated combination work will be needed to ensure that you yield a single data set for the purposes of your statistical analysis to follow.

A crucial difference follows: SurveyMonkey allows only up to 10 items per survey. This can be problematic as, for some studies, there are more than 10 items devoted just to demographics! Most instruments, other than the shortest of short-form surveys, will also have more than 10 items. This means that, for almost any application involving SurveyMonkey, you’ll to have to get a paid plan right out of the box.

One similarity here involves getting up to 100 responses, though I should note again that, with SurveyMonkey, you can collect more than 100 but will need to upgrade to be able to see them. This might be quite beneficial, actually, as you can then pull the trigger on a paid plan when you need it.

And you will need it—you can see that both services prevent you from exporting raw data in a form that is SPSS- and other program-friendly. Sure, both of them do some basic things—mostly descriptive statistics, but as I talk about in another blog post here on the site, descriptive statistical analysis is generally not considered sophisticated enough for the purposes of doctoral-level research. This means that, for both, you’ll be getting a paid plan at some point.

This is where SurveyMonkey has a big win: It costs a lot less than does a paid plan for Qualtrics. Both certainly come with many more features than do the free versions. Again, though, the important ones are the ones that I talked about above.

For SurveyMonkey, that means the ability to build a survey with more than 10 questions—again, this is likely necessary for most of you, and right from the beginning. Then another useful feature you gain access to through payment is the skip logic that comes free with unpaid service from Qualtrics. Finally, you get to export your raw data in CSV, XLS, or PDF. All of these are available at $37/month. It’s billed monthly, so you can stop anytime. This part, actually, is a big benefit—once you gather and export all of your data, you can then just not renew your monthly plan the next month.

If you’re in for the long-haul with this plan, you can also do it for $31/month if billed yearly. You’ll notice that I didn’t mention SPSS for exporting, though, and this is two steps up from free (costing only $1 more per month, though it must be billed annually—there is no monthly option).

For Qualtrics, again, you get more with the free plan—the two big drawbacks you suffer without paying are the 100-response limit and the inability to export to CSV, XLS, and SPSS. However, these benefits can be won, at a price. For Q Lite, their lowest-tier paid service, the cost is $299 per year. There is no monthly option, though if you’re looking for opt in for a year, it’s cheaper than SurveyMonkey’s yearly plans.

Recruitment and Collection

You might be thinking to yourself, “This is great! I just need to find people to take my survey.” That is a scary thought, perhaps! Again, it’s so crucial to have a sense of whom you might collect data from when you embark on a study—way back at the very beginning. When we take on that kind of dissertation help for clients, we actually start with a methods call that helps to establish both focus and access to data to prevent any issues from arising down the line.

If you do find yourself in trouble, though, or if you’re just a tech-savvy person who knows and has been planning this all along (sneaky!), then it’s actually possible to use a recruiting service for you. They’ll be a perfect complement to your statistician, in that one collects the data and the other analyzes it!

There are, just like with survey hosting, a number of options here, including Amazon’s Mechanical Turk (or MTurk). However, both Qualtrics and SurveyMonkey do this work, and given their multi-tasking with hosting, these become natural choices for doctoral candidates looking for the whole package.

Both operate on much the sample principle: You provide a set of inclusion criteria, and then the service will recruit participants who meet those criteria. You get to decide how many survey responses you need in each case (at least the minimum sample size requirement for your study) so you can be sure you get enough, and then you enter details like the number of question items in your survey, whether you’ll use screening questions, and other things like that.

Again, both services will take care of the rest, finding eligible participants and having them take your survey. This does, though, come with cost, and it varies based on your needs. If your survey is only five items and if your only inclusion criterion is “is a person,” then you’ll pay relatively little per response (you pay per response). If, however, you’re looking at C-suite executives under the age of 35 in the Pacific Northwest, then that is a bit more specific. You will pay more for those responses, simply as those participants (in the number you need) are harder to find.

In some cases, you’ll butt up against limits for both services, or combinations of characteristics that are just too specific for them to guarantee they’ll have the responses for you.

When you’re ready to get started, it’s helpful to know up front at least about how much you’ll pay. SurveyMonkey has a handy-dandy little calculator, actually, for just this purpose. You get to share information about your survey and then build your sample by describing all of the inclusion criteria. As you do this, there’s a cost estimate, per response, along the right side of the screen. This way, you know more or less right away what the cost might be.

For Qualtrics and their service called Online Sample, things are a little bit trickier, as there is no online calculator. Instead, you can request a quote by entering your contact information along with your desired number of respondents, survey length (in minutes), and your target demographics. Then they’ll respond—usually quite quickly—with a much better sense of the cost there. Provided that the price is right, you’ll have your respondents!

What to Do

I can’t actually tell you that—there are definitely pros and cons to each of the services, particularly when it comes to survey hosting. The budget-friendly choice, simply because of the ability to pay month-to-month, is SurveyMonkey, though the user-friendliness of Qualtrics during survey creation and management also makes it a great option. Again, the yearly cost is also lower than the yearly options from SurveyMonkey,

They’re about neck-and-neck when you put it all together, but either way, you’re bound to get the help you need from either that you choose. Again, the needs that we find most often in providing assistance with the dissertation here are in gaining access to participants. Having a survey hosted online allows you to access a geographically-distributed population for sampling, and it’s just so much easier than the paper-and-pencil alternative.

In addition, there are robust exporting features (though paid) for each that allow you to put the raw data set in a statistical analysis-friendly form so that you can take the final step in conducting your study. Finally, if by accident or by design you’re left without a group of potential participants, both services will find them for you.

It can hurt to not be prepared, and for most parts of the dissertation—including this one—things can go very wrong. Thankfully, you have at least a couple of options to help here.

References

• SurveyMonkey. (2018). Ask more, know more, do more. Retrieved from https://www.surveymonkey.com/mp/take-a-tour/?ut_source=megamenu
• Qualtrics. (2018). Sophisticated research made simple. Retrieved from https://www.qualtrics.com/research-core/