What’s a Better Way to Research a Market: Surveys or Experiments?

Photo by Seattle Municipal Archives

I received the following question from a reader a few weeks back:

I’m considering creating a mobile app and I want to know quick/effective ways to validate some of my assumptions. Is it more effective to put out small experiments that test your assumptions, or are surveys of the possible users a better approach?

My answer: it depends on what you’re trying to test.

In general a survey is going to give you decent results for little effort. A survey takes 20 minutes to put together (SurveyGizmo or SurveyMonkey), email it, and you get some info. It’s a simple approach that doesn’t take a lot of time.

However, it’s nowhere near as accurate as putting out experiments. Experiments that ask someone to buy something, sign up for a list, or perform some kind of behavior are the only real way to know if something works. But these kinds of things take so much longer to put together that you have to balance this level of effort with the value they provide.

If you had unlimited time I would always recommend experiments. But surveys save you time, so you’ll inevitably have to rely on them or you’ll never start building your app.

Two things you can’t get from surveys: actual conversion rates and pricing info. You can ask “would you sign up for this list” or “would you pay $x for this” but these are useless questions.

If you need this kind of knowledge, do an experiment. For most other information, surveys can be a solid, time saving approach.

I’m interested to hear your thoughts in the comments.

Start Small, Get Big
Growth Secrets for Self-Funded Startups. It'll Change Your Life.
What you get for signing up:
  • A 170-page ebook collecting my best startup articles from the past 5 years
  • Previously unpublished startup-related screencasts
  • Exclusive revenue-growing techniques I don't publish on this blog
"The ideas and information Rob provides should be required reading for anyone that wants to create a successful business on the web." ~ Jeff Lewis
Startups for the Rest of Us...
If you're trying to grow your startup you've come to the right place. I'm a serial web entrepreneur here to share what I've learned in my 11 years as a self-funded startup founder. Luckily several thousand people have decided to stick around and join the conversation.

For more on why you should read this blog, go here.


#1 Rick on 11.29.11 at 2:48 pm

Agree completely with the uselessness of the “would you pay x for…” question. I had sent out a survey asking our users about a new feature to our product. One question was, “Would you pay an additional $40 per month for this feature?” We got lots of Yes’s! So we built it.

Then we went back to those clients and said, “Ready to sign up?” And many responded, “OHHHH, yes, we’d pay more for that, but I don’t have the AUTHORITY to pay more!” 🙁

Nick Panagopoulos Reply:

Agree with @Rick.
You can ask as many questions and get many different answers, but the human mind is complex. People think they know what solution they need to solve a problem, when in fact, the only time they truly know it will work is when they experience it.

Here’s an example from the early part of my career: A customer requested that they would like automation to help perform some tasks. They said sure, and we went back and forth on some rules for automation. Then the feature was delivered, they never used it because, in the end, they really didn’t want the automation; the task was too important to automate and they wanted full control.
In my infancy collecting business needs, I never saw the customer in action…until I visited them when we deployed the product. I immediately saw that all they needed would have been satisfied with a simple UI enhancement.

#2 Dave Collins on 11.29.11 at 2:51 pm

I’m a big fan of experiments.

People’s perception of how they think they might behave is often quite different from reality.

And experiments are more fun too!

#3 Apurva Mehta on 11.29.11 at 3:46 pm

On my web app I did both, surveys and experiments. My experiment comprised of putting up a landing page and buying google adwords to drive traffic to it.

The landing page was designed to get users to signup for $8/mo.

I got a 1% conversion, which I thought was OK, given that there were no screenshots, no testimonials, and basically nothing on the site except a basic value proposition.

So I was encouraged that there was _something_ to the idea and it was worth pursuing.

But beyond that, I did not learn anything much. Honestly, for a bootstrapper just starting out, it is really hard to get enough traffic to run real experiments where you can test pricing, value props etc.

Adwords are really expensive. So are facebook ads. And if you are starting from scratch, there are no short term organic ways to get a decent amount of traffic to your site.

So I found the surveys more useful because I learnt about the habits of my target customers and also got a permission base which I could use to schedule followup interviews and questions.

Rob, I would love to hear your thoughts on running these types of experiments as a cash strapped bootstrapper? How can one do it cost-effectively?


Rob Reply:

http://onstartups.com/tabid/3339/bid/43774/The-5-Minute-Guide-To-Cheap-Startup-Advertising.aspx is an article I wrote about cheap ads – Facebook is a great source for these (10-15 cent clicks are totally possible).

Using your Twitter followers or blog are also good sources if the audience is in your target market. And exploring your network – whom do you know who has a list of the people you want to communicate with? Call in a favor.

I’ve also heard of folks using Amazon Mechanical Turk to send traffic.

It’s all about being creative and scrappy. The good entrepreneurs I know always find a way…

Apurva Mehta Reply:

Thanks for the reply, Rob. Personally, I scrapped it out by using Adwords and facebook coupons. That was sufficient to help me test the basic value prop with a fake landing page. But it only works the first time, and I did not really learn much about who the target audience is and how to talk to them.

Your tips in that post definitely have sparked a few ideas to make things cost effective, especially now that I need to run longer experiments to tune my landing page.


John McFarlane Reply:

“That was sufficient to help me test the basic value prop with a fake landing page.” isn’t it against adwords TOS to do stuff like this?

I keep reading of people using adwords and collecting email addy’s and fake this n that but ive also read of adwords accounts being shut down for this type of testing.

Apurva Mehta Reply:

@John, I was not aware that this was against the TOS of adwords. I just assumed that it was OK since so many people do it.

Honestly, I would be surprised if it were against the TOS. The behavior is pretty innocuous.

For instance, if a user did click the sign up button, I would display this message:

“Sorry, but we are having technical difficulties and cannot complete your request at this time. Please check back later”

That is not really harmful, and I suspect non-fake landing pages would use that language at times of technical difficulty as well.

John McFarlane Reply:

I agree while not harmful as you are not taking money from anyone it still might be worth checking if its safe to do things like this with adwords and any other PPC advertising.

I also agree that lots of people do this, i know that i wont be though, not worth losing an adwords account for this.

#4 Seppo on 11.29.11 at 11:23 pm

With my small resources and monies, IMO, surveys are useless, you need to experiment.

I have found StumbleUpon as best tool to drive traffic for experimenting. Their own statistics are not very good (they give you nice looking statistics but then you realize that they do not reveal the normalization functions for their user distributions, age etc…) and one has to be really careful drawing conclusions.

I only sort of trust ‘like %’ in SU and otherwise simply use SU’s ‘5c per user basic mode’ (Paid Discovery) to drive traffic and then use Google Analytics for analysis (pages per view, time on site) to give me some idea what’s working and what is not.

#5 Johannes on 11.30.11 at 6:15 am

Interesting discussion, to early get to know the potential of an idea is worth so much further on.

I use to put together a product sheet in one to two pages including the idea (the hook), the value to customers, price info etc. Then I go out and meet maybe five to ten potential customers, buy them a lunch and discuss the product. I think you can get tons of information this way. I even changed my hook slightly after input from one of those discussions.

As always in marketing I think it is important to really try to understand what can make the potential customer to get a picture of what you are doing in a really short time.

I work with a B2B software so it’s maybe a bit different, but for a idea for an iPhone app I have, my plan is to create a mock-up of the App Store screen, print it out and go out and interview a bunch of potential buyers.

Rob Reply:

Thanks Johannes; these are good tips.

#6 John McFarlane on 11.30.11 at 11:07 am

I would say that it depends if you’re getting people to complete the survey without a reward offered, because i dont think that paying people to complete a survey would be useful as the types of people completing surveys for cash are unlikely to want or pay for your product/service.

Theres a possibility that when creating a survey that you might ask the types of questions that tend to guide people into telling you what you want to hear and maybe not what you should hear.

I imagine that if you can get your survey out to people on your email list who are already showing early interest then thats probably going to work better.

Ive yet to test ideas with surveys but these are some of the things that ive thought about and will take into consideration if i do use surveys.

Maybe surveys, then at some other point in the process another survey, i imagine that some people could change their minds and opinions, ive chatted to people before (general chat, not business) that have contradicted themselves within the same setence, some people can be fickle and i feel that something stated by an individual might not be as concrete as it sounds.

You can ask a person a question and they give an answer and then a few days later, maybe being in a different mood, under stress or less stress or circumstances having changed they will then give a conflicting answer to their previous one.

#7 Andre on 12.06.11 at 12:10 pm

I know that we are always told that it is futile to ask “would you pay x for y” but I still wonder if a survey on pricing sent to existing customers would have saved Netflix a lot of grief.

John McFarlane Reply:

It’s interesting how when people walk into a shop or shop online they are offered something at a certain price and in a sense told how much something is (exceptions to this of couse) and people know that they can take it or leave it.

Yet many are hesitant to ask how much someone would pay for something, while asking someone how much they think something is worth would probably be futile, i think theres probably an approach to asking the right way if people will pay x amount for something.

#8 Kyle on 12.09.11 at 6:56 pm

Everyone above is right. It is a blend. You need to talk to possible users and build empathy to their pain to understand what to build. But, building a rapid prototype and user testing it can now be done in under and hour. I work for YouEye.com, which is a user testing tool and service, and we get this type of question alot. My simple advice is to build some user empathy, create an interactive mock with HotGloo.com or myBalsamic, and then run a few user tests using YouEye’s free account. This approach will give you a nice baseline of findings. You will better understand the messaging, the app design, and the must have features that work for your audience.

In short…
– connect with your audience and feel their pain
– brainstorm and prototype an interactive mockup solution
– user testing your prototype