Knowing What to Test
Testing is not only essential to better understand donors but also to remove our own personal biases and blind spots from the decision-making process. If you like how a donation page looks or your board chair likes how a direct mail letter sounds, those aren’t objective data points to base strategies on. In fact, often what you and your board chair like or think will work is the direct opposite of what donors respond to. Through experimentation and testing, you can be more objective and move quicker by simply creating experiments to uncover insights instead of squabbling in board rooms to see whose opinion will win out.
So, what should you test? This is the first question people ask when they start to look at testing, and it ultimately depends on who you are and what your role and responsibilities are, but some key questions to consider when looking at what to test are:
- What will we do differently in the future if this test is true? If it’s false?
- What will this test teach us about our donors?
Ultimately, you are testing to get better results in the future so if a test doesn’t have a clear learning about your donors or a clear action for the future, it may be a low-impact test. A classic example of a test with a clear learning is a button color on a donation page. If blue is better than red, what do you do differently in the future? Use blue buttons…and that’s it. What does it tell you about your donors? They prefer blue buttons on donation pages…and that’s it. So the future application is clear.
Many nonprofits struggle with volume or having a smaller sample size, so a common strategy for testing is to go higher in the donor funnel or earlier in the process. For example, only a handful of people may visit a donation page, but many more will visit a home page. So you can run a test on the home page to get more people to a donation page. Or if fewer people are clicking the links in your emails, you can test a sender or subject line to get more people to open the email in the first place.
Launching a Fundraising Experiment
For seasoned and new professionals alike, fundraising is difficult. As a fundraiser, advocate, or executive, you are always on the search for people that identify with your cause, as well as how you can explain it in a way that resonates with them and inspires them toward action. Testing is a vital tool that provides constant opportunities to tailor your engagement to the variety of personality types, interests, backgrounds, and life experiences that your supporters have.
Launching an experiment is relatively easy, but there are still some key steps to running a good test that can produce a reliable result and valuable insight. Here are eight essential steps to running a good fundraising experiment:
- Identify your goal.
This is the problem you want to solve or what you’re trying to accomplish. For a donation page or direct mail appeal, it may be to increase conversion/response rates, but for your blog or newsletter, it may be to increase engagement.
- Make sure you can measure your goal.
If you can’t track it, you can’t test it. So, in the case of your donation page, to see if you’re increasing the conversion rate, you need to be able to measure…your conversion rate.
- Craft your hypothesis.
This is the idea you’d like to test to see if it will help you reach your stated goal. For example, your donation page hypothesis may be, “Strengthening the value proposition by adding more copy about the impact of a donation will increase the conversion rate of my donation page.”
- Calculate your needed sample size.
This will show you how many people need to be included in your test—and over what time frame the test must run—to get a reliable result. The larger your sample size, the more precise you can be, and the smaller changes you can make. Whereas, the smaller your sample size, the more risk you have to take on and/or the bigger changes you need to make.
- Design your treatment.
In the case of a donation page, your current page is the “control,” so you need to create a new page which will be the “treatment” or “challenger.” Your treatment should line up with your hypothesis to be sure your results are aligned with your experiment goal.
- Set up your experiment.
Now comes the fun part: actually running the experiment. There are a lot of tools for online testing, and many of them are free, making some experiments easy to set up and monitor.
- Validate your results.
Once your test is done, you can see if you have a statistically significant or reliable result. You can also look for other ideas to test in the future.
- Share your learnings.
Just because the test is complete doesn’t mean your job is. Testing is about results, yes, but it’s also about learning, so log your experiment and share with coworkers and colleagues to make sure they can learn from the test as well.