Yes! This! User test all the things! I never understood[1] why user testing wasn't more popular in fields other than tech, and for things where the stakes are lower, like blog posts.
I previously took a brief stab at making a similar point in the context of writing readable code.
Well, I do have some hypotheses related to biases, incentives, and trivial inconveniences. ↩︎
It's weird that I have my own startup, completely understand using real users for user testing, and also barely ever "user-test" any of my writing with actual audience members.
Once you shared with me your document it became super clear I should, so thank you!
I am not sure you can read too much by immediate reaction. If the article amounts to an attack on beliefs they are vested in, then initial reactions can be strongly defensive (lawyer mode - defend a position), but a week of thinking about it can result in change. The positive sign there is coming back to you with more questions. (a shift to science mode - curiosity about truth).
I read an interesting book defending and explaining the truth of evolution written primarily for a Christian audience. The author explained the process whereby he changed from being 6-day creationist to conventional science position. It wasnt quick, it involved multiple arguments, and it also needed time for him to think without being pressured about it by evolutionists.
I agree with everything
For most posts here (such as hiring posts), I do think the author can/should aim to a changed mind "immediately", mainly because (1) it is possible and not too hard, and (2) not aiming for that can be an easy enough "excuse" to avoid admitting that the post failed at what it's supposed to do. (Almost every post will get an encouraging review response of at least "nice post!", so having that as a bar seems too low, I think).
I'll add that another thing that influences my opinion is (3) I think lots of people fill their post with counter-arguments to arguments that nobody would really make and I think this really lowers posts' quality.
For very hard posts such as "convince about evolution" - perhaps a goal of "change someone's mind immediately" is too high, but I assume you'd still agree that doing user testing would be very valuable. (?)
Good post! Stop assuming things are/aren't good and go and look.
Is it worth making live-call-reviews a feature request for the LW feedback system? (Possibly limited to higher karma than the 100 required for text feedback, as I imagine this would have a smaller bottleneck with timezones etc.?). I imagine this would encourage a lot more people to say "I’ve got to do this now!".
Are you writing an article that’s supposed to change people’s minds and/or actions? For example, an article about career advice, productivity, or getting people to apply for a job? Then this post is for you!
TL;DR: For important articles I recommend doing real user testing (even with just one person!). Try it and you’ll never go back.
I specifically recommend:
(1) Watch them read it, don’t just ask for Google Doc comments
Open a zoom, share screen, and ask them to share their thoughts live as they read.
(2) Show it to your target audience, don’t just ask others what your target audience would think
If your article is supposed to change developers’ career plans, ask a dev who might change their career plan. Don’t ask a biologist what a dev might think.
(3) Check if they changed their mind, not just if they like your writing
If they end with “I’ve got to do this now!”: That's a good sign.
If they say “sounds interesting, I’ll think about it”, that’s not as good.
I personally like to first ask the other person for their opinion (for example, “what’s your current career plan?”), then let them read the article, and then ask again.
The general intuition:
If you want to adopt my frame of mind, ask yourself: Would your idea of how to improve [the article’s persuasiveness] also improve [how engaging a startup’s app] is?
Here are some exercises, do you think these would work?
Hint: Is the test [checking if a real user from your target audience changes their mind] or [using some proxy to guess whether a real user would change their mind] ?
How to start?
Easiest: post in Bountied Rationality, or in another group that has your target audience.
A draft post you can copy & paste: “Hey, I’ll pay $10 for someone in [describe your target audience here] to review a post of mine for the EA/lesswrong forum. I want to do it live, over a video call. The call is capped at 30 minutes (and expected to be less). Comment or message me if you’re interested. Tagging or referring relevant people is also appreciated.”
Thanks to Matt Brooks and Daniel Reeves for helping make this article better <3