it's not found in any Buddhist primary source

What is a Buddhist primary source? None of the discourses were written down until some centuries after the Buddha's time. The discourses that we have do themselves exist and whatever their provenance before the earliest extant documents, they are part of the canon of Buddhism. The canon has accreted layers over the centuries, but the Kalama Sutta is part of the earliest layer, the Tripitaka.

I've heard it might be

You've heard? That it might be? :-)

It is readily available online in English translation. It attributes these words directly to the Buddha:

Come Kalamas. Do not go upon what has been acquired by repeated hearing; nor upon tradition; nor upon rumor; nor upon what is in a scripture; nor upon surmise; nor upon an axiom; nor upon specious reasoning; nor upon a bias toward a notion that has been pondered over; nor upon another's seeming ability; nor upon the consideration, "The monk is our teacher." Kalamas, when you yourselves know: "These things are bad; these things are blamable; these things are censured by the wise; undertaken and observed, these things lead to harm and ill," abandon them.

and in another translation:

Now, Kalamas, don't go by reports, by legends, by traditions, by scripture, by logical conjecture, by inference, by analogies, by agreement through pondering views, by probability, or by the thought, 'This contemplative is our teacher.' When you know for yourselves that, 'These qualities are skillful; these qualities are blameless; these qualities are praised by the wise; these qualities, when adopted & carried out, lead to welfare & to happiness' — then you should enter & remain in them.

If I had the time, I'd be tempted to annotate the passage with LessWrong links.

ETA: For the second translation, the corresponding paragraph is actually the one preceding the one I quoted. The sutta in fact contains three paragraphs listing these ten faulty sources of knowledge. Buddhist scriptures are full of repetitions and lists, probably to assist memorisation.

ETA2: Rationalist version: Do not rest on weak Bayesian evidence, but go forth and collect strong.

Great catch. Upvoted.

I've heard it might be a rough paraphrase of a quote from the Kalama Sutta, but in its original form, it would not qualify as a "rationality quote"; it's more a defense of belief in belief, advising people to accept things as true based on whether believing it is true tends to increase one's happiness.

I actually don't think this is right though. I'm pretty sure the original form is about the importance of personal knowledge from direct experience. I think the wikipedia article makes this clear, actually. I suppose you're taking your reading from:

Kalamas, when you yourselves know: "These things are good; these things are not blamable; these things are praised by the wise; undertaken and observed, these things lead to benefit and happiness," enter on and abide in them.'

But the emphasis here should be on "when you yourselves know", not "these things lead to benefit and happiness". Keep in mind the kind of teachings being addressed are often strategies for happiness so it makes sense to be concerned with whether or not a teaching really does increase happiness.

I don't see why we can't take it as an injunction to trust only experiment and observation. It seems about right to me.

(ETA: Except of course he's talking about meditation not experiment and ignores self-deception, placebo effect, brain diversity and the all important intersubjective confirmation, but I'll take what I can get from the 5th century B.C.E.)