Do you think that the Dirichlet Processes models that machine learning people use might be relevant here? As I understand it, a DP prior says that the true probability distribution is a discrete probability distribution over some countable set of points, but you don't know which set in advance. So in the posterior, this can consistently assign some nonzero probability on a single point -- in fact, if you do the math the posterior is very simple, it's a mix between a DP and some finite probability mass on the values that you did see.