Interesting, there was recently a somewhat related question posed here.

Using my experience from that question I can give a pretty large group of answers for problem 1. Pubbfr nal z, naq nffvta n ahzore bs pbvaf tvira ol n ovabzvny qvfgevohgvba jvgu a=z naq fbzr c xabja gb Obo, fb gung vg'f nf vs lbh syvccrq na vaqrcraqrag pbva sbe rnpu rairybcr.

2 is a bit tricky, since perfect mathematicians don't just eliminate the obviously wrong, they also update against the unlikely but right. Bayes' rule says that if you get a coin, you multiply your probabilities by P(got coin | X envelopes filled)/P(got coin). P(got coin) is 1/2, your chance of getting a coin if there are X coins is X/m, and your prior that there's X coins is 1/(m+1) (since 0 is a valid number of coins too). So after getting a coin, the hypothesis that there are X envelopes with coins in them gets probability 2*X/m * 1/(m+1).

Gut check stop. This means that for m=2, Bob would say, P(0) = 0, P(1) = 1/3, and P(2) = 2/3 after getting one coin. Looks right.

Each hypothesis leads to an expected value of (X-1)/(m-1). So we take the sum of 2X(X-1) / (m-1)m(m+1) (thanks wolfram) to get 2/3. No matter the m, the expected value for the second draw is 2/3! It's Laplace's rule of succession! Cool, huh? I'm going and giving damang an upvote just for how helpful his post was for this one. Shame about it making the problem unanswerable :P

EDIT: part two was answering the wrong question, see comment.