When philosopher Susan Haack wrote "Evidence and Inquiry" back in 1995, she really hit the nail on the head on this one. I'll share an extensive quotation from her, and then I'll make a couple remarks:

The obser­vation that people have many beliefs in which they are not, or not much, justified...hints, though it doesn't say explicitly, that people also have beliefs in which they are justified. And it is a legitimate question, certainly, what reasons there are for even this degree of optimism. On this issue, it may be feasible to appeal to evolutionary considerations. As I observed in chapter 9, compared with other animals, human beings are not especially fast or strong; their forte is, rather, their greater cognitive capacity, their ability to represent the world to themselves and hence to predict and manipulate it....

Returning, now, to the main thread of the argument, let me repeat that my sights are set much lower than Descartes's; I have aspired only to give reasons for thinking that, if any truth-indication is available to us, satisfaction of the foundherentist criteria of justification is as good an indication of truth as we could have. Even this very significant lowering of aspirations, however, will not in itself constitute any reply to the more notorious difficulty with Descartes's enterprise: the vicious circle into which it is generally supposed Descartes got himself. Aren't my ratificatory arguments, however hedged, however modest in what they aspire to do, bound to be viciously circular?

I don't think so.

First: I have not offered an argument with the conclusion that the foundherentist criteria are truth-indicative, one of the premisses of which is that the foundherentist criteria are truth-indicative.

Second: nor have I (like those who hope for an inductive meta­-justification of induction) used a certain method of inference or belief­ formation to arrive at the conclusion that that very method is a good, truth-conducive method....

This is probably not enough to allay all suspicions. "Yes, but how do you know that the senses are a source of information about things in one's environment, that introspection is a source of information about one's own mental goings-on?', I may be asked, echoing the familiar challenge to Descartes, 'How do you know that God exists and is not a deceiver?' The question will be put, no doubt, in a tone which suggests that the only response available to me is, 'because my evidence satisfies the foundherentist criteria', echoing the anticipated answer from Descartes, 'because I clearly and distinctly perceive it to be true' . I shall put aside the question whether Descartes has any recourse against this challenge, and concentrate on my own defence. For simplicity, let 'R' abbreviate all the direct reasons I have offered in my ratificatory argu­ment. The anticipated question, 'Yes, but how do you know that R?' is rhetorical, a challenge rather than a simple request for information, and it may be taken in either of two ways: (1) as a challenge to give my reasons for believing that R, or: (2) as a challenge to show that my reasons for believing that R are good enough for my belief to constitute knowledge. I cannot meet the second challenge without articulating my standards of evidence and showing that my evidence with respect to R satisfies them, and, at least arguably though not quite so obviously, without offering reassurance that my standards of evidence are truth­-indicative; and if so, I cannot meet it, in the present context, without circularity. But I can meet the first challenge simply by giving my reasons for believing that R. And this is enough. My reasons are good reasons if they are independently secure and they genuinely support R; and I am justified in believing those reasons, and hence in believing that R, if my evidence for believing them is good evidence. And if I am justified in believing that R, then (assuming that R is true, and whatever else is needed to avoid Gettier paradoxes) I know that R. And if I do and if R (and the indirect reasons on which it depends) are good reasons for believing that the foundherentist criteria are truth-indicative, I know that, too. Even if I can't know that I am justified in my weakly ratificatory conclusion, I can be justified in it nonetheless; and even if I can't know that I know it, I can know it nonetheless.

My main remark is that Eliezer doesn't need to rely on induction to justify induction. The mere assumption that the universe has natural laws is enough. Given that, one can look at the universe timelessly, and say, "I see evidence of this natural law, and, given that the universe has such a natural law, this will happen tomorrow." Then we aren't predicting the future based on the past; we're predicting the future based on our conception of the natural laws of the universe, which just happens to arise from observations we've made in the past. This removes circularity, but requires an additional assumption (which, if false, implies skepticism, as far as I can see).

Of course, standards of evidence do need to be evaluated circularly, but as both Eliezer and Haack noted, the circularity is not vicious.

As for Occam's Razor, Haack doesn't mention it much, and I'm still working out my own thoughts on the problem. I think Kevin Kelly's work is promising, in spite of the fact that he rejects Bayesianism (and consequently, his approval on Less Wrong was mixed when he was previously mentioned). Of course, the proofs of the optimality of Solomonoff induction are probably really, really important here, too. I need to study this more; I'll post another comment (probably not for at least 6 months) giving my position after I've actually done enough research to be confident in a position.

Edit (over a year later): I no longer have plans to do further research on Occam's Razor. I will add that, though I find Kelly's work interesting, I don't think it is a panacea answering once-and-for-all why Occam's Razor works, or is justifiable.