It seems impossible to choose whether to think of ourselves as having free will, unless we have already implicitly assumed that we have free will.

That just means that you lack the ability to think in non-free will mental frameworks. I don't think a calvinist who thinks that God is in control from everything has a problem with the word choose and thinking of himself as making choices.

More generally the entire pursuit of acting more rational is built on the implicit premise that we have the ability to choose how to act and what to believe.

Not completely. It's build on the idea that we have an influence on what we choose and what you believe.

If you take a belief like: "There's no ego depletion." it's not straightforward to acquire that belief. All the studies that tell me that having that belief means not having ego depletion are not enough to help me acquire that belief while I'm suffering from ego depletion in the present.