The "free will" debate is a confusion because, to answer the question on the grounds of the libertarians is to already cede their position. The question they ask: "Can I make choices, or does physics determine what I do?"

Implicit in that question is a definition of the self that already assumes dualism. The questions treats the self as a ghost in the machine, or a philosophy student of perfect emptiness.

The concern of libertaarians is actually that external events determine what they do. They don't mind their actions being caused by neural events in their brain. A libertarian may accept that they are constituted of physics but that is not the same thing as being determined by physics. Being constituted of physical stuff is on the face of it neutral with regard to determinism and libertarianism.

The libertarians imagine that we should be able to make decisions not only apart from physics, but apart from anything.

No. The phrase typicall used is "not entirely determined by".