I suspect my following questions demonstrate such high level of confusion, that I am not sure if they even mean what I think they mean, but still I think this is the best place to ask them:
For 1, the mental model for non-relativistic but high speeds should be "a shallow crater is instantaneously vaporized out of the material going fast" and for relativistic speeds, it should be the same thing but with the vaporization directed in a deeper hole (energy doesn't spread out as much, it keeps in a narrow cone) instead of in all directions. However, your idea of having a spacecraft as a big flat sheet and being able to tolerate having a bunch of holes being shot in it is promising. The main issue that I see is that this approach is incompatible with a lot of things that (as far as we know) can only be done with solid chunks of matter, like antimatter energy capture, or having sideways boosting-rockets, and once you start armoring the solid chunks in the floaty sail, you're sort of back in the same situation. So it seems like an interesting approach and it'd be cool if it could work but I'm not quite sure it can (not entirely confident that it couldn't, just that it would require a bunch of weird solutions to stuff like "how does your sheet of tissue boost sideways at 0.1% of lightspeed".
For 2, the problem is that the particles which are highly penetrating are either unstable (muons, kaons, neutrons...) and will fall apart well before arrival (and that's completely dodging the issue of making bulk matter out of them), or they are stable (neutrinos, dark matter), and don't interact with anything, and since they don't really interact with anything, this means they especially don't interact with themselves (well, at least we know this for neutrinos), so they can't hold together any structure, nor can they interact with matter at the destination. Making a craft out of neutrinos is ridiculously more difficult than making a craft out of room-temperature air. If they can go through a light-year of lead without issue, they aren't exactly going to stick to each other. Heck, I think you'd actually have better luck trying to make a spaceship out of pure light.
For 3, it's because in order to use ricocheting mass to power your starcraft, you need to already have some way of ramping the mass up to relativistic speeds so it can get to the rapidly retreating starcraft in the first place, and you need an awful lot of mass. Light already starts off at the most relativistic speed of all, and around a star you already have astronomical amounts of light available for free.
For 4, there sort of is, but mostly not. The gravity example has the problem of the speeding up of the craft when it has the two stars ahead of it perfectly counterbalancing the backwards deceleration when the two stars are behind it. For potentials like gravity or electrical fields or pretty much anything you'd want to use, there's an inverse-square law for them, which means that they aren't really relevant unless you're fairly close to a star. The one instance I can think of where something like your approach is the case is the electric sail design in the final part. In interstellar space, it brakes against the thin soup of protons as usual, but nearby a star, the "wind" of particles streaming out from the star acts as a more effective brake and it can sail on that (going out), or use it for better deceleration (coming in). Think of it as a sail slowing a boat down when the air is stationary, and slowing down even better when the wind is blowing against you.
I was afraid my questions might get ridiculed or ignored, but instead I've got a very gentle and simply expressed explanations helping me get out of confusion. Thank you for taking your time for writing your answer so clearly :)
Usual neutrinos or dark matter won't work, but if we go to the extremely speculative realm, there might be some "hidden sector" of matter that doesn't normally interact with ordinary matter but allows complex structure. Producing it and doing anything with it would be very hard, but not necessarily impossible.
A bit late to the party here, but here's an alternative proposal to shielding:
I disagree with the statement
it's going to take the form of a super-narrow pinprick of kinetic energy directed on a single point
0.9 c is not ultra-relativistic, the Lorentz factor at this speed is a mere 2.3, so the kinetic energy of an incoming particles is on the same order of magnitude of its rest mass. This means that relativistic beaming is going to be fairly weak.
Since the kinetic energy far outstrips the chemical binding energy of the dust grain, we can expect it to instantly disintegrate into a particle shower upon first contact with the shield. So why not place the shield hundreds of kilometers ahead of the ship and let the inverse-square law do the rest? Most of the particle shower will simply miss the ship. Multiple, thinner shields would be even better at diffusing the particle shower while taking less damage themselves.
I imagine there will only be some minor damage to the shield since most energy still lies within the particle shower if you made the shield thin enough, and you wouldn't even need advanced nanotechnology to repair, just fill the hole in the shield with ANY material. And since we're moving at 0.9 c, you'd need to make the shield at most a few hundred meters wide to prevent dust sneaking in between the shield and the ship.
Furthermore, if the ship itself already uses a magnetic field to protect itself against charged particle radiation, a soap bubble would suffice as a shield as it would be enough to vaporize and ionize the dust grain. Either way, I think there is enormous mass savings possible here.
I'd also like to bring up that the idea you mentioned of having multiple ships in a line so only the first one needs substantial dust shielding, is the same reason it makes sense to make your ships as long and thin as possible.
This is extremely speculative, but one way it could be possible to build very sturdy probes is, if we there was a phase of matter whose binding energies were typical of the nuclear forces (or some other, hitherto unknown strong force) rather than the electromagnetic force, like usual matter. Strangelets are one candidate.
(1, 2, 4, 5)
Part 3a: Dust and Explosions
To a first approximation, there's exactly one thing that sets the speed limit on going fast in space.
Dust.
In the future, dust will be a Very Big Deal, as it's the dominant constraint on the most-important instrumental goal of going fast.
Anders Sandberg's paper pointed out dust as a constraint on interstellar probe design, but I didn't realize exactly how huge of an obstacle dust was until I started playing around with a spreadsheet.
To start with, interstellar (and intergalactic) dust has a size distribution, which tells you, for a given diameter range, how many dust grains there are of that diameter in a given volume.
At least for the range of 35-120 nanometers, (which shows up especially strongly in astronomical observations) it follows a power-law distribution, with an exponent of -3.5.
However, this dust isn't what we're worried about. There's erosion from protons and small dust hitting your dust shield at relativistic speeds, but doubling the dust diameter means it's 8x as massive and hits with 8x the energy.
At 0.9 c, 180 nm dust hits with 1 joule of energy. So all the normal dust isn't that much of an issue.
Going up to dust that's 1 micrometer wide, it hits with about 100 joules of energy, the energy of a firecracker. For all the following explosion comparisions, note that it's going to take the form of a super-narrow pinprick of kinetic energy directed on a single point, which is more destructive than a simple explosion, which radiates in all directions and has much of its energy dissipated as heat.
Destructive power keeps scaling rapidly, with about a factor-of-ten increase for every doubling in dust diameter, until we reach 20 micrometer dust, which hits with the energy of a grenade.
40 micrometer dust hits with the energy of 1.5 kg of dynamite. 86 micrometer dust hits with the energy of 30 bricks of C4. 0.18 mm dust hits with the energy of half a cruise missile. 0.4 mm dust hits with the energy of the Oklahoma city bombing. 0.86 mm dust hits with the energy of the largest non-nuclear weapon, the Russian FOAB. 8.6 mm dust hits with the energy of the Fat Man nuclear weapon, and 1.8 cm dust (a ball bearing) hits with the energy of a W87 fission warhead.
So, from about 1 to 20 micrometers, we get a pretty decent amount of boom that's shieldable. Whipple shields are the current standard for micrometeor impact. They have a protective thin layer that gets hit, turning the blast into a cone of shield-vapor, and then the force of the blast is dissipated over the area of a cross-section of the cone on the main bulk of the ship, which is much more manageable. However, I'm pretty sure that at relativistic speeds, the cone gets a lot more narrow, so they get less effective.
20 micrometers to about .1 mm is handleable if your ship is really damn sturdy.
.1 mm to 1 mm requires increasingly large dust shields that will start looking more asteroid-like, getting bigger than the mass of the rest of the ship, as they have to be that big to tank a hit from the largest non-nuclear weapon focused in a single tiny pinprick and narrow cone. Remember, by relativity, there's no difference between cruising through the interstellar medium at 0.9 c and being in the beam dump of a particle accelerator that's whipping stuff up to 0.9 c. Anything larger than 1 mm requires that most of the mass of the mission is composed of an asteroid, with the size of the asteroid rapidly scaling with dust size.
So, up to about 10 micrometers, we need a decent dust shield, 10 micrometers to 0.2 mm requires the sort of dust shield that can tank a hit from a cruise missile focused in a single point, and beyond that we basically have to whip an asteroid up to 0.9 c and attach a small ship to it with very rapidly scaling asteroid size. This requires quantities of energy that could blow the crust off a planet.
We know a lot about low-diameter dust that can be conventionally shielded with little issue, but we know very little about the distribution of higher-diameter dust, and that's the dominant constraint on mission speed and colonizing the universe. Of course, if we get a really bad distribution of higher-diameter dust, we can always go slower. For non-relativistic speeds, halving the velocity cuts the impact energy by a factor of 4, and for relativistic speeds, you get a lot more than that because of decreases in the relativistic-mass of the dust grain.
Maybe we'll get lucky and find that there's a sharp dust-grain cutoff beyond a certain size. Maybe we'll get unlucky.
Part 3b: Dust Distribution Facts and Implications
There are three relevant considerations I found, trying to work it out from first principles and astronomy facts. The first is that a dust size distribution implies a certain amount mass in a volume of space by doing the appropriate integral over diameter. The -3.5 exponent means that the amount of mass diverges. In order for the integral to converge and have finite dust mass in the universe, you need an exponent a hair below -4. But we don't know the diameter where the exponent shifts down to -4 or lower.
The second is that the asteroid belt has a size distribution of -3.5, and this is apparently characteristic of fragmentation processes. The reason there isn't infinite mass in the asteroid belt is because there's a size cutoff at the mass of Ceres. And we get the intuitive result that the mass of the asteroid belt is mostly in large asteroids.
The third consideration is that dust comes from many processes. Supernovae and dying stars floof out a bunch of dust into the environment. We found a supernova grain as large as 25 micrometers once, which is worrying. But most supernova dust is a lot smaller than that. For the millimeter-size dust grains, I imagine it'd come from planetary formation discs that got disrupted, which is a different process with a different dust production rate. So I'd expect different regions of space to have different dust size distributions, some of which might come with a natural mass cutoff. Maybe molecular clouds with forming stars are especially dangerous. Maybe the void between galaxies is mostly devoid of fatal dust (relative to the hydrogen density). Maybe dust gets more and more abundant as a galaxy ages so it's much more dangerous to travel in distant galaxies that have aged by the time we get there. We don't know, but it's probably modelable.
Now, there's two more things to note.
The first is that required-asteroid-mass to shield against the largest dust grain likely to be encountered is ridiculously sensitive to the scaling exponent, and pretty sensitive to how fast you're going. Pretty much, if you make your asteroid have twice the radius, you get 8x the mass, so you can tank 8x larger explosions, right? Well, maybe tankable explosion power doesn't scale linearly with mass, I'm unsure. But more importantly, your asteroid now sweeps out 4x the area because it has 4x the area, so you're 4x more likely to hit dust of a given size. Now, overall, you're still better off, but an increase in mass doesn't buy you nearly as much dust protection power as you'd naively assume, so dust still sets a pretty hard speed limit with quite rapidly scaling asteroid mass for traveling longer distances and higher velocities.
The second is that, due to the fact that dust is the dominant obstacle to going really fast, there will be an awful lot of optimization power directed at this problem, so the standard caveats apply about concluding that even a transhuman civilization can't do high-speed missions due to dust. Two obvious improvements I can see are making materials that are really good at dissipating massive pinpoint kinetic energy strikes, and finding some way to deflect dust. I think there's ways of charging the dust ahead of you and using a magnetic field to move it out of your way, but it's hard because we're mostly interested in large dust which is a lot less susceptible to these shenanigans, though I'd have to check. Also, any dust deflection system (and the power drain imposed by it) must be running full-time over the intergalactic voyage, which brings in the standard problems about making machinery that long-lasting.
Edit: In the three hours since typing this, I found that someone invented a completely novel deflection strategy I missed, and I also invented another one on the spot, proving my "don't underestimate the future" point very well. The one I didn't come up with is throwing a bunch of liquid metal droplets ahead of your ship, enough to ensure that a dust grain hits at least one of them and explodes, like an extremely long-distance whipple shield and very slightly accelerating the whole way so you can recapture the droplets and launch them back ahead of you. This has the issue of requiring continuous acceleration, and losing mass the whole way due to cosmic ray spallation of the droplets, and droplet vaporization when they get hit by smaller dust grains. Off the top of my head, it'd be pretty decent for an in-galaxy mission, but I worry that for intergalactic missions, the cumulative mass loss from droplets getting destroyed, and the propellant/continuous engine operation required to continuously accelerate the whole way, would be a bit much, plus it doesn't work on deceleration, just coasting. No, I'm not going to redo my design from scratch to take this into account, it's eaten enough time already. As for my insight, it's that if you have many spacecraft in a line, each can protect the next one, so the volume of space swept out by the fleet is much lower. Or, heck, you can just have the first dozen in the train being inert blocks of rock and only build important attachments for the stuff in the back.
So, for my mission, I assumed we're just directly tanking the impacts on a giant block of graphite, and there's a dust scaling exponent of -5 in intergalactic space (there are less protoplanetary discs which is where I think a lot of the scary dust comes from, and there aren't a lot there), a scaling exponent of -4 in interstellar space, and -3.5 closer to a star. As an example, shifting the dust scaling exponent of intergalactic space to -4.5 increases the mass of the asteroid we have to send by about 3.4 million times. This is what I meant by mass being ridiculously sensitive to scaling exponent size. The resulting dust shield mass per supercluster-ship (mostly dust shield though) is about 120,000 tons for a squat cylinder of graphite 42 m or about 140 feet long , or about 1/5th the mass of the titanic. Also we'll need about 30 of these for a 99.9% chance that at least one survives (higher survival probabilities are attainable by just sending more) It's far more efficient on a mass basis to send a fair few ships with a moderate chance of survival than to send one big ship with a 99.9% survival chance.
So in summary, dust size is the dominant constraint by far on how fast you can go, with unacceptably rapid-scaling mass increases as the exponent on the power law goes up.
Edit: Unless transhuman or mere-human ingenuity comes up with a way to cheat some part of the dust problem, in which case we're back in business.