the huge amount of surface area would nonetheless heat up and be visible in the infrared

Unless you're spending some of the energy the Dyson sphere is collecting to actively cool that surface area.

all the energy would eventually get absorbed/used SOMEWHERE

Sure, but if you point it at another galaxy, it will take a very very long time for something to happen (and in a galaxy far, far away, too).