It depends a lot on the quality of the speed-up algorithm. One cheap way of speeding up audio is to drop samples, but this significantly reduces audio quality.

Personally, I find anything above 1.2x to be annoying, but I still do it - not to save time, but to improve my japanese-understanding capability.