I don't know how true it is in normal cases, but I can tell you that when I had my stroke, which radically truncated my ability to juggle multiple inputs, this suddenly became very concretely true. Every distinct object in my visual field was a constant distraction; keeping my space free of stuff (not just clutter, but also decorations, gifts, books, distinct objects of all sorts) made working on cognitive exercises, or even having a conversation, much much easier.