In canon, Voldemort has extra "legilimency" style power to know harry's thoughts and influence him due to the scar, even though their connection makes it dangerous for the two to touch (at least while he was in Quirrel). Obviously this isn't great evidence, but I think it's relevant.