I probably blatantly reveal my ignorance by asking this, but do only agents who know what they want have a utility-function?

Nope - in theory, all agents have a utility-function - though it might not necessarily be the neatest way of expressing what they value.