Very true. Staff notation essentially says "Here are the pitches and rhythms, now it's your job to figure out how to make them happen on your instrument." As you point out, a very real alternative to staff notation exists in tablature, which (in general) is any notation system that instead says "Here's what you need to do physically on your instrument. Follow these instructions and the notes will automatically be the right ones—you don't need to worry about what they 'are'."

Tablatures are surprisingly old, apparently going back 700 years or so in various forms. Of course, their drawbacks as general musical notation are clear enough. Namely, if you want to understand what's going on in the music or play music on a different instrument, tablature is really only a kind of lookup table for actual notes, and often a very cumbersome one.