If I'm interacting with people, I treat it as rude to pull out my phone without asking.

If I'm already not-interacting-with-people, I don't see why it would be any worse than a book. So many other people have smart phones that "socialize while waiting" is dying off regardless of what I do, and a book generally kept people from trying to strike up a conversation anyway.

As to the "not aware I'm not aware"... I've always felt equally towards books and smart phones. Possibly a bit more aware with my smart phone, actually, since dropping it or having it stolen is a much bigger deal.