Observing the sky is good and productive science. Perhaps he meant that as an emperor (or responsible senator, etc) he should not have been drawn into a serious scientific or philosophical career, but for those who can afford the time and effort, it's a fine pursuit.