SVMs are pretty bad for hyperparameters too

How so? Linear SVM main hyperparameter is the regularization coefficient. There is also the choice of loss and regularization penalty, but these are only a couple of bits.
Non-linear SVM has also the choice of the kernel (in practice it's either RBF or polynomial, unless you are working on special types of data such as strings or trees) and one or two kernel hyperparameters.

I struggle to see how DNNs can have hundreds of hyperparameters - looking at the code for the paper I linked to, they seem to have learning rate, 2 parameters for simulated annealing, weight cost and batch size. That's 5, not counting a few others which only apply to reinforcement learning DNNs. Admittedly, there is the choice of sigmoid/rectilinear, and of the number of neurons, layers and epocs,

I haven't read all the paper, but at glance you have: Number of convolutional layers, number of non-convolutional layers, number of nodes in each non-convolutional layer, for each convolutional layer number of filters, filter size and stride. There are also 16 other hyperparameters described here.
You could also count the preprocessing strategy.

Other papers have even more hyperparameters (max-pooling layers each with a window size, dropout layers each with a dropout rate, layer-wise regularization coefficients, and so on).