Tiling the wall with impossible geometry seems reasonable, but from what I recall about the objects in Dumbledore's room, all the story said was that Hermione kept losing track. Not sure whether artist intent trumps reader interpretation, but at first glance it seems far more likely to me that magic was causing Hermione to be confused than that magic was causing mathematical impossibilities.