Interestingly, the boxes from the movie Primer can be made to avoid that problem.
A short recap of how they work. You switch the box on, walk away from it to avoid running into past you, come back to the box several hours later, switch it off, climb inside, sit there for several hours, and climb out at the moment the box was switched on. One reason this model is cool is that it avoids one common problem with fictional time travel, the changing location of the Earth. You don't end up in interplanetary space because you travel back along the path of the box in spacetime.
So here's how you make the Primer boxes obey conservation of mass as well. The idea is that a box containing a time-reversed human should weigh less than an empty box. Let's say you weigh 70kg, and the box weighs 100kg when empty and switched off. When you switch the box on, a past version of you climbs out, and the box now weighs 30kg. Several hours later, a future you climbs in and the box now weighs 100kg, at which point the box switches off and sits there as empty as before.
At first I felt pretty smart for figuring this out because this whole issue never came up in the movie at all. And then I remembered the small detail that the boxes in the movie were an accidental invention, whose original purpose was to reduce the mass of objects. Wow.
That got me thinking about the other possible hole in the movie, namely all the abandoned timelines. Can this model of time travel be made to work correctly with not just spacetime paths and conservation of mass, but also causality and probabilistically branching timelines? For example, if you travel back in time and kill your past self, can that yield a unique consistent assignment of probabilities to timelines, where all time travelers "come from somewhere" and can't affect their "probability weight"? The result was this comment, for which I later found a proof of consistency which this margin is too small to contain ;-)
This post is a bit of entertainment for scientifically inclined Harry Potter fans.
Time turner from the Harry Potter series (and from the Eliezer Yudkowsky's venerable HPMoR fanfic) is a very useful device if you have some unfinished business in the recent past, like attending an extra class or saving a friend from a certain death. However, General Relativity has a few words to say about them, and they are not very flattering. I will only address one issue here: Energy conservation. TL;DR: if you use a time turner to vanish into the past, those around you will see you blown to tiny bits of Merlin-knows-what, quickly disappearing from view. When you appear in the past, this explosion appears in reverse.
Before we get to the time turners, however, let us consider an aside.
Let us start with a common question: if the Sun stop shining this instant, when would we notice? The common answer: it takes light 8.5 minutes to travel the distance of 150,000,000 km between the Sun and the Earth, so that's how long it will take. This glosses over the issue of what does "this instant" mean exactly at two different points in space, which is not so trivial given the relativity of simultaneity in Special Relativity. It is easily patched up, however, once we fix a global frame of reference. The Cosmic Microwave Background (CMB) is a natural one to use, and both the Earth and the Sun travel with a negligible fraction of the speed of light relative to the CMB. Anyway, the answer is still very close to 8.5 min.
Now another, deceptively similar question: if the Sun disappears this instant, how long before the Earth will stop orbiting the point where it used to be? The common answer: gravity travels with the speed of light, so also 8.5 min. This answer is obvious, simple and wrong. Yes, dead wrong. Why? because static gravity is not like light, it's more like electric field, only worse.
Let's first think of how you would make the Sun disappear. Maybe it turned into a black hole? Well, this would not really mean disappearance of gravity, the mass of the black hole will still be that of the Sun, and the Earth will happily (or unhappily, as the case may be) continue orbiting the Sun's corpse. So, in this case the answer is "it won't stop orbiting".
OK, so black hole was a bad example. How about a wormhole instead? You know, the evil Vogon-like aliens need to clear the room for a hyperspace bypass, and they build a wormhole from far away and suck all the matter in the Sun through it out of the way. What would happen then? There are a couple of hints: one is that from outside a wormhole is indistinguishable from a black hole, and the other is the Gauss Law. Both hints lead one to the same answer: just like with turning the Sun into a black hole, there is very little gravitational effect on the surrounding space. The rest of the now ex-Solar system will continue merrily on its way around the point where our Sun used to be.
An aside for those curious about the Gauss Law argument. The law in its integral form states that the flux of the gravitational field inward through any closed surface encompassing the Sun is proportional to the Sun's mass. To change the field, you need to remove some mass from inside this imaginary surface, by having it physically cross the surface. This last point may not be obvious, but it follows from General Relativity. Specifically, the Einstein's most misunderstood theory says that the spacetime curvature is determined by the (past and present) distribution of matter in spacetime. There are some exceptions, like the fixed-mass spherical objects, such as black holes and wormholes, which contains no matter, and gravitational radiation, which can carry away energy. But if you take a spherical object like the Sun and try to calculate what happens if you decrease its mass, General Relativity tells you that this mass has gone outward from the Sun in all directions in some form. It is not fussy about the form, as long as just the right amount of mass/energy has gone out.
Let me repeat for those who skipped the above paragraph: if you take the Sun and decrease its mass, the only way it can happen if this mass leaves the Sun outward and disappears into space. This happens all the time, of course, the Sun constantly loses its mass through radiation and solar wind, or in more drastic cases through Supernova explosions. Effects like this propagate no faster than light, of course. So they take forever to propagate all the way to infinity.
Now, back to the time turners. Hermione Granger might be but a small if incredibly studious girl, but she still has mass. If you were to peek at her using a time turner and disappear, her mass, small though it may be, still has to go some place, just like the disappearing Sun's mass had to go some place. The options are few: she can blow into tiny pieces flying past you, or disappear in a flash of brilliant light (and it takes a lot of light to carry away 50kg, what's with E=mc^2) . Basically, it will not be a pretty sight. What cannot happen is her simply vanishing, with no ill effects whatsoever. Well, it cannot happen if we are willing to keep Relativity around. Maybe we don't have to, what's with a certain deputy mistress turning into a cat and back, probably instantly changing her mass, with no ill effects on her or her surroundings. But if you give up on General Relativity, quite a few things will unravel, like all four Newton's laws.
Also don't forget the other side of the time turner action: Hermione appearing out of thin air just before walking into her extra class. The above process has to happen in reverse: an amount of matter equivalent to her mass has to travel inwards out of nowhere and coalesce into a person. Where did this matter come from? How did it form before collapsing into a person? How did it know that it would need to time its arrival into a certain point perfectly with whatever time turner will have been set to? That's some hard-core magic right there. Also, suck it, the Second law of Thermodynamics.
So, let me summarize: mass cannot just disappear, it has to spread out. mass cannot just appear, it has to coalesce. Thus time turners cannot be used inconspicuously, everyone around would be well aware of one's use, assuming they survive it. Actually, it probably cannot be used at all without breaking General Relativity and/or Thermodynamics. But hey, that's what magic is for.
EDIT: this post currently sits at -2 karma with 6 downvotes. I'd appreciate if any of the people who thought "I want less of this" explicate their logic to me, so I can do better next time.
EDIT2: OK, no one replied to my request... I'm guessing that some of you guys just quietly hate me :)