Ideally you should also point a room fan at the inside of the computer after you take off a panel. Some systems, especially small form factor PCs and rack mount servers, actually need the case in order to be properly cooled. Removing a panel means that e.g. an exhaust fan no longer forces air across passively cooled components near the intake.