Wood movement from stress and humidity....not on a 3D finish pass. You'd have to stop the machine and wait between each pass to make it show that much variation. Did your holddown become slightly loose? You don't say what type/model of machine you're using, but if it is rack and pinion, check the grub screws on the Z axis. Make sure they're seated and tight and that the pinion is firmly seated into the rack. Check for any play in the Z and Y cars. Debris under a V roller on the axis that is rastering. Loss of steps in both directions as it jerks the Z up and then back down. If you have a standard SB, this is quite possible if you don't run proper ramping and feedrates.
Spindle or router? If router, a bearing going bad might be possible.
5 hours!? How big is this? A model like that, depending on the size, shouldn't take anywhere near that long. If it's in the range of 4x6 or 6x8, you should be able to knock that out in no more than 1 to 2 hours (with 8% - 10% stepover). Again, ramps and feedrates apply here as well.
Read up on it here.