Lee,
I've cut quite a bit of hardwood, including cherry and have not seen that bad of a problem. A number of factors could be hurting you but will need more information. 1st, cutting into your hold down bolts may have damaged your bit enough to cause this (trust me we have all done that) and we would need your feed rates ( x, y, z). This will help us to see what may be contributing to your problem. Of course Steve has a very good suggestion and I have read this several times on this forum and must admit I keep forgetting to follow it myself.
Joe
2005 PRT Alpha 48x96
2013 Colombo 3hp spindle
Indexer (converted lathe)
Aspire 9.0