For drilling only sometimes the best answer is to chuck up a drill bit and run spindle spin as slow as it can go and make your plunge max at .50"/sec.
There are drill bits made for plastic also that have a bit of the edge filed so they don't grab and pull the material as you go through.
You didn't say what you are using the ball nose for. If you broke it at those speeds you were probably cutting too deep although a slower spin is likely called for.
Second on using an o-flute for cutting plastic.