You made the right decision to throw away the collet.

If you still have a problem the bit is too small.