good idea with the bushing
Get it out and be done with it. Center punch, small drill, bigger, bigger... use the bushing. I've used a piece of whatever I've had around in similar fashion. Once you get close heat it up, use penetrating oil, wax, heat it up... sometimes you can peel the remaining bits away from the threads.
Take it easy with the die grinder
