Sounds good on the side of the finances ... of course one can drive on a track reasonably fast without spending that kind of dough. Take a lightly modified car and keep it to a few events a year. You'll still have a daily driver / weekend car when you aren't at the track.
You have your life in your own hands if it comes to lifestyle (food, exercise, vices). You do not have that if you ride a motorcycle: You put your life in the hands of the average driver. That's what I think is scary.
Coming on a gearhead board to ask if you should buy another combustion engine driven toy is like going to a cigar aficionado site to ask if you should quit smoking.
Whatever you decide - good luck!
George