At first I thought it might be the column mounting bolts because there was so much play and it was more up/down then left/right. The bushing did eliminate the movement, so in my case it wasn't the mounting bolts.

However, I understand the mounting bolt problem can be serious because it can loosen to the point where the steering column just falls down altogether.