These are the style bits that you should have used. Yes, a allen bit will fit...but doesn't work well...as you found out.
Sunex 9719 8 Piece 3/8" Drive Triple Square Bit Socket Set - Walmart.com
If you remove the two screws under the dash that hold the black plastic tube...it will allow you to move it out of the way enough to work on the upper bolt. I believe there is a threaded plate or nuts welded in place inside the door frame.
Brian.