Looks like there are 5 bolts that attach to that bracket. 4 look structural, and a 5th looks to be an adjustment bolt for the pulley.
If it's one of the structural ones, I'm sure 3 out of the 4 wont do much harm if it's a pain to get it back in there
