[RickM]

A smaller compressor will not run air intensive tools like an impact wrench or larger sander. Sand blasting will pretty much be out of the question.
A larger compressor will not empty the tank if appropriately sized. The motor will kick on and top off the tank as you work when a certain (lower) pressure is achieved.
A smaller compressor will work harder, will take a long time to fill it's tank and introduce a lot moe moisture into the system.

That said I've used a 1 hp / 13 gallon Sears compressor for 21 years. Still going strong but with it's limitations.