I also slow down and let them pass. It's easier on everyone and the 911 if you don't invoke an accident by slamming on your brakes and make them pay for being 100% wrong.
Times like that I wish I was driving a Hummer or Abrams Tank and then I'd definitely slam on my brakes for no apparent reason except to see that moron splattered on my rear bumper!