I looked at diagram of the lubrication system. If there was a foreign object loose in the cam housing, then it would either stay in the housing (and maybe fell out when you removed the valve covers as you said) or more likely it went into the oil return tubes then traveled to the bottom of the crankcase. It wouldn't fall out when you drained the oil because there is an oil strainer above the sump plate. The strainer would prevent it from getting sucked into the oil pump, because the inlet tube to the pump is below the strainer.
So I would think you should be concerned that the piece remains at the bottom of the crankcase. I have my engine out on an engine stand now with the sump plate off, and I would think, with the oil drained, that you could remove your sump plate and strainer and peek in there with a flashlight and mechanics mirror, and maybe fish it out with a magnet. Especially if it was a loose nut.
I just looked at the bottom of my engine and I am pretty sure you can. It seems like there are a lot of places at the bottom of the crankcase for it to hide, but you might get lucky.
Anyways, just a thought. Trying to be helpful. Good luck!
