I would guess that it's your capture card or it's drivers.
I set up a system for my mother's office a while back that used the mini security cameras, which we hid inside light fixtures, air vents, etc. Unfortunately I don't remember the name of the software, but I know it had options for motion detection, brightness control, etc. It looks like there's a fair amount of
free software out there. Sorry I can't point you to something more specific.