If you want to maintain first-shot/cold-bore accuracy (hunting, etc.), then you probably want to leave the barrel somewhat fouled. If you zero it and then thoroughly clean it afterwards, you might be (unpleasantly) surprised the next time you take it out (when it counts). Might not be drastic, but could be enough to miss the shot. Some barrels may be more sensitive to that than others as well (stainless vs. chrome-lined, type of rifling/manufacturing process, etc.).
If you are just punching paper, then I wouldn't worry about it...Just clean the bore with more boolits. YMMV.
