The slight unevenness of cam and rocker surfaces is the reason often given for keeping track of the rockers when doing a rebuild which does not involve refacing cams or rockers (and that is a very common rebuild), so that the same rocker can be put back on the same cam lobe.
Since hearing of this rationale, I have gone back to keeping track of the rockers, but I can't say I ever saw any bad results from the years where I just put them back in any old which way. What I looked for was pitting suggesting the hardening was wearing through, or something approaching a ridge at either end of the wear surface of the rocker, or some seriously uneven looking wear.
Me, I wouldn't worry about what you have. Chances are good that the wear process will continue to even things out as the two surfaces wear in together.
My suspicion is that short of a lack of oiling (which will quickly lead to machining the cam lobe down to its base circle), the motor will put up with quite a lot. Not that mismatches are good, but just that there is a tolerance.
Of course, you can spend more $ to get machine shop level matches - if the cam itself was ground exactly true. If the rocker grinding setup is a little off, it won't be true. If the hone for the bushing is a bit off, ditto. And there seem to be ways to use the hone to compensate for the surface if your bushings were sufficiently undersize in the ID to start with.
I don't think the shaft's wear has a lot do do with this, though. Most of us home brew guys just reinstall the shafts so the least worn part takes the load, and that part is the one which is the most true.
Here are a couple of pictures I have saved from this forum. First, a machinist checking rockers for true faces - one is, one isn't. Then pictures (from the same source) showing a precise method of grinding the faces. Then a home brew setup which doesn't look quite all that trustworthy, though if the feeble bracketry were replaced with something better it might be all that is needed. There is a U-tube video out there of a machinist grinding faces - what is needed is something which allows the pivot to be true while it is slid in and out as the grinding happens. Naturally the stone's surface also needs to be true, which means something like the diamond tipped rods used on valve grinding machines, and that is going to be difficult for those of us with inexpensive bench grinders.
