[Aerkuld]

I suppose the improbability of the whole situation comes down to a matter of scale. The half-way points would end up getting so ridiculously small that they would become meaningless to the man moving them so that the distance between him and the far wall would become negligable.

Look at the problem a different way to eliminate the scale problem.
Imagine the distance the man is attempting to travel is 8ft and the man's stride is 4ft. It should take him two paces to reach the far wall. But if he shrinks to half his size with each step forward and his step is reduced accordingly then he can never reach the wall, thus the journey is infinite (not allowing for the fact that he would eventually get to the point where he would need to reach a sub-molecular size). For the time in this instance you could say that he takes 1 second per step regardless of length. Now it's more obvious that the time is infinite.