Obviously you won't find the difference in switch (part # is the same). It is the rest of the wiring in the car, the mirror motors, and the way they are controlled. As tough as the diagram below is to see, a quick compare to the '84 diagram above should help.
I wired '84/'85 mirrors on my '81 to work the mirrors left/right, but not up/down. I spent hours testing them with a meter thinking I had transposed the harnesses incorrectly until I finally started comparing wiring diagrams.
