Most modern cars have some sort of idle control valve, or idle stabilizing valve, that allows a small amount of idle air to bypass the throttle plate. Before modern computer controlled cars, idle was controlled by having the throttle plate open a crack. Now, the throttle plate closes completely, and this idle valve, which is controlled by the computer, is what allows the idle air to get into the engine. I’m sure this engine must have one as well. It is common for these valves to get gummed up with carbon and get sticky, or stuck closed, or operate slowly. I have two cars in my garage that have this problem. The symptoms are exactly as you describe. The simple solution is to gain access to the valve and clean it out with carb cleaner or brake cleaner (no residue).
Look near your throttle body for some way for air to get past the throttle plate. You should be able to find the idle valve. It will have some wires attached, which the computer uses to open and close it. The whole thing, including the air inlet (which will typically take filtered air at a point past the mass air flow sensor, the valve, and the route past the throttle plate, might be embedded into the throttle body itself.
Good luck. You could also try this forum, or one of the other forums on this board, for help. That engine is used in a number of Toyota models, so maybe find the most active board they have and ask there.
http://www.yotatech.com/forumdisplay.php?f=28