Before you spend a dime on any of this stuff you should read John Palmer's "How To Brew", IMO. I think there is enough of the book posted at the link below to get you started:
How to Brew - By John Palmer - Introduction
Get some basic equipment and get your process down. Once you have a feel for the recipes you like and the overall process you can upgrade your system, move to all grain, or build a full blown gas or electric brewery in your garage.
Try your local homebrew shop first. The owner will probably be a homebrewer and will more than likely provide a lot of support through your first few batches. If you don't have a decent local homebrew shop you will have to order something online.
I buy most of my supplies from the following:
Austin Homebrew Supply
Midwest Supplies - Homebrewing and Winemaking
Northern Brewer - Home Brewing Supplies and Winemaking Supplies
You are probably going to want to start with extracts and a kettle on your stovetop. A 5 gallon thin wall kettle for your stove will probably cost $30. Next, expect to spend $100-$150 on a decent 5 gallon starter equipment kit. The kit that will look something like the pic below. Your ingredients will run $25-$40 minimum per 5 gallon batch. You can buy ingredient kits for the style of beer you want to brew. You will also need to start saving bottles.
Have fun. It's a great hobby.