There are several airports that fit your category, although I'm not sure what's the deal now if you're within the boundaries of the New York Class B airspace (Controlled airspace). Linden, NJ (LDJ) would've been a possibility, it's a few miles south of Newark. Last I heard though, only way you could get in and out of there is via IFR flight plan, so that's out. I believe Teterboro kicked out all the piston airplanes to make room for jets, so I don't think there are any flight schools there anymore. So in New Jersey, your options might be Lincoln Park (N07), Caldwell (Essex Co) (CDW), Morristown (MMU), and Central Jersey (47N). Of all these, Central Jersey lies just outside boundary of the Class B, so I'm positive you can take lessons there. As far as Long Island goes, you have to go all the way to Republic (FRG) or Long Island MacArthur (ISP), but I think that would be too far for you if you're travelling from Manhattan.
Again, I don't know what the deal is now if you want to fly VFR underneath NY's Class B airspace. For the flight school phone numbers, go to www.airnav.com, do a search for airports, and put in the 3 letter identifiers for all these airports I mentioned. You'll be able to find all the data on those airports, including the phone number of their flight schools.
I'd say $2500 is around the minimum you need for the private, somewhere between $3000 and $4000 might be more realistic. It depends on what kind of plane you rent and how many hours it takes you to get the license. FAA regulations require a minimum of 20 hours of dual time with an instructor, plus 20 of solo time. Rental for a two seater goes for about $50-55 per hour. The instructor charges about $20-25 per hour, so you can do the numbers there. Try calling the different flight schools to check their prices.
If you have any other questions, don't hesitate to ask!