When I'm in NYC, I always check out Colony Music, in Times Square. It's about a block away from the TKTS booth. It is pricey, but always has a great selection all types of music and amazing imports/out-of-print stuff.
Be prepared for some attitude at Colony though. The other place to try is Footlight Records on East 12th Street between 3rd & 4th Avenue. Be prepared for even MORE attitude there! Footlight has a lot more theme park and foreign stuff. You can occasionally find good stuff at Tower Records, but try the other 2 places first.
I would agree with both of those responses. Those were the 2 "biggies" I used to hit when I lived in NY. There were other, smaller places, mainly in the Village, but it's been 15 years or more since the last I looked and who knows if they're even still around.