I have about 40 PS2 games. I've beaten only a handful of them. (Example: right now, I'm playing through FF12 for the first time.) If you already have a PS2 and intend to keep it, you might as well forgo PS3 backwards compatibility and just keep the PS2. If you'd rather have everything in one console though, call around and see if the Gamestops or EBs near you have used PS3s with backwards compatibility. If their hard drive sizes aren't large enough, take the money you saved on buying used, go over to Best Buy or somewhere, and grab a 2.5'' hard drive for the PS3.
Used is a good way to go, but BB is likely not the best place to get a great deal on a 2.5" drive. Newegg and plenty of other places have better prices.