What you really really want is a Leica Digilux 2. Or the cheaper Leica Digilux 1.
Well, probably not, unless money is unlimited.
You didn't specify how you'll use the camera, but I assume you want a camera that's small enough to carry around without a large camerabag.
I chose my point and shoot camera based on:
1: Small size 2:Use standard batteries 3:Use SD memory cards for price and compatibility. Plus sturdy built, fast and easy to use. And last but not least; price.
I have had a Casio QV-R4, and used it for a couple of years. I then sent this to my mom as I found a Casio QV-R61 for a very good price. Both models have the "telescoping lens" and I have had no problems with them. I think the design is good as the lens is completely sheltered from rubbing surfaces in a pocket when the camera is shut off.
I think the problem might be the quality of the brands you have tried. Vivitar is cheap, both in price and quality. The lower end Fuji cameras as well. So stepping up to a better brand might be the solution.
If you don't mind proprietary batteries, Canon PowerShot SD550 might be a great little camera, and the other small ones from Canon as well.
If you don't mind a little bulkier camera, I would suggest Canon PowerShot A620. It's got great reviews, have a fold out/swiwel LCD monitor, and uses ordinary batteries (read rechargeable NiMH). And the prices for the A620 have dropped a lot lately. (Approx. $270 shipped). Unfortunately, it's too big for most small pockets.
But, all the Casios and Canons have lenses retracting quite a bit, or completely. So maybe the Leica is your choice
A digital SLR would be a solution also. I'm seriously considering one. But it will not be my "always carry with me"-camera. I'm using my compact Casio for that.
Good luck with your decision, and it would be interesting to hear what you end up buying. I find
http://www.dpreview.com/reviews/specs.asp as a very useful source of info. Check it out if you haven't already.