Last spring, we were seriously considering buying a place on/near the lake as purely a rental/investment, (we would never use it or have emotional attachment to it). We zero'd in on a water-access ranch that was listed for just under $400K. It had a rental history of somewhere in the $1200-1500/week during the summer, if I remember correctly. It was kinda a dump and needed a fair amount of work, although I distinctly remember the realtor saying to the effect of, "renters don't care...don't waste your money sprucing it up..."
Bottom line is that even at 100% rented from say the end of June till early Sept, we would still be just shy of covering the mortgage, taxes, insurance, and other misc costs, (and that would've been with $100K down payment). Perhaps there'd be some rental money in the off-season, but in terms of our analysis, I did not count on that coming in. Perhaps as well, there'd be the increase in the value of the property over time, but in the end, I figured that I would keep my $100K in a portfolio of stocks and that would yield a better return with a lot less headache of dealing with renters and another property.
We walked away without making an offer, but amazingly, the place sold within a week of being listed at full list price!
I think it's a totally different story, however, if you are buying a place to be your family vaca place/2nd home that you just happen to rent out part of the summer to help with the expenses. In this scenario, maybe it's worth not covering the costs via rental, because you are making memories and most likely keeping the place long-term.