+1 to this. Call on many country gents here on a Saturday morning, and you'll find them in frayed cotton drills and a jumper with a couple of holes in it. You'd certainly find me in that. But I'd still change into a neatly pressed pair of cavalry twills to go out for lunch!I'm with you all the way.
Old, frayed Bills are great beach and boat pants.
But they get that way soon enough. I avoid the prewashed, preworn-looking ones too. I'll wear them casually 'til they fall apart, but I like my new ones looking new, crisp, and presentable.