I agree entirely. This has been one of my concerns. I have a collection of US military uniform jackets, as well as field jackets like the M-41, M-53 and M-65. I am extremely careful about wearing uniform items since I was never part of the US Forces.Having never served, but having great respect for people who do and have (like @eagle2250), I would never wear something that made it appear I had served. There is a line as many military styles have made it into the mainstream: I have no issue wearing a military-like item as long as it's clear it's just a civilian variation. I avoid anything that gives the impression I served; so fake insignias, etc. are out for me.
The M-series field jackets, without any campaign ribbons, decorations, or other insignia, are OK for civilians to wear, as far as I know -- so I wear them with respect. If there is a regimental patch left on, I would personally hesitate to wear it in public, although I have seen such jackets worn by civilians many times (of course, they could have been former members of the regiment, although I rather doubt it). None of my M-series jackets have such patches, although one does have a dark area where there used to be a name patch that was removed.
As for the uniform jackets that are part of standard military dress uniforms (the kind worn with a jacket or tie, or the closed-collar blue Marine dress jackets), I admire them and keep them well-cared for on hangers, but do not wear them in public. Even devoid of any insignia, patches, ribbons or other such items, I feel hesitant about wearing them.
The two exceptions are the surplus US Navy issue pea coat I have long had, and the USAF staff sergeant's coat I acquired last week These have no decorations or insignia, although the brass buttons have the US Navy or USAF emblems on them. I do wear the pea coat in public, and I have had no negative comments about this, even from friends who actually served in the US Navy. I imagine it would be similar when I wear the USAF coat.
I did do National Service in India (a paramilitary outfit called the NCC, a feeder unit for the Officer Corps of the Indian Military, somewhat like our ROTC), so I would perhaps be a bit more entitled to wear one of the NCC uniform items -- but I never kept any of those, they were from a long time ago. Because my instructors were Gurkha regiment officers and NCOs, I do wear a Gurkha Regiment tie on occasion with civilian attire to show my respect and admiration for the Gurkhas, who are among the finest soldiers in the world. I don't think they would object, LOL.