OK – we’re talking about a holstered firearm while in a conscious state, not waking up in the middle of the night because your two year old is a nightwalker (I had one of those – he’s 21 now - but I digress).
When the (expletive deleted) hits the fan your training takes over on an unconscious level. In other words you do as you have trained. In my case the safety Always comes off when the gun comes out of the holster – it does so automatically and without thinking. Rule #3 (keep your finger off the trigger until the sights are on the target) also comes into play.
IMHO and according to at least some state CCW laws, you should never draw unless you have determined it’s your only recourse to save you life.
In this state “brandishing” a weapon is considered aggravated assault and the "assultee" can press charges. [Someone stopping you on the street to ask for directions might really just be lost.] Remember – God obviously loves stupid people, because he made so many.

Placing your hand on your gun (under your jacket -out of sight) might be a different story and could be the only thing needed to deter an attack. Plus, in this mode you have already negated most of the time/motion need to complete the draw. Ever get stopped for a traffic violation and notice the cop’s hand resting on his pistola as he walks up to your window? –Same thing.
Unconscious competence (and likewise unconscious incompetence) is taught in my club’s CCW class and is not original thinking on my part. BTW Massad Ayoob (in whom I have great respect – others on this board do not for some reason) has an article on same in the Nov/Dec issue of American Handgunner. It’s worth a read.
------------------
Whoever said you can’t buy happiness never owned a good 1911.