I'm a cigar smoker too, and I've had the same thoughts.
As long as you're not drinking and smoking, it's possible you could be careful enough. One the powder is in the tube, about the worst that could happen would be to ignite one cartridge worth. When it's time, you can just put the cigar down, reload the powder hopper, pick it back up, what's the risk?
That said, I don't. It just seems like tempting fate to me to do it. It never fails that when you have every failure mode analyzed, another crops up to bite you that you didn't consider. I can just see some floating hot ash wandering over into the top of the powder can I forgot to put the lid on.
If you could do everything just right every time, no problem. One mistake though, and you are in deep weeds. It reminds me of a quote I read recently that said something to the effect of "manned space flight is not especially dangerous, but it is extremely unforgiving of human error".
Your original post didn't mention drinking, but the responses did bring the subject up. From painfull personal experience, alcohol and guns don't mix under any circumstances. I suppose the same applies to loading.