How rigid is your setup? If you've got a tight machine, try carbide or cobalt. If it's one of the benchtop models with a bit of slop or backlash, stick with high speed steel drills. Not cheap hardware store drills, but actual HSS machine drills.
For carbide, run about 150 SFM. HSS and cobalt, about 50 SFM.
3.82 X SFM / drill diameter = RPM
For example, 1/8" carbide drill: 3.82 X 150 / .125 = 4584 RPM
Carbide and cobalt need constant feed, no pecking. You don't need to peck with HSS either, unless you get problems with chip clearance. If so, feed one drill diameter, clear chips, repeat.
If you've got power feed or a DRO, shoot for .001"-.002" per rev for carbide and cobalt, .002"-.003" per rev for HSS. This assuming, of course, small diameter drills. If the drill is .250" or larger, double the feed per rev.
Same 1/8" carbide drill for example: 4584 RPM X .002" = 9.168 inches per minute feed.
However, you've created another problem for yourself if the 3/32" drill is anywhere close to your finish size. Drills do not cut on the shank, only the drill point. If there's no material under the point, the shank will just grab and break. General rule, never pre-drill more than 1/2 of the finish diameter. IOW, unless the finish diameter is 3/16" or more, you risk breaking drills without removing any material. You may have limited yourself to endmills. If so, make sure you get a bottom cut, many small mills aren't.
In fact, I'd recommend a 5/32" two-flute endmill instead of your 4mm drill. Use the same feed and speed as the drill, depending on tool material.
As for coolant for small tools, high RPM and work hardened material, use kerosene. Keep a small container capable of supplying as steady a trickle as possible. A brush and cup would work, too, and be far preferrable to having a large container full of flammable liquid in proximity to a possible high heat source.