The OP said "a Bomar style" so I am assuming he is installing an adjustable rear sight that is a knockoff of the original Bomar.Sure it's Bomar?
I don't understand how removing material from the sight dovetail bottom,I would bet the STI unit is made by Kensight and just branded STI. You could measure the STI sight dovetail for comparison. If the new base dovetail is a tad wider, place a large mill or bastard file flat on a bench surface and draw the new sights base or dovetail flat along the file. Do this til the sight starts to fit in the slide dovetail. I like to get a sight to hand fit the dovetail maybe 1/3 rd the way, then with a leather or rubber mallet hit it home. On that style sight be sure, one ,that the set screw and windage screw are out. Two ,that the leaf tang is up and out of the way if your slide is cut with a "pocket" set up. Be sure you only hit the base and not the tang to prevent damage to the hinge Good luck.
That only works when the sight dovetail is deeper than the slide dovetail and there is gap between the bottom of the sight and the top of the slide. Removing material from the bottom of the sight MIGHT bring that top gap down and the sight may fit but what do you do if the sight dovetail and slide dove tail are the same depth or size? All you're doing is adding gap between the bottom of the sight dovetail and slide dovetail and the sight still won't fit.Really? Think about a triangle. As you go up from the bottom, the two other sides get closer together.