There are some that will tell you: it is no big deal. There are others that will tell you: it will be a great catastrophe.
The problem is both the 40 and the 10mm headspace on the case mouth and that will produce excessive headspace with the 40 in a 10mm chambered firearm. You are relying on the extractor to hold the case tight enough against the breech face to ignite the primer and not let it back partially/completely out of the primer pocket causing leakage/flame cutting and damaging the breech face.
Personally I would grab another barrel and have it fitted to my custom gun. It is your gun do what you think you can get away with.
The extractor is not holding the 40SW case firmly against the breach, it is only preventing the round falling forward from gravity. That space between the claw and the rear of the extractor is the same, on both rounds the ideal position if for the rim to not touch the extractor at all. With the 40 in place it can be all the way forward resting on the claw. If the gun has a short firing pin, it might not fire at all. But when the firing happens, the case slams against the breach exactly the same and starts the reloading process.
The firing of the 40 SW in a 10mm poses no risk to the gun. However, it poses a risk in a carry gun. As I have explained above, the normal routine is for the round to be stripped from the magazine and the extractor slips over the rim. Normally not a problem. However, for the gun to function 100% this must occur 100% of the time. As I also explained, if you drop a round into the chamber, two bad things can happen. First, without the claw to hold it from falling into the barrel, it falls into the barrel. The firing pin cannot reach and it will not fire.
The second risk is that it doe get forward of the claw and does fire, but it will not extract because the claw is behind the rim. S
So, on a shooting range no problem. But if you have a jam, a limp wrist or something goes wrong with the firing. When it tries to eject and a second round is trying to enter the chamber, there can be a fumble factor. A fumble factor is an event when a 1911 fails, you tap the magazine, you eject the rounds in the chamber and you chamber another round.. Except, with 40 SW ammo in the gun, the bullets are shorter than the design. I have practiced this. It is possible for a round to get forward of the claw, and the gun will not fire.
My opinion is that this only happens when using 10mm magazines in the gun and not likely if you use proper 40 SW magazines. But, my experience is too limited to conclude that for certain.
So, firing 40 SW a 10mm or any purpose other than defense is fine, it does not harm the gun. But defense cannot risk that small chance of a failure. Carry 10mm in a 10mm if it is for real.
So, look at this picture of a perfectly tuned 10mm extractor and point out the flaws?
Bill Wilson has long said that the extractor should be tensioned, so that it holds the rim securely in place. That being a snug fit. Different from the picture above.
And an example from Bills Ut ube on tuning extractors.
Wilson teaches that the extractor should hold an empty case in place. Now try that with both the 40 SW and the 10mm, guess what? Samo, samo. The tuned extractor holds them both.
However, others feel the rim should sit midway in the gap between the claw and the rear of the extractor base. However, we know that loose fitting extractors usually run just fine.
So, can anyone say that the way the extractor fits over the 10mm and 40SW will be different just because one is shorter or the other longer? The headspace keeps a round from falling forward into the barrel, the case mouth stops the case at one point.
There are lots of theories of what should be and what not should be. Shooting 22 short in a 22 long rifle sure has detractors. But other than that little carbon ring you remove when cleaning it hurts nothing. My take based on personal experience is that shooting 40 SW in a 10mm barrel is about the same. I can buy 40 SW for about $20 box, 10mm is about 25% more. So, for range use, I see zero problem, but as I explained above, NOT FOR DEFENSE.
Shooting the caliber marked on the barrel is good advice.
TO the OP. Whether a custom gun or not, it does not matter whatsoever. The issues are the same with any 1911 regardless of whether itis engraved or gold plated or not. They are just mechanical pieces, all about the same. A $10 extractor properly tuned will function with either the 40 or 10 as one that costs $50. There is no mechanical or practical difference. A titanium piece might last longer, but mechanically they are the same for the life of the product.
So, go shoot a box and let us know what you think.