There has to be some interference issue with the gas key on the top of the BCG and the charging handle. The magazine follower cannot do anything except push up on the BCG, and the only part of the BCG that could touch the charging handle is the gas key (where it passes through the hole on the front of the charging handle).
The only way that the charging handle can be unlatched and forced to the rear is if the BCG is grabbing / snagging it and pushing it rearward with enough force to make the latch pop off of its catch on the upper receiver. The only part of the BCG that could be doing this are the gas key. The rest of the BCG ride under the charging handle and should not be touching it at all. Once the BCG has traveled in recoil far enough that the gas key has passed rearward through the hole in the front end of the charging handle, it cannot have any contact with it.
Whatever is happening to force the charging handle rearward must be taking place in the first couple of inches of BCG recoil. If this is what is happening, there should be some indication by way of bare metal or some area exhibiting unusual wear.
Check to be sure that the gas key is properly screwed down and staked to the BCG. It could be slightly loose or misaligned on the BCG. If it's really loose, I suppose it could be pushed up by the BCG into the charging handle when the mag follower catches the front of the BCG to hold it open. This should be really obvious, though.
It is remotely possible that the BCG is recoiling too far to the rear and the gas key is hitting the closed charging handle at its inside rear. There should be evidence of this inside the charging handle. This could possibly happen if the rifle is way over-gassed (way too light buffer and / or recoil spring). If this was the case, though, I should imagine that the problem wouldn't just happen on the last round and an empty mag. (I suppose this could also happen if the inside of the gas key channel in the charging handle wasn't machined properly.)
When the BCG is locked to the rear by the empty mag, can the charging handle be returned forward and latched? If there's some interference with the BCG to the rear, this should manifest itself when trying to return the charging handle forward. It shouldn't touch anything other than the channel and grooves inside the upper receiver.
Other than this, the latch on the charging handle must be bouncing off its latching notch on the receiver, and the only way that can happen is if the spring is too weak, the spring is missing, the latch "tooth" is misshapen (rounded over, not deep enough, or something) or the latch notch on the upper receiver is misshapen (again, rounded edge, too shallow, or something).
Finally, check that the charging handle is actually straight and not bent somehow. A bend in the handle might be enough to cause an interference with the movement of the gas key during recoil.
I'm sure you've looked to be sure the charging handle latch notch on the receiver isn't fouled with something to prevent the latch tooth from engaging it.