I’m wondering if inserting the compression plug slightly increases the OD of the steerer tube because I’ve had trouble fitting a 1.125 stem onto it. Anyone know if this is the case?

If so, is there a way to remove the compression plug when swapping stems?