Personally, I never remove them. I don't need my fruit to be perfectly formed, some catfacing and bulging doesn't bother me a bit. I just trim off the inedible parts, big tomatoes have plenty of flesh left for a salad or fresh salsa. Now, if you are growing for farmer's market sales or plan to donate to a food pantry, that would be different as folks there look for a perfect handsome tomato.
It has been said that bees are more attracted to mega blossoms, so if you save unbagged seed you may have more chance of an unwanted cross. Of course bagging would solve that.
It's a personal choice, no right or wrong. I'm short season too, so I don't want to waste any time in getting a ripe tomato no matter what it looks like.
