I don' think they spawn in town. If they are spawning at the bridge that is normal for very full servers. The problem seems to be that they faze through the ground and buildings and then start floating up to the mansion. The same thing happens during Tortuga invasions. Although, I don't think the Governor's Mansion or the Faithful Bride start taking damage until the other barricades are destroyed, they just hang out in front. I could be wrong about that, I have not thoroughly tested that theory.
I believe you're right, the only exception would be Jolly Roger reaching the mansion or tavern, in which case the entire invasion will immediately end in failure since he seems to not be bound to barrier restrictions. So we do have upsides and downsides to these glitches.
Upsides:
1. Roger can sometimes get stuck and not continue up his usual path, which can sometimes cause another glitch that allows him to strike at people with an unlimited range. The upside is that he cannot reach his destination, which means you've already won.
2. Roger's Army will ignore barricades pretty commonly, but the best part is that if they get to their destination while other barricades still stand, they'll just stand outside their destination and do nothing except wait on you to destroy them. This is perfect for if you want to make the best use of every Roger Invaded server during his invasions. When done with one server, move to another where his men are patiently waiting for you to arrive.
Downsides:
1. When Roger gets all angry from being stuck off his beaten path and starts hitting people from unlimited distances, it can create the possibility that someone could get killed and miss the xp gotten from defeating him. This most likely only happens because someone pressed "Go to Jail" while they were downed, often because they thought they were killed by some kind of glitch and that it'd be better to leave and come back in order to fix it. Honestly, your best bet is to wait and hope they beat Roger before you get sent to Jail.
That seems like everything that I know of. There's a few other annoying things but they aren't even really worth mentioning, and aren't glitches that are specifically revolved around Invasions.
