After the previous deployment problem, we made a ton of upgrades to ensure deployments wouldn't take down the service for people. More redundancy. Blue-green deployments. More and better tests.
That had been working well... but rolling out this batch of changes caused a bunch of database connections to have invalid caches while a migration was applying. That broke auth with a cascade of 'cached plan must not change result type' errors.
Impact
Aqua was totally down for 10 to 15 minutes.
Fixes
No more yolo migrations. Deployment synchronization.
