All impacted branches are recovered.
PlanetScale upgraded development Vitess branches with Vectors enabled from MySQL 8.0.x to 8.4.x. PlanetScale runs an internal fork containing support for PlanetScale's MySQL Vectors indexes.
A bug in the 8.4.x version of our Vectors builds resulted in the upgrade immediately failing on startup.
Because Vitess development branches only have one instance - the primary - impacted development branches did not have a primary during the impact window.
The incident was resolved by reverting impacted branches to their previous MySQL version.
PlanetScale will fix the bug in the MySQL Vectors 8.4.x build before re-attempting the upgrade at a later date.
Resolved
All impacted branches are recovered.
PlanetScale upgraded development Vitess branches with Vectors enabled from MySQL 8.0.x to 8.4.x. PlanetScale runs an internal fork containing support for PlanetScale's MySQL Vectors indexes.
A bug in the 8.4.x version of our Vectors builds resulted in the upgrade immediately failing on startup.
Because Vitess development branches only have one instance - the primary - impacted development branches did not have a primary during the impact window.
The incident was resolved by reverting impacted branches to their previous MySQL version.
PlanetScale will fix the bug in the MySQL Vectors 8.4.x build before re-attempting the upgrade at a later date.
Monitoring
Branches are recovering and we are monitoring the situation.
Identified
We have identified the issue and are rolling impacted development branches from 8.4 back to 8.0.
Investigating
PlanetScale initiated a system wide upgrade of branches with Vectors enabled and running on MySQL 8.0 to MySQL 8.4. A number of development branches were disrupted and have not yet recovered. We are investigating and remediating.
