The database is restored to normal operation. Now verifying that all application are properly connecting. We will complete this shortly and report back.
Mysql50-8-master on cluster EY02 appears to be down. We are responding immediately.
Customers using shared MySQL database server mysql50-8-master on our ey02 cluster experienced a brief outage at around 1500 PDT when the mysqld instance on that server segfaulted. We were notified and brought the database server back online, however we do not have a root cause determined as yet.
Customers may have experienced a brief unavailability of their GFS partition while we performed a necessary and emergency reboot of a SAN device on cluster ey04. The affected slices were rebooted after the SANs device was restarted. All slices have been verified as up and working. This is now being marked as resolved. Total outage time lasted around 18 minutes.
The MySQL database mysql50-3-master on EY01 is up and running as normal. If you are on cluster EY01 and are continuing to experience any issues with mysql50-3-master database, please alert us immediately by IRC, ticket or phone call.
The MySQL process on mysql50-3-master has come back up and DBAs are checking databases for corruption.
mysql50-3-master has come back up and the MySQL process has started and is recovering.
The DBAs have managed to safely shutdown the MySQL process and are preparing to perform a reboot of the server. Once brought up all databases will be checked for corruption.
The recovery of mysql50-3-master is ongoing, though slower than expected. DBAs are actively working to speed up the recovery.
DBAs are still actively working on the recovery of mysql50-3-master.
