Resolved — mysql50-2-master on ey01 experiencing heavy load

The load issue on mysql50-2-master is now resolved. The issue was caused by a bad query, we’ve addressed the query and are currently taking measures to assure this issue will not resurface.

mysql50-2-master on ey01 experiencing heavy load

mysql50-2-master on ey01 is experiencing heavy load at the moment and is causing issues with applications trying to access this database.  Our engineers are working to resolve the issue now.

ey00 node 0 resolved

The node is back online and all slices are booted.  At this time we do not have a clear cause for the failure.  When we have completed our analysis we will update accordingly.

ey00 node 0 update

The node is now back online.  Our cluster engineers are performing checks before bringing slices back online.

ey00 node 0 update

An engineer is enroute to the data center now. All customers with slices on this node remain online, as their slices on others nodes are live and fully functioning.  This means 0 site outages.

ey00 node 0

Node 0 on cluster ey00 appears to be down. We are investigating now.

Resolved: mysql50-4-master restart

We were able to recover the database without restarting it. ey01-s00009 is in good health and the DB is running normally now. 

mysql50-4-master restart

We need to restart the mysql database on ey01-s00009 (mysql50-4-master) due to a hung table. The DB is going down right now and we will recover it as quickly as possible. We will keep posting progress updates here.

resolved: cluster ey00, node 0

All slices are online and the node is function properly.

cluster ey00, node 00

The node is back online and we are bringing slices up now.