Some queries were causing high load average on the mysql server. We killed the queries and database is back to normal. Sites are back up.
mysql50-5 on ey02 has a high load average. Sites are down. We are solving the problem and we will give you an update shortly.
At 23:00 PST we took down the ey02 mysql50-staging-1 server because of file system corruption.
There does not appear to be any data loss from the procedure and the database is now up and running.
We are now checking slices using the msyql50-staging-1 database.
Our system administrators detected a problem with hypervisor on Node 4 this afternoon. It was decided to perform an emergency reboot to prevent the issue from becoming more serious.
All slices should now be online. If you are continuing to experience an issue, please file a ticket with support.
All slices should now be online. We are manually checking them to verify operational condition.
We are continuing to work on bringing everything back on-line now. Individual slices are now starting.
The node has been rebooted. We are beginning to bring everything back up.
We will be restarting EY00 Node4 right now. Expected down time ~ 30 minutes.
There has been an issue with our Load Balancers on the ey00 cluster. Anyone who is on the LVS load balancer was affected.
The issue has since been addressed and all sites should be up and functioning normally.
