Engine Yard Status http://engineyard.wordpress.com Maintenance and downtime status updates in real time Sat, 10 May 2008 23:47:20 +0000 http://wordpress.org/?v=MU en ey02 mysql50-5 resolved http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5-resolved/ http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5-resolved/#comments Sat, 10 May 2008 23:47:20 +0000 engineyard http://engineyard.wordpress.com/?p=190

Some queries were causing high load average on the mysql server. We killed the queries and database is back to normal. Sites are back up.

]]> http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5-resolved/feed/ ey02 mysql50-5 http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5/ http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5/#comments Sat, 10 May 2008 23:33:43 +0000 engineyard http://engineyard.wordpress.com/?p=189

mysql50-5 on ey02 has a high load average. Sites are down. We are solving the problem and we will give you an update shortly.

]]> http://engineyard.wordpress.com/2008/05/10/ey02-mysql50-5/feed/ ey02 mysql50-staging-1 Emergency Maintenance http://engineyard.wordpress.com/2008/05/05/ey02-mysql50-staging-1-emergency-maintenance/ http://engineyard.wordpress.com/2008/05/05/ey02-mysql50-staging-1-emergency-maintenance/#comments Mon, 05 May 2008 07:07:52 +0000 engineyard http://engineyard.wordpress.com/?p=188

At 23:00 PST we took down the ey02 mysql50-staging-1 server because of file system corruption.

There does not appear to be any data loss from the procedure and the database is now up and running.

We are now checking slices using the msyql50-staging-1 database.

]]> http://engineyard.wordpress.com/2008/05/05/ey02-mysql50-staging-1-emergency-maintenance/feed/ EY00 Node 4 - Resolved http://engineyard.wordpress.com/2008/05/02/ey00-node-4-resolved/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-resolved/#comments Fri, 02 May 2008 21:18:54 +0000 engineyard http://engineyard.wordpress.com/?p=187

Our system administrators detected a problem with hypervisor on Node 4 this afternoon.  It was decided to perform an emergency reboot to prevent the issue from becoming more serious.  

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-resolved/feed/ EY00 Node 4: Update http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-4/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-4/#comments Fri, 02 May 2008 20:56:55 +0000 engineyard http://engineyard.wordpress.com/?p=186

All slices should now be online.  If you are continuing to experience an issue, please file a ticket with support.

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-4/feed/ EY00 Node 4: Update http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-3/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-3/#comments Fri, 02 May 2008 20:42:42 +0000 engineyard http://engineyard.wordpress.com/?p=185

All slices should now be online.  We are manually checking them to verify operational condition.  

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-3/feed/ EY00 Node 4: Update http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-2/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-2/#comments Fri, 02 May 2008 20:26:51 +0000 engineyard http://engineyard.wordpress.com/?p=184

We are continuing to work on bringing everything back on-line now.  Individual slices are now starting.

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update-2/feed/ EY00 Node 4: Update http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update/#comments Fri, 02 May 2008 20:08:17 +0000 engineyard http://engineyard.wordpress.com/?p=183

The node has been rebooted.  We are beginning to bring everything back up.

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-update/feed/ EY00 Node 4: Going down for reboot http://engineyard.wordpress.com/2008/05/02/ey00-node-4-going-down-for-reboot/ http://engineyard.wordpress.com/2008/05/02/ey00-node-4-going-down-for-reboot/#comments Fri, 02 May 2008 19:53:10 +0000 engineyard http://engineyard.wordpress.com/?p=182

We will be restarting EY00 Node4 right now.  Expected down time ~ 30 minutes.

]]> http://engineyard.wordpress.com/2008/05/02/ey00-node-4-going-down-for-reboot/feed/ ey00 Cluster Is Back http://engineyard.wordpress.com/2008/05/01/ey00-cluster-is-back/ http://engineyard.wordpress.com/2008/05/01/ey00-cluster-is-back/#comments Fri, 02 May 2008 05:52:02 +0000 engineyard http://engineyard.wordpress.com/?p=181

There has been an issue with our Load Balancers on the ey00 cluster.  Anyone who is on the LVS load balancer was affected.  

The issue has since been addressed and all sites should be up and functioning normally.

]]> http://engineyard.wordpress.com/2008/05/01/ey00-cluster-is-back/feed/