mysql50-3 downtime

Around 21:15 PST the mysql50-3 server was unavailable as the server ran out of disk space. The issue was resolved a couple of minutes later by removing a few days worth of binary logs (normally rotated out weekly) and it freed up about 44GB of space. This affected all customers on ey01 who are using this particular mysql server. We will schedule a maintenance for all clients using this mysql server so we can add more disk space and we’ll be reviewing our binary log rotation policy. We apologise for any inconvenience caused.Monitoring is being updated to monitor this more closely. 

Cluster 2 Recovery

As of 3:30am PST, cluster 2 has had all problems resolved.  A notification of what happened will be going out soon to notify customers of what exactly happened.  Thank you for being patient. A full report is forthcoming.

C2 Critical downtime at 1:30am PST

Around 1:30am PST, C2 suffered a critical clustering outage, and its machines are being rebooted. The exact nature of the problem isn’t known yet. Please watch here for updates. We apologize for the inconvenience.

Herakles Data Center routing update

From our data center

Subject: Alter.Net Service Update

Date: Tue, 20 Nov 2007 11:20:06 -0800

From: “Darren E. Canady”

Good Morning All,

The latest update from Verizon (alter.net) is that they’ve identified the problem hardware and have scheduled a maintenance for early tomorrow morning. This maintenance will replace a defective SONET card. The allotted maintenance window is from 12AM through 6AM Pacific, tomorrow, November 21, 2007.

Between now and then, Herakles will continue to leave the alter.net interface in a disabled state to prevent experiencing the recent, frequent intermittent service disruptions encountered when utilizing this path. We’re also reviewing options to enable the interface, yet have outbound customer traffic prefer to exit our other router so service points remain automated, versus manual.

We thank you once again for your patience, understanding, support and inputs and will update once again, as soon as any further updates are received.

Regards,

Darren E. Canady

Systems Engineer

Herakles, LLC

 

30 minute maintenance, Sunday November 18th, 12:05 AM PST

We will be performing a 30 minute maintenance on Sunday, November 18th, at 12:05 AM PST (GMT-8).During this maintenance we will be swapping out a SAN Shelf and replacing it with a new unit.A small number of customers will experience about 15 minutes of downtime during the swap.Please let us know if you have any questions. As always, we will keep any downtime to a minimum.

Engine Yard Status Blog

Welcome to the Engine Yard Status blog. We’ll post any important downtime, maintenance or other important notices here.