Drive Shelf Problems (ey01) Resolved

We discovered a locked drive shelf (number 10) which affected some customers.  We’re rebooting the slices that used that shelf now, a notification will be sent to all affected.

Drive Shelf Problem (ey01)

We’re currently experiencing problems with one shelf of drives on ey01, our engineers are looking into a fix as we speak.

Brief outage for ey00 customers

Some ey00 customers had a brief outage (3-5 minutes) due to transient routing problems. We are investigating and will post details as soon a possible.

Minor Upstream Routing Issue

Engine Yard received many outage notices starting at approximately 2:58am PST.  They continued for about four minutes, after which all sites were accessible.

Our internal monitoring gave no notices, only the external monitoring.  During the outage, our technicians had varying ability to access the sites based largely on geographical distribution.

It is our belief at this time that this interruption was a transient upstream routing issue.  We are following up with the data center, and an update will be posted here and in our subsequent e-mail notice.

Load balancer issue fixed

We’re back up and fully online.

Load balancer failure on ey00 (c1)

We’re resolving this issue now. More to come. 

Shelf 10 rebooted

We’ve remotely rebooted the shelf. It came up OK, is rebuilding parity so performance will be slightly impacted. Affected slices are now being rebooted.

Issue With Disk Shelf 10 On Cluster ey01

We’re experiencing some kind of issue with disk shelf 10 on cluster ey01.  We are in the process of having it serviced remotely.All disks and data appear to be fine.  It appears to be a soft issue.

ey01 Shelf 10 Outage

We believe this to be completely fixed.  Should the shelf manifest a similar problem, we have a spare on-hand to replace it with.  We will be watching it closely.  If you are experiencing further issues, please open a ticket or call (or both, if it’s urgent). 

All slices affected by shelf 10 now up

We’ve successfully brought all affected slices back online.We’re checking our site monitoring to verify that affected sites are also back online.