The node has successfully been rebooted, the faulty drive has been removed, and the node is functioning properly once again.
We will be beginning our scheduled maintenance of ey06-n02 in just a few moments.
All affected sites have been brought and verified online. If you see any further issues regarding this issue please contact us immediately. A postmortem will follow and we will contact affected customers where we will explain the issue with further details.
Again, we apologize for the inconvenience. Please reach out to us in our ticketing systems or IRC to follow up if you’ve been affected and have further inquiries.
All slices have been brought back online. If you still are encountering any issues, please lodge a ticket or contact us in irc://irc.freenode.net/#engineyard .
We are running final checks to ensure all affected sites are back online.
The complications have come up again, and interfering with a final resolution. Some slices are still down. All hands are still on deck trying to resolve this as soon as we can.
We will update the blog with more updates as soon as we know more and we apologize again for the inconvenience.
The ey02 maintenance is still in place. There were complications with the maintenance that led to a need to reboot many ey02 nodes. A full sysadmin team and a dba are online and have been working hard to get things back online.
The nodes are back up. Slices should be brought online within the next 15-20 minutes. The db masters will be hand-checked at startup, and all applications on the affected nodes will be hand-checked by our Application Support team. We apologize for the inconvenience and will update as soon as we know more.
The emergency shelf replacement on ey02 is still underway.
The I/O to the SAN shelf ey02-shelf0 has been halted while we swap the shelf for a new one. Once we have swapped the unit we will rebuild any degraded raid arrays. We will update one the shelf has been swapped and I/O resumed.
The node was rebooted without incident and all slices have been started, checked, and verified.
We will be beginning our reboot of ey06-n02 in just a few moments. Redundant application environments that have no affected databases should not experience any downtime.
