Gateway changes for EY06 completed without incident

Changes for EY06 underway.

The mysql process on mysql50-1-master in the ey04 cluster crashed and had to be restarted. We are investigating the cause of the incident and apologise for any inconvenience.

Gateway changes for EY06 completed without incident

Changes for EY06 underway.

A security vulnerability in all versions of nginx has been reported. Attackers can exploit this vulnerability by intercepting SSL sessions and compromising encryption key renegotiation via a known plaintext injection, allowing the attacker to read the plaintext of the SSL session. A patch has been released for this vulnerability.

In response, we have patched all versions of Nginx in our portage tree. It is our recommendation that all customers upgrade Nginx to our latest patched version (0.6.35-r22). Alternatively, we can re-install the current version of Nginx which will also apply the patch. The upgrade will only require a restart of Nginx, so no downtime will be involved. The upgrade is not expected to cause any behavioral changes, but we prefer to be cautious with all upgrades. That being the case, please follow these steps to upgrade:

1) Submit a new ticket with the subject “Nginx Security Upgrade”

2) We will upgrade staging first, prompt you for a review and approval to proceed with production.

Upgrades will be handled in the order that the requests are received.

Multiple raidsets rebuilding parity on ey04 during routine disk maintenance started consuming a significant portion of IO bandwidth. This caused elevated IO Wait and CPU load on environments on the affected disks. Engine Yard staff has since slowed the rate of the rebuilds to compensate and bring IO back to normal.

Gateway changes for EY06 completed without incident

Changes for EY06 underway.

The node has successfully been rebooted, the faulty drive has been removed, and the node is functioning properly once again.