ey00 postgresql upgrade

We have finished the postgresql upgrade on ey00.  The new server is now running 8.2.7 and has considerably more RAM.  We have brought all customers back online and are verifying site operation.

ey00 postgresql

We will be beginning the process of upgrading postgres on ey00 shortly.  Affected customers should have already been notified.

ey01 mysql50-7 resolved (update)

The mysql50-7 server had OOM issues and had to be manually restarted. We are working to increase the memory for this slice so that we can avoid this happening in the future.

ey01 mysql50-7 resolved

mysql50-7 server on ey01 was down briefly today from 04:31 to 04:41 PST. We restarted it and everything seems to be working smoothly now.

ey01 node 6 resolved

Node 6, and all affected slices including mysql50-5-master are back online and confirmed working.  All affected customers will receive a refund for downtime as well as an email notification. Please file a ticket for any outstanding issues.

ey01 node 6 update

Node 6 is back online and all slices and databases are functioning. We will post a final update once all database tables have been checked.

ey01 node 6 update

Node 6 is back online.  Database masters are being checked and slices are being powered online now.

ey01 node 6 update

Our engineers have identified the failed hardware for node 6 is likely the power supply or other power related component. If swapping the power supply does not yield a working node, we will proceed with a full hardware swap.

ey01 node 6 update

On site engineers have confirmed the hardware needs to be replaced and are in the process of doing so now.

ey01 node 6 update

Waiting on update from on site engineers.  Will post again once we have additional information.