RUN@cloud routing outage resolved

There was an outage in the RUN@cloud shared routing layer that resulted in a 22 minute downtime window for apps from 01:24AM to 01:46 AM PDT where apps may have been inaccessible.  Applications running on dedicated routers were not affected.

The engineering team is reviewing this outage to isolate the root cause and identify ways to avoid this error in the future.  If you would like information about upgrading to a dedicated router, please review the Application SSL docs.

We apologize for any service disruption that you experienced because of this issue.

RUN@cloud Console and API outage [resolved]

The RUN@cloud Web console and API endpoints experienced a two hour outage tonight that was preventing the Web Console from rendering and prevented some API commands from working properly.  This issue has been fully resolved and running apps were not affected. 

The root cause of this error has been identified and the Engineering team is working on changes to prevent and reduce the impact/recovery-time of this specific error in the future. 

Our apologies for any service disruption you experienced while these endpoints were inaccessible.

RUN@cloud shared revproxy intermittent errors resolved

Earlier today a network partition event occurred that caused some applications to intermittently return HTTP 502 errors when they should have been available - this was for some applications running on the "shared" revproxy/routing service (non SSL applications).

These issues are normally routed around and resolved before hitting applications and causing errors, but in this case we did not detect and resolve this in time.

This is now been resolved and we are looking at ways of preventing this.

Shared database locking issue resolved

Issues with read-only connections on databases running on one of the RUN@cloud shared database clusters have been resolved.  The root cause analysis of this issue has determined that it was related to backups and additional controls for detecting and handling this issue are being put in place.

DEV@cloud Upgraded to Fedora 17/x86_64

DEV@cloud build machines now run Fedora 17 by default. Support for 32 bit build machines has been deprecated. You can temporarily use the legacy labels if either of these cause you any problem.

Packet loss resolved

The network problems are once again going away - systems are returning to normal. We are monitoring things actively and will move/restart apps as needed. The backlog is cleared now, hopefully you didn't have any issues - and if you did - we apologise.

Packet loss update

Network issues are still intermittent - but we are working to remove impact on users apps and services as much as possible. Some applications are being restarted to migrate them to other locations, and backlogs/queues are being worked through. Thank you for your patience.