Please see http://status.cloudbees.com for status indicators and high level system status information.
For support, please visit support.cloudbees.com or email support@cloudbees.com

Monday 4 November 2013

MongoHQ password reset has completed

The MongoHQ password reset has completed and the MONGOHQ_URL entry for all resource definitions have been updated.  Applications with resource bindings declared for MongoHQ databases have been automatically restarted.

We recommend checking on any applications that are using MongoHQ databases to verify they are now working as expected.  If you are experiencing MongoHQ connectivity problems after this reset, please review the guidance offered in our previous blog post on the CloudBees status website.  If you are experiencing issues with a paid production application, please open a support ticket with the appropriate severity level set and the word “MongoHQ” in the ticket name.

We apologize for any impact the MongoHQ security breach has had, but we hope the coordination between CloudBees and MongoHQ has minimized the operational problem of updating passwords.

MongoHQ Password Reset is Underway

As communicated to MongoHQ subscribers via email,  MongoHQ is in the process of resetting the 'cloudbees' user password for all MongoHQ instances that have been created by CloudBees customers. CloudBees is working with MongoHQ to co-ordinate this reset process and minimize its impact on customers.  
  • If you have any applications running on CloudBees using resource bindings to connect to a MongoHQ database, we will automatically restart your application after the password is reset. No action is required on your part. These existing applications using the MongoHQ service will continue to work after the restart.
  • If you have any applications or external resources that are directly connecting to MongoHQ using the current ‘cloudbees’ username/password, these credentials will no longer work after the automated password reset. In this case, you should take steps now to ensure that your MongoHQ clients are using credentials that you have created using the MongoHQ admin->users page. Until you make changes to use the new credentials, existing applications making direct connections using the ‘cloudbees’ username/password will not work once the password reset has taken place.
  • If you have taken proactive steps already to remove the ‘cloudbees’ user and to adjust your application application code to initialize your MongoHQ clients with the new credentials that you have provisioned, you will not be impacted by the upcoming reset process for the ‘cloudbees’ user.

In summary, for CloudBees users of MongoHQ

  • If you take no action and are using CloudBees resource bindings to connect applications to MongoHQ, an upcoming reset of the ‘cloudbees’ password, followed by an application restart will take place to properly secure your database.
  • If you connect to MongoHQ directly or you want to take action before the upcoming password reset, you need to reset your passwords and modify your application logic accordingly.
  • If you connect to MongoHQ directly using the ‘cloudbees’ username (unlikely but possible), the upcoming password reset will require you to create a new username/password combination and modify your application logic accordingly.

We will post an update once the MongoHQ password reset process is completed.  If you have any questions, please open a CloudBees support ticket.

Additional resources:

Wednesday 7 August 2013

RUN@cloud routing outage resolved

There was an outage in the RUN@cloud shared routing layer that resulted in a 22 minute downtime window for apps from 01:24AM to 01:46 AM PDT where apps may have been inaccessible.  Applications running on dedicated routers were not affected.

The engineering team is reviewing this outage to isolate the root cause and identify ways to avoid this error in the future.  If you would like information about upgrading to a dedicated router, please review the Application SSL docs.

We apologize for any service disruption that you experienced because of this issue.

Saturday 15 June 2013

RUN@cloud Console and API outage [resolved]

The RUN@cloud Web console and API endpoints experienced a two hour outage tonight that was preventing the Web Console from rendering and prevented some API commands from working properly.  This issue has been fully resolved and running apps were not affected. 

The root cause of this error has been identified and the Engineering team is working on changes to prevent and reduce the impact/recovery-time of this specific error in the future. 

Our apologies for any service disruption you experienced while these endpoints were inaccessible.

Wednesday 29 May 2013

RUN@cloud shared revproxy intermittent errors resolved

Earlier today a network partition event occurred that caused some applications to intermittently return HTTP 502 errors when they should have been available - this was for some applications running on the "shared" revproxy/routing service (non SSL applications).

These issues are normally routed around and resolved before hitting applications and causing errors, but in this case we did not detect and resolve this in time.

This is now been resolved and we are looking at ways of preventing this.

Friday 1 March 2013

Shared database locking issue resolved

Issues with read-only connections on databases running on one of the RUN@cloud shared database clusters have been resolved.  The root cause analysis of this issue has determined that it was related to backups and additional controls for detecting and handling this issue are being put in place.