Please see http://status.cloudbees.com for status indicators and high level system status information.
For support, please visit support.cloudbees.com or email support@cloudbees.com

Friday 21 March 2014

API outage causing delayed masters and executors resolved

Earlier today there was an extended EC2 api outage - this had the effect of delaying the provisioning of masters (mainly for new accounts) - but also build services that were not using the new "mansion" based service. Service should be restored now but there may be a short backlog as everything recovers.

Resolved: Database service outage

A problem with our infrastructure provider this evening caused problems on slave-less CloudBees database clusters during the nightly database backup routine.  Customers with apps connecting to the affected databases were likely to experience database connection hangs and timeouts.  If apps are not coded with proper timeouts and retries, the apps may need to be restarted to re-establish connections to the database.

This problem did not affect customers who are paying for dedicated databases with slave configurations or customers using our recently added ClearDB multi-tenant database clusters which are configured with redundancy by default.

This was a problem that we have never encountered before and identification/recovery of the underlying problem was slower than we aim for.  We apologize for the downtime and are working to identify ways to avoid this specific problem and to improve our recovery time.

Tuesday 18 March 2014

Resolved: SauceLabs Downtime Affecting Some Jenkins Users

This issue is resolved.
---
SauceLabs is experiencing an extended outage which is affecting Jenkins users (CloudBees DEV@cloud or otherwise) who have installed the SauceLabs plugin. If users have configured the SauceLabs Badge column in their main view, any attempt to view the Jenkins index pages or jobs will hang. This is due a bug in the SauceLabs plugin

You can temporarily disable the SauceLabs plugin by navigating to <Jenkins URL>/pluginManager/installed and searching for the word "Sauce", uncheck the checkbox, and then press "Restart when no jobs are running" at the bottom of the page. After Jenkins restarts, you should be able to view your jobs, but you may lose Sauce-related configuration if you edit any of your jobs which contain such.

You can follow the SauceLabs Ops Team on twitter.

Wednesday 12 March 2014

Revproxy service problems resolved

Around 9PM UTC on the 11th March, there was a problem with the revproxy service that services most non SSL apps in the US region. This was resolved shortly after bit did result in some apps incorrectly returning 500 errors intermittently.