The main Maven repository server which is used by DEV@cloud builds went offline earlier today. This may have resulted in hung or failed builds.
We have restarted the server which recovered after the reboot.
Tuesday, 21 July 2015
Monday, 15 June 2015
Rollback of 1.609.1 Upgrade
As we communicated on Friday, we had planned to rollout a release of Jenkins LTS 1.609.1 to DEV@cloud today. In the course of deploying this, we found a severe regression in the core Jenkins LTS: a deadlock which occurs under high load. Since it will take us some time to verify the fix, we have rolled back the release.
Most instances were not upgraded to 1.609.1. We have automatically downgraded any upgraded instances back to 1.580.3.4 which does not exhibit this issue.
We apologize for any disruption to your business and will be in touch with further details on the release when ready.
Sincerely,
The DEV@cloud Team
Thursday, 14 May 2015
Nexus Upgrade
CloudBees' Nexus instance http://repo.cloudbees.com has been upgraded from 1.9.1 to 1.9.2.4.
This should help with intermittent Error 500 issues that were being experienced by a small number of customers.
In addition to the version bump we have also internally moved from custom Tomcat hosting to the Sonatype preferred Nexus hosting model using Jetty.
While this should not be visible externally (or cause issues in your builds), any unexpected behaviour should be reported to our Support Team.
Failed builds should be retried.
This should help with intermittent Error 500 issues that were being experienced by a small number of customers.
In addition to the version bump we have also internally moved from custom Tomcat hosting to the Sonatype preferred Nexus hosting model using Jetty.
While this should not be visible externally (or cause issues in your builds), any unexpected behaviour should be reported to our Support Team.
Failed builds should be retried.
Tuesday, 5 May 2015
Slave Provisioning Outage
We have just recovered from an outage in the DEV@cloud slave provisioning service. This outage was due to a problem with a backend authentication API. This outage also affected the CloudBees Console.
Wednesday, 8 April 2015
Maven Repository Server (repo.cloudbees.com) Outage
The main Maven repository server which is used by DEV@cloud builds went offline earlier today. This may have resulted in hung or failed builds.
We have restarted the server and are adding additional monitoring to detect this problem in the future.
We have restarted the server and are adding additional monitoring to detect this problem in the future.
Friday, 6 March 2015
AWS Maintenance Event Affecting DEV@cloud
This issue is resolved.
---
CloudBees DEV@cloud Jenkins may be intermittently affected by an ongoing maintenance event within Amazon Web Services, which provides the hosting for DEV@cloud. While Amazon has tried to limit the impact of this event, we are seeing issues with internal network routing between some instances.
We are working to move Jenkins instances off of these affected EC2 hosts. For the minority of Jenkins instances affected, this will result in a few minutes of downtime during the migration. We apologize in advance for any inconvenience. We are working with Amazon to better understand how to avoid this in the future.
---
CloudBees DEV@cloud Jenkins may be intermittently affected by an ongoing maintenance event within Amazon Web Services, which provides the hosting for DEV@cloud. While Amazon has tried to limit the impact of this event, we are seeing issues with internal network routing between some instances.
We are working to move Jenkins instances off of these affected EC2 hosts. For the minority of Jenkins instances affected, this will result in a few minutes of downtime during the migration. We apologize in advance for any inconvenience. We are working with Amazon to better understand how to avoid this in the future.
Wednesday, 4 February 2015
Sonar service outage
In order to provide a more reliable database backend for our Sonar services, we migrated the Sonar database to a new database server. However the data-migration failed for some Sonar instances.
Sonar instances impacted by this issue would be unable to connect to the database and thus unavailable during the outage.
To maintain the integrity of affected Sonar instances we have successfully re-migrated the impacted databases.
Upon restart of Sonar, some Sonar instances were in an inconsistent state having had plugin updates via the UI which were incompatible with the version of Sonar currently deployed at CloudBees. Our engineers have completed correctly upgrading impacted Sonar instances.
The Sonar service is now back online.
Sonar instances impacted by this issue would be unable to connect to the database and thus unavailable during the outage.
To maintain the integrity of affected Sonar instances we have successfully re-migrated the impacted databases.
Upon restart of Sonar, some Sonar instances were in an inconsistent state having had plugin updates via the UI which were incompatible with the version of Sonar currently deployed at CloudBees. Our engineers have completed correctly upgrading impacted Sonar instances.
The Sonar service is now back online.
Subscribe to:
Posts (Atom)