Please see http://status.cloudbees.com for status indicators and high level system status information.
For support, please visit support.cloudbees.com or email support@cloudbees.com

Tuesday 20 October 2015

DEV@cloud CA Certificate Issue - 21 October 2015

DEV@cloud CA Certificate Issue - 21 October 2015

Timeframe (UTC)

October 20 2015 4am - October 21 2015 2am

Impact

  • Jenkins master access to HTTPS services using command line tools would fail due to missing Root CA certificate chain

Root Cause

A component on the Jenkins masters instance was upgraded - however due to a failure in the package system, the Root CA certificate list (that lives on-disk in a ca-certificates.crt file) was no longer available.

As this file was missing, anything that relied on its existence was no longer able to access HTTPS protected services - this was typically limited to command line tools such as curl and git.

Resolution


The Root CA certificate list was reinstalled.

Data Loss / Security Implications

There are no data-loss or security implications.

Followup

  • We are improving the robustness of our testing and change control processes to help limit and subsequently eliminate failure of this nature in our upgrade process.
  • We are amending our status monitoring to detect this fault (our monitoring jobs all connect to Git over SSH - and hence did not fail under this scenario)