DEV@cloud CA Certificate Issue - 21 October 2015
Timeframe (UTC)
October 20 2015 4am - October 21 2015 2am
Impact
- Jenkins master access to HTTPS services using command line tools would fail due to missing Root CA certificate chain
Root Cause
A component on the Jenkins masters instance was upgraded - however due to a failure in the package system, the Root CA certificate list (that lives on-disk in a ca-certificates.crt file) was no longer available.
As this file was missing, anything that relied on its existence was no longer able to access HTTPS protected services - this was typically limited to command line tools such as curl and git.
Resolution
The Root CA certificate list was reinstalled.
Data Loss / Security Implications
There are no data-loss or security implications.
Followup
- We are improving the robustness of our testing and change control processes to help limit and subsequently eliminate failure of this nature in our upgrade process.
- We are amending our status monitoring to detect this fault (our monitoring jobs all connect to Git over SSH - and hence did not fail under this scenario)
