Skip to main content

Manage secrets

You can add, remove or rotate secrets as necessary.

This content explains:

  • how secrets are stored and used in Concourse
  • how to list secrets
  • how to add, remove and rotate secrets

Prerequisites

You must have production access and install the gds-cli tool to manage secrets.

How secrets are stored and used in Concourse

All of our secrets are used by our Concourse pipelines, and are stored as Concourse secrets.

Some secrets are used by the pipeline to perform its task. For example, the daily statistics task needs an OAuth token.

Some secrets are used by an app and are set by the app’s deployment task. For example, the GOV.UK Notify API key.

Secrets are referenced in the Concourse configuration using ((secret name)). For example with the daily statistics task, the report-bearer-token is a secret made available to the task through the BEARER_TOKEN environment variable:

- task: trigger-bigquery-export
  file: git-main/concourse/tasks/trigger-bigquery-export.yml
  params:
    CDN_DOMAIN: account.publishing.service.gov.uk
    BEARER_TOKEN: ((report-bearer-token))

List secrets

  1. Sign into the GOV.UK London PaaS by running the following in the gds-cli:

    cf login -a api.london.cloud.service.gov.uk --sso
    
  2. Sign into Concourse:

    fly login -t cd-govuk-tools -c https://cd.gds-reliability.engineering -n govuk-tools
    
  3. To list secrets with names and values used by the GOV.UK account manager app, run the following in the gds-cli:

    cf env govuk-account-manager
    
  4. To list the names of the secrets used by all GOV.UK tools, run the following:

    gds cd secrets ls cd-govuk-tools
    

Add or change a secret

  1. Sign into Concourse:

    fly login -t cd-govuk-tools -c https://cd.gds-reliability.engineering -n govuk-tools
    
  2. Add a new secret or change an existing secret by running the following:

    gds cd secrets add cd-govuk-tools govuk-account-manager-prototype/<secret-name> "<secret-value>"
    

Remove a secret

  1. Sign into the GOV.UK London PaaS by running the following in the gds-cli:

    cf login -a api.london.cloud.service.gov.uk --sso
    
  2. Sign into Concourse:

    fly login -t cd-govuk-tools -c https://cd.gds-reliability.engineering -n govuk-tools
    
  3. Remove a secret by running the following:

    gds cd secrets rm cd-govuk-tools govuk-account-manager-prototype/<secret-name>
    
  4. Unset the secret from the PaaS environment:

    cf unset-env govuk-account-manager <secret-name>
    

Rotate a secret

If a secret expires or is compromised, you will need to rotate it. This is a currently a manual process.

If a secret is compromised, this is a security incident. Follow the incident management process as well as rotating the secret.

  1. Sign into Concourse:

    fly login -t cd-govuk-tools -c https://cd.gds-reliability.engineering -n govuk-tools
    
  2. Generate a new secret value, which varies per secret.

  3. Set the new value using the gds-cli:

    gds cd secrets add cd-govuk-tools <pipeline-name>/<secret-name> "<secret-value>"
    
  4. If this is a secret used by an app, deploy the app.

Generate new secret values

Secrets used by multiple pipelines

grafana-api-key

Generate a new API key in Grafana, and deactivate the old API key.

paas-username and paas-password

Change the password by triggering a password reset, which gets sent to the govuk-accounts-developers Google Group.

sentry-dsn

Generate a new DSN from the Sentry project settings page, which you can reach by:

  1. Navigating to the GOV.UK Sentry organisation.
  2. Selecting the project from the drop-down list.
  3. Clicking the cog icon next to the project name.
slack_webhook_url

Tell IT Support, through the Web Helpdesk, that our Slack webhook has been leaked, and that we need a new one.

Secrets used by the account manager

basic-auth-username and basic-auth-password

These are arbitrary strings. After changing them, update the “deployed applications” Trello card.

bigquery-credentials-production

Generate new JSON credentials in the Google Cloud Console, and deactivate the old credentials.

notify-api-key-production and notify-api-key-staging

Generate a new API key from GOV.UK Notify, and deactivate the old API key.

oidc-pepper-production and oidc-pepper-staging

Generate an arbitrary string using rake secret.

This secret is used to generate subject identifiers, and so rotating it will cause services to see a new subject identifier for a returning user. This means that services will not be able to identify returning users, and will treat them as new users.

If this secret needs to be rotated, we will need to engage with other services, so that they can migrate their data.

oidc-signing-key-production and oidc-signing-key-staging

Generate a new RSA private key.

password-pepper-production and password-pepper-staging

Generate an arbitrary string using rake secret.

This secret is used to hash account passwords, and so rotating it will invalidate all of the current passwords. This means that users will have to reset their password to log in.

If this secret needs to be rotated on short notice, we may want to send an email to all users explaining the problem. If it can be rotated over a longer period, we could rehash passwords when users log in, and only disable the old value when most users have been changed. This is a feature which would need to be implemented.

report-bearer-token-production

Generate a new OAuth token in the Account Manager with the reporting_access scope, and expire the old token.

secret-key-base-production and secret-key-base-staging

Generate an arbitrary string with rake secret.

This secret is used to sign session cookies, and so rotating it will log out all users.

zendesk-api-username and zendesk-api-key

Tell User Support, in the #user-support Slack channel, that our Zendesk credentials have been leaked, and that we need new ones.

Secrets used by the attribute service

account-manager-token-production and account-manager-token-staging

Generate a new OAuth token in the Account Manager with the deanonymise_tokens scope, and expire the old token.

This page was last reviewed on 11 February 2021. It needs to be reviewed again on 11 August 2021 .
This page was set to be reviewed before 11 August 2021. This might mean the content is out of date.