Skip to main content

Resolve issues with GOV.UK Account

If you are the on-call support developer, you may receive an alert if there is a problem with GOV.UK Account. The Senior Management Team (SMT) may also contact you if you need to scale up GOV.UK Account at short notice.

This content covers the reasons you may get called and what to do in each case.

Prerequisites

Before going on call, make sure you have access to everything you might need and know where our code is.

Get access to the GOV.UK PaaS

  1. Sign up to the London PaaS instance.

  2. Ask #govuk-accounts-tech to give you the space developer role in the “production” space for the “govuk-accounts” organisation.

  3. Set up the Cloud Foundry CLI.

  4. Check you can log in to the PaaS space:

   $ cf login -a api.london.cloud.service.gov.uk --sso

Repositories

The following sections contain information on our repositories.

GOV.UK Account Manager Prototype

Handles user interactions with accounts (e.g. logging in, signing up, brokering attribute store data, seeing “their dashboard”).

See the GOV.UK Account Manager Prototype GitHub repository for more information.

GOV.UK Attribute Service Prototype

A key-value store for user attributes. This prototype currently only stores users’ checker answers.

See the GOV.UK Attribute Service Prototype GitHub repository for more information.

GOV.UK Account Architecture

Contains information on how the system architecture works.

See the GOV.UK Account Architecture GitHub repository for more information.

GOV.UK Account Tech Docs

Contains our technical documentation.

See the GOV.UK Account Tech Docs GitHub repository for more information.

GOV.UK Docker

You can spin up a local development version of the account system, including the transition checker integration, with the latest GOV.UK Docker.

See the GOV.UK Docker GitHub repository for more information.

Diagnosing the problem

Try to determine which app is having issues and what the nature of the problem is:

  1. Check for govuk-account-manager and govuk-attribute-service errors in Sentry (also sent to #govuk-accounts-tech).

  2. Connect to the office VPN and check the metrics in Grafana.

The apps communicate, so a problem in one could cascade to the other.

Solving the problem

GOV.UK Account is still part of GOV.UK, so make sure to follow the GOV.UK developer documentation incident guidance, in addition to diagnosing and fixing any accounts-specific issue.

There’s a problem with the PaaS

If the problem is with the PaaS itself (for example, requests are not even making it to the app, despite the app being healthy), you can escalate the issue to the PaaS on-call support.

See the PaaS support page and rota for more information.

Someone is attacking us

See the section on security incidents in the GOV.UK developer documentation incident guidance for more information.

An app is unable to keep up with the volume of requests

If an app is receiving more requests than it can handle, you can scale that app up by:

  • changing the deployment pipeline
  • using the Cloud Foundry CLI

If you are adding more then 50GB of RAM then please let PaaS know (in #paas) as they may need to scale to accommodate our new usage.

See also the following continuous deployment documentation.

Change the deployment pipeline

You change the deployment pipeline by editing the concourse/pipeline.yml file.

See the example pull request for more information.

Using the Cloud Foundry CLI

Run the following commands:

$ cf scale -i <number of app instances> <app name>
$ cf scale --process worker -i <number of worker instances> <app name>

The <app name> is either govuk-account-manager or govuk-attribute-service.

Postgres or Redis is unable to keep up

We have one Redis instance, used by the Account Manager, and 2 Postgres instances, one for each app.

You can list them by running

$ cf services

If one of these needs to scale up, for example:

  • new Sidekiq jobs cannot be enqueued because Redis is out of memory
  • new users or attributes cannot be stored because Postgres is out of disk space

You can do so with

$ cf update-service <service name> -p <plan name>

To list all plans, run cf marketplace.

See the GOV.UK PaaS documentation on scaling Postgres and scaling Redis for more information.

There is a bug in the code

If there are a large number of Sentry errors, there may be a bug in the code which has suddenly reared its head (for example, maybe it only happens on certain pages, and a link to such a page has been tweeted).

If you can track down the bug and fix it, please do!

See also the following continuous deployment documentation.

Nuclear options

I need to suspend the Account Manager

See the documentation on suspending GOV.UK Account for more information.

Notes on continuous deployment

Our applications are continuously deployed via Concourse:

The pipelines both deploy code and also scale the app. This means that:

  • if you make a code change, that change will deploy automatically after being merged

  • if you scale the app by editing the pipeline, that change will automatically deploy after being merged

  • if you scale the app through the Cloud Foundry CLI, the app will scale back down when the next deployment happens.

You can disable automatic deployments by pausing the relevant pipeline. Select the pause button in the top-right corner (next to your username) of the pipeline. A paused pipeline turns blue.

This page was last reviewed on 10 December 2020. It needs to be reviewed again on 10 June 2021 .
This page was set to be reviewed before 10 June 2021. This might mean the content is out of date.