Rotating secrets

The secrets that are used by the deployment might expire or otherwise need to be rotated. The following information describes how to rotate the secrets in a containerized environment.

The process for rotating secrets is as follows:

• Make a copy of the existing secrets.
• Generate new secrets.
• Update the secrets in the running containers.
• Update the secrets in the environment variables and re-run the containers.

Procedure

1. Make a copy of the existing secrets directory or secrets.
2. Generate new secrets by running secrets-utility.
3. Remove the existing containers but keep all volumes.

Solr and ZooKeeper

1. Run the Solr and ZooKeeper containers with the previous secrets that you made a copy of. To do this you can update the docker run command to specify the backup directory for the secrets to mount or update the values in the secret environment variables.

   For example, the docker run command for ZooKeeper might look like:

   docker run --restart always -d \
     --name "zk1" \
     --net "eia" \
     --net-alias "zk1.eia" \
     -p "8080:8080" \
     -p "2181:2181" \
     -p "2281:2281" \
     -p "3888:3888" \
     -p "2888:2888" \
     -v "zk1_data:/data" \
     -v "zk1_datalog:/datalog" \
     -v "zk1_logs:/logs" \
     -v "/copy-of-secrets/simulated-secret-store/zk1:/run/secrets" \
     -e "ZOO_SERVERS=server.1=zk1.eia:2888:3888 server.2=zk2.eia:2888:3888 server.3=zk3.eia:2888:3888" \
     -e "ZOO_MY_ID=1" \
     -e "ZOO_SECURE_CLIENT_PORT=2281" \
     -e "ZOO_CLIENT_PORT=2181" \
     -e "ZOO_4LW_COMMANDS_WHITELIST=ruok, mntr, conf" \
     -e "SERVER_SSL=true" \
     -e "SSL_PRIVATE_KEY_FILE=/run/secrets/server.key" \
     -e "SSL_CERTIFICATE_FILE=/run/secrets/server.cer" \
     -e "SSL_CA_CERTIFICATE_FILE=/run/secrets/CA.cer" \
     "i2eng/i2eng-zookeeper:3.9"
   

   Where the -v "/copy-of-secrets/simulated-secret-store/zk1:/run/secrets" volume mount is the location of the backed up secrets.

   For more information about running the containers, see Solr and ZooKeeper.

2. Upload the new security.json file to Solr by using the run_solr_client_command function. For example:

   run_solr_client_command bash -c "echo \"\${SECURITY_JSON}\" > /tmp/security.json && solr zk cp /tmp/security.json zk:/security.json -z ${ZK_HOST}"
   

3. Remove the Solr container.

4. Update the ZooKeeper password by using the change_zk_password function. The function takes the old password and the new password as arguments. For example:

   change_zk_password "old-password" "new-password"
   

5. Run the Solr container with the new secrets. For more information about running the Solr container, see Solr.

Connectors

If you have any connectors, run the connector containers again with the new secrets.

Prometheus and Grafana

1. Run the Prometheus and Grafana containers with the previous secrets that you made a copy of. To do this you can update the docker run command to specify the backup directory for the secrets to mount or update the values in the secret environment variables.

   For example, the docker run command for Prometheus might look like:

   docker run -d \
   --name "prometheus" \
   --net "eia" \
   --net-alias "prometheus.eia" \
   -p "9090:9090" \
   -v "/home/<user-name>/analyze-deployment-tooling/examples/pre-prod/prometheus/web-config.yml:/etc/prometheus/web-config.yml" \
   -v "/home/<user-name>/analyze-deployment-tooling/examples/pre-prod/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml" \
   -v "prometheus_data:/prometheus" \
   -v "/copy-of-secrets/simulated-secret-store/prometheus:/run/secrets" \
   "i2group/i2eng-prometheus:2.53"
   

   Where the -v "/copy-of-secrets/simulated-secret-store/prometheus:/run/secrets" volume mount is the location of the backed up secrets.

   For more information about running the containers, see Prometheus and Grafana.

2. Update the Grafana password by running the following command:

   docker exec "${GRAFANA_CONTAINER_NAME}" bash -c "grafana-cli admin reset-admin-password ${new_grafana_password}"
   

3. Remove the Prometheus and Grafana containers.

4. Run the Prometheus and Grafana containers with the new secrets. For more information about running the containers, see Prometheus and Grafana.

Database

If your deployment pattern uses a database, update the database user passwords.

1. For SQL Server:

   1. Run the SQL Server container with the previous secrets that you made a copy of. To do this you can update the docker run command to specify the backup directory for the secrets to mount or update the values in the secret environment variables.

      For example, the docker run command for SQL Server might look like:

      docker run -d \
      --name "sqlserver" \
      --network "eia" \
      --net-alias "sqlserver.eia" \
      -p "1433:1433" \
      -v "sqlserver_data:/var/opt/mssql" \
      -v "sqlserver_sqlbackup:/backup" \
      -v "/copy-of-secrets/simulated-secret-store/sqlserver:/run/secrets/" \
      -v "/home/<user-name>/analyze-deployment-tooling/prereqs/i2analyze/toolkit/examples/data:/var/i2a-data" \
      -e ACCEPT_EULA="Y" \
      -e MSSQL_AGENT_ENABLED=true \
      -e MSSQL_PID="Developer" \
      -e SA_PASSWORD_FILE="/run/secrets/SA_PASSWORD_FILE" \
      -e SERVER_SSL=true \
      -e SSL_PRIVATE_KEY_FILE="/run/secrets/server.key" \
      -e SSL_CERTIFICATE_FILE="/run/secrets/server.cer" \
      "i2group/i2eng-sqlserver:4.4.6"
      

      Where the -v "/copy-of-secrets/simulated-secret-store/sqlserver:/run/secrets" volume mount is the location of the backed up secrets.

      For more information about running the container, see SQL Server.

   2. Update the passwords for the i2analyze, etl, i2etl, dba, and dbb users by using the change_sql_server_user_password client function. The function takes the username, new password, and sa user's password as arguments. For example:

      change_sql_server_user_password "username" "new-password" "sa-password"
      

2. For Postgres:

   1. Run the Postgres container with the previous secrets that you made a copy of. To do this you can update the docker run command to specify the backup directory for the secrets to mount or update the values in the secret environment variables.

      For example, the docker run command for Postgres might look like:

      docker run -d \
      --name "postgres" \
      --network "eia" \
      --net-alias "postgres.eia" \
      -p "5432:5432" \
      -v "postgres_data:/var/lib/postgresql" \
      -v "postgres_sqlbackup:/backup" \
      -v "/copy-of-secrets/simulated-secret-store/postgres:/run/secrets/" \
      -v "i2a_data_server:/var/i2a-data" \
      -e POSTGRES_USER="postgres" \
      -e POSTGRES_PASSWORD="POSTGRES_PASSWORD" \
      -e SERVER_SSL=true \
      -e SSL_PRIVATE_KEY_FILE="/run/secrets/server.key" \
      -e SSL_CERTIFICATE_FILE="/run/secrets/server.cer" \
      "i2group/i2eng-postgres:4.4.6"
      

      Where the -v "/copy-of-secrets/simulated-secret-store/postgres:/run/secrets" volume mount is the location of the backed up secrets.

      For more information about running the container, see Postgres.

   2. Update the passwords for the i2analyze, etl, i2etl, dba, and dbb users by using the change_postgres_server_user_password client function. The function takes the username, new password, and postgres user's password as arguments.

      For example:

      change_postgres_server_user_password "username" "new-password" "postgres-password"
      

Liberty

1. Run the Liberty container with the new secrets. For more information, see Liberty.

HA Proxy

1. Run the HA Proxy container with the new secrets. For more information, see HA Proxy.

What to do next

The system is now deployed with the new secrets. Verify that the system is running as expected.