Managing container security

File permissions

Containers by design are independent of the host OS they are run on. They can have their own UIDs and with the use of volumes their own independent storage. However, files sometimes need to be moved between containers and hosts. The approach to managing this depends on the type of image.

Image documentation can be found in Images and containers

Server File permissions

All the server images are setup to run with a specific UID. These are documented for each image.

The server images require access to persistent file storage. This is provided by the use of either a volume or bind mount for each directory that needs to be persisted beyond the life of a container. The directory locations are documented for each image.

When files are created by running containers they will be created with the specific UIDs for each server. If for any reason these files are manipulated externally the UIDs need to be retained to ensure the servers function correctly.

The use of volumes is encouraged as as this will largely avoid any permission issues which can be encountered when using bind mounts.

Tool File permissions

Tools containers are used to perform one off operations. As such they don't have any need for persistent storage. However, it is common to want to consume or generate files.

To facilitate these operations bind mounts are used to attach the local file system to documented directory locations for the tools.

To ensure that the storage can be accessed correctly the tool images do not have pre-defined users. Instead the container must be run with the UID and GID of the local user. All file operations will then be performed as this user.

Typically this would be achieved by using the --user flag when running a Docker container to set the UID and GID. However, to make this easier the user switch is implemented in the container entrypoint by passing in the UID and GID as environment variables. Do NOT attempt to use the --user flag.

Care must be taken that the local user chosen has the correct permissions for any files being created/read by the tool.

Visual Studio Code Dev Container File permissions

The analyze-deployment-tooling project uses a Dev Container, which requires a bind mount of the workspace (and any others paths referenced in path-configuration.json file). The user that gets read, write, and execute permissions is called vscode and the container user's UID/GID is updated at runtime to match your local user (host). To learn more about Dev Container users, see Add a non-root user to a container.

SSL certificates in the deployment

The example deployment is configured to use SSL connections for communication between clients and i2 Analyze, and between the components of i2 Analyze.

To achieve this, the appropriate certificate authorities and certificates are used. The create-dev-environment script is used to simulate the process of creating the required keys and acquiring certificate authority-signed certificates.

Note: The keys and certificates used are set to expire after 365 days. To regenerate the certificates, delete the environment-secrets folder and its contents. Then run the create-dev-environment again. Finally, run the deploy script with the -t clean option and then redeploy. You must also reinstall your certificates on Windows, Mac, and Firefox.

Certificate Authorities (CA)

In the example, two CAs are used to provide trust:

Internal CA The internal CA is to provide trust for the containers that are used for the components of i2 Analyze. Each container's certificates are signed by the internal CA.
External CA The external CA is to provide trust for external requests to the i2 Analyze service via the load balancer. In our example, the external certificate authority is generated for you. However, in production a real certificate should be used.

Container certificates

To communicate securely using TLS each container requires the following certificates:

Private key
Certificate key
Internal certificate authority

The containers will generate truststores and keystores based on the keys provided to the container. For more information about how the keys are passed to the containers securely please see Secure Environment variables.

Secure communication between containers

When the components communicate, the CA certificate is used to establish trust of the container certificate that is received.

Each container has its own private key
ZooKeeper requires client authentication to initiate communication. The i2 Analyze, i2 Analyze Tool, and Solr client containers require container certificates to authenticate with ZooKeeper.

Creating keys and certificates

The following steps show a simplified sequence of creating a container certificate from the certificate authority and using it to establish trust:

The certificate authority's certificate is distributed to the client.
The private key is generated on the server.
In the generate-secrets script, the key is created by:
```
openssl genrsa -out server.key 4096
```
The public part of the private key is used in a Certificate Signing Request (CSR).
In the generate-secrets script, this is completed by:
```
openssl req -new -key server.key -subj "/CN=solr1.eia" -out key.csr
```
The common name that is used for the certificate is the server's fully qualified domain name.
The CSR is sent to the certificate authority (CA).

The CA signs and returns a signed certificate for the server.
In the generate-secrets script, the CA signing the certificate is completed by:

openssl x509 -req -sha256 -CA CA.cer -CAkey CA.key -days 90 -CAcreateserial -CAserial CA.srl -extfile x509.ext -extensions "solr" -in key.csr -out server.cer

When communication is established, the container certificate is sent to the client. The client uses its copy of the CA certificate to verify that the container certificate was signed by the same CA.

Additional certificates

If you need to connect to an external system, such as a database or authentication provider, that uses SSL then the system must trust the certificate that it uses. For security reasons, certificate verification is not optional; there is no "ignore verification and automatically trust all certificates" option.

You need to provide the trust certificates to the deployment so that the deployment knows to "trust" it. This file is a text file in OpenSSL PEM format, i.e. certificates should start "-----BEGIN CERTIFICATE-----".

You can provide any external systems public certificate's using the following mechanisms:

In the configuration/secrets/additional-trust-certificates.cer file that is built into the image.
As the value of the SSL_ADDITIONAL_TRUST_CERTIFICATES environment variable.

Do NOT include the private key in the file. That should be private to your external system.

Application secrets

You can provide additional variables to the environment through the APP_SECRETS environment variable.

The value for this environment variable is a .json file containing variable name definitions and values. The variables defined in the file can be used by Liberty.

For example:

{
  "CLIENT_SECRET": "TOKEN"
}

To reference the file in the Liberty configuration, use the ${ } format. For example, ${CLIENT_SECRET}

You can provide any additional secrets variables using the following mechanisms:

In the configuration/secrets/app-secrets.json file that is built into the image.
As a JSON string for the value of the APP_SECRETS environment variable.

Password generation

The example simulates secrets management performed by various secrets managers provided by cloud vendors. The generate-secrets generates these secrets and populates the environment-secrets/simulated-secret-store with secrets required for each container. The docker desktop does not support secrets, but the example environment example simulates this by mounting the secrets folder. For more information see Manage sensitive data with Docker secrets.

After these passwords have been generated, they can be uploaded to a secrets manager. Alternatively you can use a secrets manager to generate your passwords.

Solr Basic Authentication

Solr authentication can be enabled using the BasicAuthPlugin. The basic auth plugin defines users, user roles and passwords for users. For the BasicAuthPlugin to be enabled, solr requires a security.json file to be uploaded. In our example the security.json file is created by the generate-secrets and located in environment-secrets/generated-secrets/solr/security.json.

For more information about solr authentication see Basic Authentication Plugin

Prometheus Basic Authentication

Prometheus authentication can be enabled using the web-config.yml. The YAML file defines users and passwords for users. In our example the password is the same as the username.

Grafana Basic Authentication

Grafana authentication can be enabled by passing GF_SECURITY_ADMIN_USER and GF_SECURITY_ADMIN_PASSWORD environment variables to the container. In our example the password is created by generate-secrets and located in environment-secrets/generated-secrets/grafana/admin_PASSWORD.

Secure Environment variables

In general secrets used by a particular container can be supplied via an environment variable containing the path to a file containing the secret, or an environment variable specifying the literal secret value, for example: Note: Secrets can be passwords, keys or certificates.

docker run --name solr1 -d --net eia
  --secret source=SOLR_SSL_KEY_STORE_PASSWORD,target=SOLR_SSL_KEY_STORE_PASSWORD \
  -e SOLR_SSL_KEY_STORE_PASSWORD_FILE="/run/secrets/SOLR_SSL_KEY_STORE_PASSWORD_FILE"

docker run --name solr1 -d --net eia
  -e SOLR_SSL_KEY_STORE_PASSWORD="jhga98u43jndfj"

The docker files in the analyze-deployment-tooling repository have been modified to accept either. The convention is that the environment variable must match the property being set with "_file" appended if the secret is in a file, and without if a literal value is being used instead.

In the example scripts, each container gets the relevant key stores mounted along with the correct secrets files in the /run/secrets directory.

All the containers use the same environment variables to define the location of certificates. These are then used to generate appropriate artifacts for the particular container. There is also a standard way of turning on and off server SSL.

Security switching variables

Environment variable	Description
`SERVER_SSL`	Can be set to `true` or `false`. If set to `true`, the container is configured to use encrypted connections.
`LIBERTY_SSL_CONNECTION`	Can be set to `true` or `false`. If set to `true`, connections to the Liberty container use an encrypted connection.
`DB_SSL_CONNECTION`	Can be set to `true` or `false`. If set to `true`, connections to the database use an encrypted connection
`SOLR_ZOO_SSL_CONNECTION`	Can be set to `true` or `false`. If set to `true`, connections to ZooKeeper and Solr use an encrypted connection.
`GATEWAY_SSL_CONNECTION`	Can be set to `true` of `false`. If set to `true`, connections to i2 Connect connectors use an encrypted connection.
`SSL_ENABLED`	Can be set to `true` or `false`. If set to `true`, the connector container communicates with the i2 Connect gateway using an encrypted connection.

Security environment variables

Environment variable	Description
`SSL_PRIVATE_KEY`	The private key for the container certificate.
`SSL_CERTIFICATE`	The container certificate.
`SSL_CA_CERTIFICATE`	The Certificate Authority certificate.
`SSL_ADDITIONAL_TRUST_CERTIFICATES`	A .cer file that contains additional certificates that can provide trust.
`SSL_OUTBOUND_PRIVATE_KEY`	The private key for the Liberty container, which is used for outbound connections.
`SSL_OUTBOUND_CERTIFICATE`	The private certificate for the Liberty container, which is used for outbound connections.
`SSL_OUTBOUND_CA_CERTIFICATE`	The certificate authority used for verifying outbound connections.