Managing container security
File permissions
Containers by design are independent of the host OS they are run on. They can have their own UIDs and with the use of volumes their own independent storage. However, files sometimes need to be moved between containers and hosts. The approach to managing this depends on the type of image.
Image documentation can be found in Images and containers
Server File permissions
All the server images are setup to run with a specific UID. These are documented for each image.
The server images require access to persistent file storage. This is provided by the use of either a volume or bind mount for each directory that needs to be persisted beyond the life of a container. The directory locations are documented for each image.
When files are created by running containers they will be created with the specific UIDs for each server. If for any reason these files are manipulated externally the UIDs need to be retained to ensure the servers function correctly.
The use of volumes is encouraged as as this will largely avoid any permission issues which can be encountered when using bind mounts.
Tool File permissions
Tools containers are used to perform one off operations. As such they don't have any need for persistent storage. However, it is common to want to consume or generate files.
To facilitate these operations bind mounts are used to attach the local file system to documented directory locations for the tools.
To ensure that the storage can be accessed correctly the tool images do not have pre-defined users. Instead the container must be run with the UID and GID of the local user. All file operations will then be performed as this user.
Typically this would be achieved by using the --user
flag when running a Docker container to set the UID and GID. However, to make this easier the user switch is implemented in the container entrypoint by passing in the UID and GID as environment variables. Do NOT attempt to use the --user
flag.
Care must be taken that the local user chosen has the correct permissions for any files being created/read by the tool.
Visual Studio Code Dev Container File permissions
The analyze-deployment-tooling project uses a Dev Container, which requires a bind mount of the workspace
(and any others paths referenced in path-configuration.json file).
The user that gets read, write, and execute permissions is called vscode
and the container user's UID/GID is updated at runtime to match your local user (host).
To learn more about Dev Container users, see
Add a non-root user to a container.
SSL certificates in the deployment
The example deployment is configured to use SSL connections for communication between clients and i2 Analyze, and between the components of i2 Analyze.
To achieve this, the appropriate certificate authorities and certificates are used. The create-dev-environment
script is used to simulate the process of creating the required keys and acquiring certificate authority-signed certificates.
Note: The keys and certificates used are set to expire after 365 days. To regenerate the certificates, delete the
environment-secrets
folder and its contents. Then run thecreate-dev-environment
again. Finally, run the deploy script with the-t clean
option and then redeploy. You must also reinstall your certificates on Windows, Mac, and Firefox.
Certificate Authorities (CA)
In the example, two CAs are used to provide trust:
- Internal CA The internal CA is to provide trust for the containers that are used for the components of i2 Analyze. Each container's certificates are signed by the internal CA.
- External CA The external CA is to provide trust for external requests to the i2 Analyze service via the load balancer. In our example, the external certificate authority is generated for you. However, in production a real certificate should be used.
Container certificates
To communicate securely using TLS each container requires the following certificates:
- Private key
- Certificate key
- Internal certificate authority
The containers will generate truststores and keystores based on the keys provided to the container. For more information about how the keys are passed to the containers securely please see Secure Environment variables.
Secure communication between containers
When the components communicate, the CA certificate
is used to establish trust of the container certificate
that is received.
Each container has its own private key
ZooKeeper requires client authentication to initiate communication. The i2 Analyze, i2 Analyze Tool, and Solr client containers require container certificates to authenticate with ZooKeeper.
Creating keys and certificates
The following steps show a simplified sequence of creating a container certificate from the certificate authority and using it to establish trust:
The certificate authority's certificate is distributed to the client.
The private key is generated on the server.
In thegenerate-secrets
script, the key is created by:openssl genrsa -out server.key 4096
The public part of the private key is used in a Certificate Signing Request (CSR).
In thegenerate-secrets
script, this is completed by:openssl req -new -key server.key -subj "/CN=solr1.eia" -out key.csr
The common name that is used for the certificate is the server's fully qualified domain name.
The CSR is sent to the certificate authority (CA).The CA signs and returns a signed certificate for the server.
In thegenerate-secrets
script, the CA signing the certificate is completed by:openssl x509 -req -sha256 -CA CA.cer -CAkey CA.key -days 90 -CAcreateserial -CAserial CA.srl -extfile x509.ext -extensions "solr" -in key.csr -out server.cer
When communication is established, the
container certificate
is sent to the client. The client uses its copy of theCA certificate
to verify that thecontainer certificate
was signed by the same CA.
Additional certificates
If you need to connect to an external system, such as a database or authentication provider, that uses SSL then the system must trust the certificate that it uses. For security reasons, certificate verification is not optional; there is no "ignore verification and automatically trust all certificates" option.
You need to provide the trust certificates to the deployment so that the deployment knows to "trust" it. This file is a text file in OpenSSL PEM format, i.e. certificates should start "-----BEGIN CERTIFICATE-----
".
You can provide any external systems public certificate's using the following mechanisms:
- In the
configuration/secrets/additional-trust-certificates.cer
file that is built into the image. - As the value of the
SSL_ADDITIONAL_TRUST_CERTIFICATES
environment variable.
Do NOT include the private key in the file. That should be private to your external system.
Application secrets
You can provide additional variables to the environment through the APP_SECRETS
environment variable.
The value for this environment variable is a .json
file containing variable name definitions and values. The variables defined in the file can be used by Liberty.
For example:
{
"CLIENT_SECRET": "TOKEN"
}
To reference the file in the Liberty configuration, use the ${ }
format. For example, ${CLIENT_SECRET}
You can provide any additional secrets variables using the following mechanisms:
- In the
configuration/secrets/app-secrets.json
file that is built into the image. - As a JSON string for the value of the
APP_SECRETS
environment variable.
Password generation
The example simulates secrets management performed by various secrets managers provided by cloud vendors. The generate-secrets
generates these secrets and populates the environment-secrets/simulated-secret-store
with secrets required for each container.
The docker desktop does not support secrets, but the example environment example simulates this by mounting the secrets folder. For more information see Manage sensitive data with Docker secrets.
After these passwords have been generated, they can be uploaded to a secrets manager. Alternatively you can use a secrets manager to generate your passwords.
Solr Basic Authentication
Solr authentication can be enabled using the BasicAuthPlugin. The basic auth plugin defines users, user roles and passwords for users. For the BasicAuthPlugin to be enabled, solr requires a security.json
file to be uploaded. In our example the security.json
file is created by the generate-secrets
and located in environment-secrets/generated-secrets/solr/security.json
.
For more information about solr authentication see Basic Authentication Plugin
Prometheus Basic Authentication
Prometheus authentication can be enabled using the web-config.yml. The YAML file defines users and passwords for users. In our example the password is the same as the username.
Grafana Basic Authentication
Grafana authentication can be enabled by passing GF_SECURITY_ADMIN_USER
and GF_SECURITY_ADMIN_PASSWORD
environment variables to the container. In our example the password is created by generate-secrets
and located in environment-secrets/generated-secrets/grafana/admin_PASSWORD
.
Secure Environment variables
In general secrets used by a particular container can be supplied via an environment variable containing the path to a file containing the secret, or an environment variable specifying the literal secret value, for example: Note: Secrets can be passwords, keys or certificates.
docker run --name solr1 -d --net eia
--secret source=SOLR_SSL_KEY_STORE_PASSWORD,target=SOLR_SSL_KEY_STORE_PASSWORD \
-e SOLR_SSL_KEY_STORE_PASSWORD_FILE="/run/secrets/SOLR_SSL_KEY_STORE_PASSWORD_FILE"
or
docker run --name solr1 -d --net eia
-e SOLR_SSL_KEY_STORE_PASSWORD="jhga98u43jndfj"
The docker files in the analyze-deployment-tooling
repository have been modified to accept either. The convention is that the environment variable must match the property being set with "_file" appended if the secret is in a file, and without if a literal value is being used instead.
In the example scripts, each container gets the relevant key stores mounted along with the correct secrets files in the /run/secrets
directory.
All the containers use the same environment variables to define the location of certificates. These are then used to generate appropriate artifacts for the particular container. There is also a standard way of turning on and off server SSL.
Security switching variables
Environment variable | Description |
---|---|
SERVER_SSL |
Can be set to true or false . If set to true , the container is configured to use encrypted connections. |
LIBERTY_SSL_CONNECTION |
Can be set to true or false . If set to true , connections to the Liberty container use an encrypted connection. |
DB_SSL_CONNECTION |
Can be set to true or false . If set to true , connections to the database use an encrypted connection |
SOLR_ZOO_SSL_CONNECTION |
Can be set to true or false . If set to true , connections to ZooKeeper and Solr use an encrypted connection. |
GATEWAY_SSL_CONNECTION |
Can be set to true of false . If set to true , connections to i2 Connect connectors use an encrypted connection. |
SSL_ENABLED |
Can be set to true or false . If set to true , the connector container communicates with the i2 Connect gateway using an encrypted connection. |
Security environment variables
Environment variable | Description |
---|---|
SSL_PRIVATE_KEY |
The private key for the container certificate. |
SSL_CERTIFICATE |
The container certificate. |
SSL_CA_CERTIFICATE |
The Certificate Authority certificate. |
SSL_ADDITIONAL_TRUST_CERTIFICATES |
A .cer file that contains additional certificates that can provide trust. |
SSL_OUTBOUND_PRIVATE_KEY |
The private key for the Liberty container, which is used for outbound connections. |
SSL_OUTBOUND_CERTIFICATE |
The private certificate for the Liberty container, which is used for outbound connections. |
SSL_OUTBOUND_CA_CERTIFICATE |
The certificate authority used for verifying outbound connections. |