ETL Client

An ETL Client container is an ephemeral container that is used to run ETL tasks.

Building an ETL Client image

The ETL Client image is built from the Dockerfile in images/etl_client. It uses the i2 Analyze Tools image as the base image.

Docker build command

The following docker build command builds the ETL Client image:

docker build -t "etlclient_redhat:4.4.4" "images/etl_client" \
    --build-arg BASE_IMAGE="i2a_tools_redhat:4.4.4"

Running an ETL Client container

A ETL Client container uses the ETL Client image. In the docker run command, you can use -e to pass environment variables to the container. The environment variables are described in environment variables.

The container will run with a default User ID and Group ID of 1000 (this can be changed with the USER_ID and GROUP_ID env variables). All files in mounted directories will be created with these IDs. If files are manipulated externally these IDs must be retained or the container will not function correctly.

For more information about the command, see docker run reference.

Docker run command

The following docker run command runs the ETL Client container:

docker run --rm \
    --network "eia" \
    -v "/home/<user-name>/analyze-deployment-tooling/examples/pre-prod/configuration/logs:/opt/configuration/logs" \
    -v "/home/<user-name>/analyze-deployment-tooling/prereqs/i2analyze/toolkit/examples/data:/var/i2a-data" \
    -e USER_ID="$(id -u)" -e GROUP_ID="$(id -g)" \
    -e DB_SERVER="sqlserver.eia" \
    -e DB_PORT="1433" \
    -e DB_NAME="ISTORE" \
    -e DB_DIALECT="sqlserver" \
    -e DB_OS_TYPE="UNIX" \
    -e DB_INSTALL_DIR="/opt/mssql-tools" \
    -e DB_LOCATION_DIR="/var/opt/mssql/data" \
    -e ETL_TOOLKIT_JAVA_HOME="/opt/java/openjdk/bin" \
    -e DB_USERNAME="i2etl" \
    -e DB_PASSWORD="DB_PASSWORD" \
    -e DB_SSL_CONNECTION="true" \
    -e SSL_CA_CERTIFICATE="SSL_CA_CERTIFICATE" \
    "etlclient_redhat:4.4.4" "$@"

The local user ID is required so that a user is created in the Docker container with a USER_ID that is the same as the local user. The user is required to ensure that the local user can access any files that are generated on the container. The value of $id comes from your shell.

For an example of the docker run command, see run_etl_toolkit_tool_as_i2_etl function in the utils/client_functions.sh script. For an example of how to use run_etl_toolkit_tool_as_i2_etl function, see run_etl_toolkit_tool_as_i2_etl.

Bind mounts

  • Logs:
    The ETL Client container generates logs when running ingestion commands. To make these logs easily accessible, the directory where they are generated must be mounted into the container. In the example, the logs can be accessed in the examples/pre-prod/configuration/logs directory.

  • Data:
    The ETL Client container requires access to the data directory to run the ingestion scripts. To access the data, the data directory must be mounted into the container.

Environment variables

The ETl Client is built on top of the SQL Client. Any environment variables referenced in the SQL Client can be used in the ETL Client.

Additional Environment variables

Environment Variable Description
DB_DIALECT The database dialect. Currently sqlserver and postgres are supported
DB_OS_TYPE The Operating System that the database is on. Can be UNIX, WIN, or AIX.
DB_INSTALL_DIR Specifies the database CMD location.
DB_LOCATION_DIR Specifies the location of the database.
ETL_TOOLKIT_JAVA_HOME Specifies the location of Java bin directory.