External dependencies

The system requires access to the internet to download a number of "external" dependencies: Docker images, Maven artifacts, and JDBC drivers. Where the system needs to be deployed into an environment that does not have internet access, these dependencies must be downloaded on a "low-side" environment that does have internet access before they are transferred to the air-gapped "high-side" environment that does not have internet access.

This document outlines the steps to set up the ADT client and its dependencies in such an environment.

  1. The process starts on the low-side, which has internet access, and where any necessary resources are downloaded and processed.
  2. The resources are transferred to the high-side.
  3. Once the resources are on the high-side, ADT can be run without internet access.

The example commands in this document assume that the low-side and high-side are unix environments running Docker.

Where user-customization is required, the modifiable parts of the commands are marked with an end-of-line comment # User-editable.

Low-side setup

  1. Pull the required Docker images:

    ALL_DOCKER_IMAGES=(
    docker/dockerfile:1.3
    i2group/i2eng-analyze-containers-base:ubi-jdk17
    i2group/i2eng-analyze-containers-client:3.1.1
    i2group/i2eng-analyze-containers-connectors-base:18
    i2group/i2eng-analyze-containers-connectors-base-minimal:18
    i2group/i2eng-haproxy:2.9
    i2group/i2eng-liberty:ubi-jdk17
    i2group/i2eng-solr:9.7
    i2group/i2eng-zookeeper:3.9
    i2group/i2eng-connector-designer:1
    i2group/i2eng-connector-designer-connectors-base:1
    i2group/i2eng-grafana:11
    i2group/i2eng-postgres:17
    i2group/i2eng-prometheus:2.53
    i2group/i2eng-sqlserver:2022-1
    i2group/i2eng-textchart-data-access:7.5.0.6
    i2group/i2eng-textchart-manager:7.5.2.0
    i2group/i2eng-textchart-worker:7.5.2.0
    # User-editable
)
for DOCKER_IMAGE in "${ALL_DOCKER_IMAGES[@]}"; do
    docker pull "${DOCKER_IMAGE}"
done

    If you have connector images that you require then include those in the list, both here and in the later steps where ALL_DOCKER_IMAGES are processed.

  2. Run bootstrap to pull pre-requisites and create the ac_m2 docker volume:

    ./bootstrap -v -o 3.1.1

    Note that this process will also create some other docker images but these will not be used on the low-side.

Data transfer

The precise method of data transfer from low-side to high-side will depend on the security requirements of the environment. For example, the data transfer may be done using removable storage, a secure file transfer protocol, or having a docker registry that's accessible by both the low-side and high-side (See How to run a local docker registry for more information on setting up a local docker registry) .

Choose the method that is most appropriate for your environment from the following options:

  1. Removable storage
  2. Network file transfer
  3. Docker registry

Data transfer using removable storage

Low-side actions for removable storage

  1. Environment setup:

    The example commands use an environment variable REMOVABLE_STORAGE_MOUNT_POINT to refer to the location of the removable storage. You must set this variable to the location of the removable storage before running the commands.

    The commands also use the CLIENT_IMAGE variable to refer to the ADT client image but this does not require user-editing.

    REMOVABLE_STORAGE_MOUNT_POINT='/path/to/removable/storage' # User-editable
ALL_DOCKER_IMAGES=(
    # Same as the list used to pull the images
    # User-editable
)
CLIENT_IMAGE=$( docker images --format "{{.Repository}}:{{.Tag}}" i2group/i2eng-analyze-containers-client | sort -r | head -1 )

    You must run this in any terminal you wish to use the subsequent commands in. e.g. if you close one terminal session and open another then you will need to repeat this step in the newly-opened terminal.

    The ALL_DOCKER_IMAGES variable content must be the same as the one used to pull the images during the Low-side setup step.

  2. Prepare the ac_m2 docker volume for transfer:

    Create a tar file of the docker volume:

    docker run --rm -v ac_m2:/mnt/ -u 0:0 --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" cf - -C /mnt/ . > "${REMOVABLE_STORAGE_MOUNT_POINT}/ac_m2.tar"

    This command will create a tar file ac_m2.tar containing the contents of the docker volume in the removable storage.

  3. Prepare ADT pre-requisites for transfer:

    Archive the files that you want to transfer by cding to the ADT directory and then running:

    ADT_FILES=( 'pre-reqs/i2analyzeMinimal.tar.gz' 'pre-reqs/jdbc-drivers' ) # User-editable
docker run --rm -v "$(pwd):/mnt/" -u "$(id -u):$(id -g)" --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" cf - --owner=none:0 --group=none:0 -C /mnt/ "${ADT_FILES[@]}" \
  > "${REMOVABLE_STORAGE_MOUNT_POINT}/adt-pre-reqs.tar"

    Note: This command will only copy the pre-requisites; you will also need to copy ADT itself and your config separately. You can do this using a tar command like the one above if you choose.

  4. Prepare docker images for transfer:

    Create a tar file of the docker images using the docker save command:

    docker save -o "${REMOVABLE_STORAGE_MOUNT_POINT}/all_images.image.tar" "${ALL_DOCKER_IMAGES[@]}"

    Note that this will create a single tar file containing all of the images listed. If you have images that you do not want to transfer, you can remove them from the list by modifying the ALL_DOCKER_IMAGES variable before running the docker save command.

Physically move the data

Remove the storage from the low-side (don't forget to unmount it first) and move it to the high-side.

High-side actions for removable storage

  1. Extract all of the docker images:

    If using removable storage, you must now restore the docker images from the tar file you created on the low-side:

    REMOVABLE_STORAGE_MOUNT_POINT='/path/to/removable/storage' # User-editable
for FILE in "${REMOVABLE_STORAGE_MOUNT_POINT}"/*.image.tar; do
    docker load -i "${FILE}"
done

    This will load each image from the removable storage into the local docker daemon on the high-side.

  2. Optional: Verify docker images:

    To verify that all of the images on the low-side are now on the high-side, you can run the following command on both sides and compare the output:

    docker images --format "{{.ID}}\t{{.Repository}}:{{.Tag}}" | sort -k2

    If the images are the same, the output will be identical on both sides. The ID field will be the same when the content of the image is the same; if the IDs are different, the content of the images is different.

  3. Create and populate the ac_m2 docker volume:

    Extract the ac_m2 tar file you created into a new docker volume as follows:

    docker volume rm -f ac_m2
docker volume create ac_m2
docker run --rm -i -v ac_m2:/mnt/ -u 0:0 --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" xf - -C /mnt/ < "${REMOVABLE_STORAGE_MOUNT_POINT}/ac_m2.tar"

    This will extract the contents of the ac_m2.tar file into a newly-created ac_m2 docker volume, erasing any previous volume of that name.

  4. Extract the ADT files:

    Extract the tar file of ADT files into the ADT directory as follows:

    cd '/path/to/adt' # User-editable
cat "${REMOVABLE_STORAGE_MOUNT_POINT}/adt-pre-reqs.tar" | \
  docker run --rm -i -v "$(pwd):/mnt/" -u "$(id -u):$(id -g)" --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" xf - --no-same-owner -C /mnt/

Data transfer using network file transfer

If you are using a network file transfer method, for example scp, sftp or rsync (via e.g. ssh), then the process is similar to that for removable storage. Use the commands for removable storage with a few modifications:

Low-side actions for network file transfer

Follow the same steps as for removable storage but set the REMOVABLE_STORAGE_MOUNT_POINT variable to the location of an empty temporary directory.

Transfer the data using network file transfer

Use the network to transfer the files from the REMOVABLE_STORAGE_MOUNT_POINT temporary directory on the low-side to a temporary directory on the high-side.

High-side actions for network file transfer

Follow the same steps as for removable storage but set the REMOVABLE_STORAGE_MOUNT_POINT variable to the location of the temporary directory where you transferred the files to.

Data transfer using a docker registry

If you have a docker registry that is accessible from both the low-side and high-side then you can use that as an intermediary, uploading the data as images from the low-side and pulling them from the registry on the high-side.

This makes it easier to transfer docker images (as they are already in image format) but you must encapsulate all other data in docker images too.

Low-side actions for docker registry

  1. Environment setup:

    The example commands use an environment variable PRIVATE_REGISTRY to refer to the private docker registry. You must set this variable before running the commands. If the registry requires authentication then you will need to do docker login "${PRIVATE_REGISTRY}" too.

    The commands also use the CLIENT_IMAGE variable to refer to the ADT client image but this does not require user-editing.

    PRIVATE_REGISTRY='docker_registry_hostname:port' # User-editable
ALL_DOCKER_IMAGES=(
    # Same as the list used to pull the images
    # User-editable
)
CLIENT_IMAGE=$( docker images --format "{{.Repository}}:{{.Tag}}" i2group/i2eng-analyze-containers-client | sort -r | head -1 )

    You will need to run this in any terminal you wish to use the subsequent commands in. e.g. if you close one terminal session and open another then you will need to repeat this step in the newly-opened terminal.

    The ALL_DOCKER_IMAGES variable content must be the same as the one used to pull the images during the Low-side setup step.

  2. Prepare the ac_m2 docker volume for transfer:

    Encapsulate the docker volume in a docker image and push it to the registry.

    docker image inspect 'ac_m2_image:latest' >/dev/null 2>&1 && docker rmi 'ac_m2_image:latest'
docker container inspect 'ac_m2_container' >/dev/null 2>&1 && docker rm 'ac_m2_container'
docker run -v ac_m2:/mnt/ -u 0:0 --name 'temp_ac_m2_container' --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" cf /volume.tar -C /mnt/ .
docker commit 'temp_ac_m2_container' 'ac_m2_image:latest'
docker rm -f 'temp_ac_m2_container'

  3. Prepare ADT pre-requisites for transfer:

    Create a docker image containing the files that you want to transfer by cding to the ADT directory and then running:

    ADT_FILES=( 'pre-reqs/i2analyzeMinimal.tar.gz' 'pre-reqs/jdbc-drivers' ) # User-editable
docker image inspect 'adt_pre_reqs_image:latest' >/dev/null 2>&1 && docker rmi -f 'adt_pre_reqs_image:latest'
docker container inspect 'temp_adt_pre_reqs_container' >/dev/null 2>&1 && docker rm -f 'temp_adt_pre_reqs_container'
docker run --rm -v "$(pwd):/mnt/" -u "$(id -u):$(id -g)" --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" cf - --owner=none:0 --group=none:0 -C /mnt/ "${ADT_FILES[@]}" | \
  docker run -i -u 0:0 --name 'temp_adt_pre_reqs_container' --entrypoint /bin/bash \
    "${CLIENT_IMAGE}" -c 'cat > /volume.tar'
docker commit 'temp_adt_pre_reqs_container' 'adt_pre_reqs_image:latest'
docker rm -f 'temp_adt_pre_reqs_container'

    Note: This command will only copy the pre-requisites; you will also need to copy ADT itself and your config separately. You can do this using a tar command like the one above if you choose.

Transfer the data using the docker registry

  1. Push the images from the low-side to the registry:

    IMAGES_TO_PUSH=(
    'ac_m2_image:latest'
    'adt_pre_reqs_image:latest'
    "${ALL_DOCKER_IMAGES[@]}"
)
for DOCKER_IMAGE in "${IMAGES_TO_PUSH[@]}"; do
    docker tag "$DOCKER_IMAGE" "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}"
    docker push "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}"
    docker rmi "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}"
done

  2. Extract all of the docker images on the high-side:

    Pull the images from that into the local docker daemon. You must replace the ... in the command below with the list of images you generated in the previous step.

    If your registry requires authentication then you will need to do docker login "${PRIVATE_REGISTRY}" first.

    ALL_DOCKER_IMAGES=(
    # Same as the list used to pull the images
    # User-editable
)
IMAGES_TO_PULL=(
    'ac_m2_image:latest'
    'adt_pre_reqs_image:latest'
    "${ALL_DOCKER_IMAGES[@]}"
)
for DOCKER_IMAGE in "${IMAGES_TO_PULL[@]}"; do
    docker pull "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}"
    docker tag "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}" "$DOCKER_IMAGE"
    docker rmi "${PRIVATE_REGISTRY}/${DOCKER_IMAGE}"
done

    The ALL_DOCKER_IMAGES variable content must be the same as the one used to pull the images during the Low-side setup step.

  3. Optional: Verify docker images:

    To verify that all of the images on the low-side are now on the high-side, you can run the following command on both sides and compare the output:

    docker images --format "{{.ID}}\t{{.Repository}}:{{.Tag}}" | sort -k2

    If the images are the same, the output will be identical on both sides. The ID field will be the same when the content of the image is the same; if the IDs are different, the content of the images is different. Unwanted images can be removed using the docker rmi command.

High-side actions for docker registry

  1. Extract the ac_m2 docker volume from the image:

    docker volume rm -f ac_m2
docker volume create ac_m2
docker run --rm -v ac_m2:/mnt/ -u 0:0 --entrypoint /usr/bin/tar 'ac_m2_image:latest' xf /volume.tar -C /mnt/
docker rmi 'ac_m2_image:latest'

    This will extract the contents of the tar file embedded in the ac_m2_image:latest image into a newly-created ac_m2 docker volume, erasing any previous volume of that name and erasing the image afterwards.

  2. Extract the ADT files from the image:

    cd '/path/to/adt' # User-editable
docker run --rm -u 0:0 --entrypoint /usr/bin/cat 'adt_pre_reqs_image:latest' /volume.tar | \
  docker run --rm -i -v "$(pwd):/mnt/" -u "$(id -u):$(id -g)" --entrypoint /usr/bin/tar \
  "${CLIENT_IMAGE}" xf - --no-same-owner -C /mnt/
docker rmi 'adt_pre_reqs_image:latest'

High-side setup

After the data transfer is complete and the resources have been extracted, you can deploy "as normal" on the high-side, but with a few extra steps to ensure that the system does not try to pull any images from the internet.

  1. Tell the high-side to bootstrap without re-fetching images.

    Create a file docker.env (place it in the same directory as the bootstrap script file), populate it with contents ALWAYS_PULL_IMAGES=false, and then run the bootstrap script with environment variable USE_LOCAL_IMAGE set to true:

    cd '/path/to/adt' # User-editable
echo 'ALWAYS_PULL_IMAGES=false' > 'docker.env'
export USE_LOCAL_IMAGE=true
./bootstrap -v -o 3.1.1

  2. Use it:

    All subsequent interactions also require USE_LOCAL_IMAGE set to true, e.g.:

    cd '/path/to/adt' # User-editable
export USE_LOCAL_IMAGE=true
./scripts/deploy -v -c <config_name>

Footnotes

How to run a local docker registry

While a fully "air-gapped" environment is possible using some form of removable storage, it is often more convenient to have a minimal network connection between the low-side and high-side, and one common use case is to have a local docker registry that is accessible from both sides.

To create a Docker registry you will need to install Docker on a machine that is accessible from both the low-side and high-side, and then run the following command to create a local registry:

docker pull registry:2
docker run -d -p 5000:5000 --restart=always --name registry registry:2

This will create a local docker registry that is accessible on port 5000 of the machine it is running on, so where the previous instructions said PRIVATE_REGISTRY='docker_registry_hostname:port' you can set it to hostname:5000 for the hostname of the machine running the registry.

For information about deploying a Docker registry, see Deploy a registry server.

Docker image reference

ADT makes use of a number of docker images, some of which are always required for the system to deploy and run, and some of which are optional and may be required (or not) depending on the configuration being deployed. These are further subdivided into "external" images that are pulled from the (public) docker registry and "local" images that are built by this code (based on the external ones).

Externally-pulled images

External mandatory images
docker/dockerfile:1.3
i2group/i2eng-analyze-containers-base:ubi-jdk17
i2group/i2eng-analyze-containers-client:3.1.1
i2group/i2eng-analyze-containers-connectors-base:18
i2group/i2eng-analyze-containers-connectors-base-minimal:18
i2group/i2eng-haproxy:2.9
i2group/i2eng-liberty:ubi-jdk17
i2group/i2eng-solr:9.7
i2group/i2eng-zookeeper:3.9
External optional images
i2group/i2eng-connector-designer:1
i2group/i2eng-connector-designer-connectors-base:1
i2group/i2eng-grafana:11
i2group/i2eng-postgres:17
i2group/i2eng-prometheus:2.53
i2group/i2eng-sqlserver:2022-1
i2group/i2eng-textchart-data-access:7.5.0.6
i2group/i2eng-textchart-manager:7.5.2.0
i2group/i2eng-textchart-worker:7.5.2.0

Locally-built images

Local mandatory images
ac_tools:4.4.5.0
etlclient_redhat:4.4.5.0
i2a_tools_redhat:4.4.5.0
liberty_redhat:4.4.5.0
solr_client_redhat:4.4.5.0
solr_redhat:4.4.5.0
Local optional images
postgres_client_image:4.4.5.0
sqlserver_client_redhat:4.4.5.0

Full list of images as JSON

You can generate the full list of images required for the deployment by running the following command:

manage-environment -t create-image-list

This command creates a file called adt-image-list.json in the ADT directory containing the full list of images. The file is in JSON format and will look like this:

{
  "i2group-core": [
    {
      "name": "docker/dockerfile",
      "version": "1.3"
    },
    {
      "name": "i2group/i2eng-analyze-containers-base",
      "version": "ubi-jdk17"
    },
    {
      "name": "i2group/i2eng-analyze-containers-client",
      "version": "3.1.1"
    },
    {
      "name": "i2group/i2eng-analyze-containers-connectors-base",
      "version": "18"
    },
    {
      "name": "i2group/i2eng-analyze-containers-connectors-base-minimal",
      "version": "18"
    },
    {
      "name": "i2group/i2eng-haproxy",
      "version": "2.9"
    },
    {
      "name": "i2group/i2eng-liberty",
      "version": "ubi-jdk17"
    },
    {
      "name": "i2group/i2eng-solr",
      "version": "9.7"
    },
    {
      "name": "i2group/i2eng-zookeeper",
      "version": "3.9"
    }
  ],
  "i2group-optional": [
    {
      "name": "i2group/i2eng-connector-designer",
      "version": "1"
    },
    {
      "name": "i2group/i2eng-connector-designer-connectors-base",
      "version": "1"
    },
    {
      "name": "i2group/i2eng-grafana",
      "version": "11"
    },
    {
      "name": "i2group/i2eng-postgres",
      "version": "17"
    },
    {
      "name": "i2group/i2eng-prometheus",
      "version": "2.53"
    },
    {
      "name": "i2group/i2eng-sqlserver",
      "version": "2022-1"
    },
    {
      "name": "i2group/i2eng-textchart-data-access",
      "version": "7.5.0.6"
    },
    {
      "name": "i2group/i2eng-textchart-manager",
      "version": "7.5.2.0"
    },
    {
      "name": "i2group/i2eng-textchart-worker",
      "version": "7.5.2.0"
    }
  ],
  "i2a-core": [
    {
      "name": "ac_tools",
      "version": "4.4.5.0"
    },
    {
      "name": "etlclient_redhat",
      "version": "4.4.5.0"
    },
    {
      "name": "i2a_tools_redhat",
      "version": "4.4.5.0"
    },
    {
      "name": "liberty_redhat",
      "version": "4.4.5.0"
    },
    {
      "name": "solr_client_redhat",
      "version": "4.4.5.0"
    },
    {
      "name": "solr_redhat",
      "version": "4.4.5.0"
    }
  ],
  "i2a-optional": [
    {
      "name": "postgres_client_image",
      "version": "4.4.5.0"
    },
    {
      "name": "sqlserver_client_redhat",
      "version": "4.4.5.0"
    }
  ]
}