client_functions.sh
A set of client functions that you can use to perform actions against the server components of i2 Analyze.
Overview
The list of client functions groups:
- Environment Utilities
- Secret Utilities
- Status Utilities
- Database Security Utilities
- Execution Utilities
- Volume Utilities
Index
- check_env_vars_are_set
- get_secret
- get_secret_or_default
- get_solr_status
- get_async_request_status
- wait_for_indexes_to_be_built
- wait_for_connector_to_be_live
- change_sql_server_user_password
- change_sa_password
- change_postgres_server_user_password
- change_postgres_password
- create_dba_login_and_user
- create_db_login_and_user
- add_user_to_role
- change_zk_password
- run_solr_client_command
- run_i2_analyze_tool
- run_i2_analyze_tool_as_external_user
- run_sql_server_command_as_etl
- run_postgres_server_command_as_etl
- run_postgres_server_command_as_first_start_postgres
- run_postgres_server_command_as_postgres
- run_postgres_server_command_as_dba
- run_postgres_server_command_as_dbb
- run_sql_server_command_as_first_start_sa
- run_sql_server_command_as_sa
- run_sql_server_command_as_dba
- run_sql_server_command_as_dbb
- run_etl_toolkit_tool_as_i2_etl
- run_etl_toolkit_tool_as_dba
- update_volume_from_volume
- update_volume
- get_volume
Environment utilities
Checks if all required environment variables are set.
check_env_vars_are_set
Checks if all required environment variables are set.
Function has no arguments.
Secret utilities
get_secret
Gets a secret such as a password for a user.
Example
get_secret "solr/ZK_DIGEST_PASSWORD"
get_secret "additional-trust-certificates.cer" true ''
Arguments
- $1 (string): The partial path to secret
- $2 (boolean): Optional, defaults to false. Whether user manged or not
- $3 (string): Optional, no default. If set, provides the default secret value (which can be empty) to be used if the secret is not defined. If no default value is given AND the secret does not exist then an error will be logged and the script will exit.
get_secret_or_default
Gets a secret such as a password for a user. and defaults to "default" if the file doesn't exist.
Example
get_secret "solr/ZK_DIGEST_PASSWORD"
Arguments
- $1 (string): The partial path to secret
Status utilities
Get Solr component status from a certain point in time.
get_solr_status
Get Solr component status from a certain point in time.
Example
status_message="$(get_solr_status "${SINCE_TIMESTAMP}")"
Arguments
- $1 (string): "since" timestamp for docker logs commands
Output on stdout
- The status of the Solr component.
get_async_request_status
Takes a request to the Asynchronous Collection API and check the state is marked as completed in the JSON response returned. If the state is not marked as completed, the function returns the response message which contains any error messages that are reported with the asynchronous request. For more information about the Asynchronous Collection API, see REQUESTSTATUS: Request Status of an Async Call.
Arguments
- $1 (string): The request id of the asynchronous request to get the status of.
Output on stdout
- The JSON response or error messages.
wait_for_indexes_to_be_built
Wait for the index to be build by running an admin request against Solr API and checking the index is in the "Ready" state.
Example
wait_for_indexes_to_be_built "match_index1"
Arguments
- $1 (string): Index Name
Exit codes
- 0: If index was built in 75 seconds.
- 1: If index was NOT built in 75 seconds.
wait_for_connector_to_be_live
Sends a request to the connector's/config endpoint. If the response is 200, the connector is live. If the connector is not live after 10 retries the function will print an error and exit.
Arguments
- $1 (string): The fully qualified domain name of a connector
- $2 (string): The configuration path of the connector
Exit codes
- 0: If the connector is live.
- 1: If the connector is not live after 10 retries.
Output on stdout
- Retry attempts, error messages or success message.
Database Security Utilities
Change password for the specified user.
change_sql_server_user_password
Change password for the specified user.
change_sa_password
Change the initial password for the SA user. Uses the generated secrets to call the change_sa_password.sh with the initial (generated) sa password and the new (generated) password. For more information, see change_sa_password.
Function has no arguments.
change_postgres_server_user_password
Change password for the specified user.
change_postgres_password
Change the initial password for the postgres user.
Function has no arguments.
create_dba_login_and_user
Creates a database login and user for the dba user, and assigns the user to the provided role. For more information, see db_users.
Arguments
- $1 (boolean): Defines if dba user creation script should be executed. [Default: true]
create_db_login_and_user
Creates a database login and user for the provided user, and assigns the user to the provided role. For more information, see db_users
Arguments
- $1 (string): The database user name
- $2 (string): The database role name
add_user_to_role
Assigns the user to the provided role. For more information, see create_db_login_and_user.
Arguments
- $1 (string): The database user name
- $2 (string): The database role name
Execution Utilities
Use an ephemeral ZK client container to change ZK digest password.
change_zk_password
Use an ephemeral ZK client container to change ZK digest password.
Arguments
- $1 (string): The old password
- $2 (string): The new password
run_solr_client_command
Use an ephemeral Solr client container to run commands against Solr. For more information about the environment variables and volume mounts that are required for the Solr client, see Running a Solr client container. The run_solr_client_command function takes the command you want to run as an argument. For more information about commands you can execute using the Solr zkcli, see Solr ZK Command Line Utilities
Example
run_solr_client_command "/opt/solr/server/scripts/cloud-scripts/zkcli.sh" -zkhost "${ZK_HOST}" -cmd clusterprop -name urlScheme -val https
Arguments
- ... (string): The command you want to run on the Solr client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_i2_analyze_tool
Run an ephemeral i2 Analyze Tool container to run the i2 Analyze tools.
Example
run_i2_analyze_tool "/opt/i2-tools/scripts/updateSecuritySchema.sh"
Arguments
- ... (string): Command you want to run on the i2 Analyze Tool container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_i2_analyze_tool_as_external_user
Use an ephemeral i2 Analyze Tool container to run commands against the i2 Analyze service via the load balancer as an external user. The container contains the required secrets to communicate with the i2 Analyze service from an external container. For example, if you would like to send a Curl request to the load balancer stats endpoint, run:
Example
run_i2_analyze_tool_as_external_user bash -c "curl \
--silent \
--cookie-jar /tmp/cookie.txt \
--cacert /tmp/i2acerts/CA.cer \
--request POST \"${INTERNAL_FRONT_END_URI}/j_security_check\" \
--header 'Origin: ${INTERNAL_FRONT_END_URI}' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'j_username=Jenny' \
--data-urlencode 'j_password=Jenny' \
&& curl \
--silent \
--cookie /tmp/cookie.txt \
--cacert /tmp/i2acerts/CA.cer\
\"${INTERNAL_FRONT_END_URI}/api/v1/admin/indexes/status\""
Arguments
- ... (string): Command you want to run on the i2 Analyze Tool container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_sql_server_command_as_etl
Uses an ephemeral SQL Client container to run database scripts or commands against the Information Store database as the etl user. For more information about running a SQL Client container and the environment variables required for the container, see SQL Client.
Example
run_sql_server_command_as_etl bash -c "/opt/mssql-tools/bin/sqlcmd -N -b -S \${DB_SERVER} -U \${DB_USERNAME} -P \${DB_PASSWORD} -d \${DB_NAME} -Q
\"BULK INSERT IS_Staging.E_Person
FROM '/var/i2a-data/law-enforcement-data-set-2-merge/person.csv'
WITH (FORMATFILE = '/var/i2a-data/law-enforcement-data-set-2-merge/sqlserver/format-files/person.fmt', FIRSTROW = 2)\""
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_postgres_server_command_as_etl
Uses an ephemeral Postgres Client container to run database scripts or commands against the Information Store database as the etl user. For more information about running a Postgres Client container and the environment variables required for the container, see Postgres Client.
Example
run_postgres_server_command_as_etl bash -c "/usr/lib/postgresql/bin/psql -w -X -q --set=client_min_messages=warning -h \${DB_SERVER} -p \${DB_PORT} -d \${DB_NAME} -c
\"COPY IS_Staging.E_Person (source_id, p_description_of_mark, p_accent, p_aka, p_build, p_citizenship, p_date_of_birth,
p_description, p_identification_number, p_eye_color,p_facial_hair, p_first_given_name, p_hair_color, p_hair_type,
p_height_from, p_height_to, p_family_name, p_middle_name, p_additional_informatio, p_occupation, p_unique_reference,
p_gender, source_ref_source_location, source_ref_source_type, source_ref_source_image_url)
FROM '/var/i2a-data/law-enforcement-data-set-2-merge/person.csv' CSV HEADER ENCODING 'UTF8' NULL AS ''\""
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_postgres_server_command_as_first_start_postgres
Uses an ephemeral Postgres Client container to run database scripts or commands against the Information Store database as the postgres user with the initial postgres password. For more information about running a Postgres Client container and the environment variables required for the container, see Postgres Client.
Arguments
- ... (string): Command you want to run on the Postgres Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_postgres_server_command_as_postgres
Uses an ephemeral Postgres Client container to run database scripts or commands against the Information Store database as the postgres user. For more information about running a Postgres Client container and the environment variables required for the container, see Postgres Client.
Example
run_postgres_server_command_as_postgres "/opt/i2-tools/scripts/database-creation/runStaticScripts.sh"
Arguments
- ... (string): Command you want to run on the Postgres Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_postgres_server_command_as_dba
Uses an ephemeral Postgres Client container to run database scripts or commands against the Information Store database as a dba user. For more information about running a Postgres Client container and the environment variables required for the container, see Postgres Client.
Example
run_postgres_server_command_as_dba "/opt/i2-tools/scripts/database-creation/runStaticScripts.sh"
Arguments
- ... (string): Command you want to run on the Postgres Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_postgres_server_command_as_dbb
Uses an ephemeral Postgres Client container to run database scripts or commands against the Information Store database as the dbb (the backup operator) user. For more information about running a Postgres Client container and the environment variables required for the container, see Postgres Client.
Example
run_postgres_server_command_as_dbb bash -c "pg_dump '/backup/istore.pgb'"
Arguments
- ... (string): Command you want to run on the Postgres Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_sql_server_command_as_first_start_sa
Uses an ephemeral SQL Client container to run database scripts or commands against the Information Store database as the sa user with the initial SA password. For more information about running a SQL Client container and the environment variables required for the container, see SQL Client.
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_sql_server_command_as_sa
Uses an ephemeral SQL Client container to run database scripts or commands against the Information Store database as the SA user. For more information about running a SQL Client container and the environment variables required for the container, see SQL Client.
Example
run_sql_server_command_as_sa "/opt/i2-tools/scripts/database-creation/runStaticScripts.sh"
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_sql_server_command_as_dba
Uses an ephemeral SQL Client container to run database scripts or commands against the Information Store database as the dba user. For more information about running a SQL Client container and the environment variables required for the container, see SQL Client.
Example
run_sql_server_command_as_dba "/opt/i2-tools/scripts/clearInfoStoreData.sh"
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_sql_server_command_as_dbb
Uses an ephemeral SQL Client container to run database scripts or commands against the Information Store database as the dbb (the backup operator) user. For more information about running a SQL Client container and the environment variables required for the container, see SQL Client.
Example
run_sql_server_command_as_dbb bash -c "/opt/mssql-tools/bin/sqlcmd -N -b -C -S sqlserver.eia,1433 -U \"\${DB_USERNAME}\" -P \"\${DB_PASSWORD}\" \
-Q \"USE ISTORE;
BACKUP DATABASE ISTORE
TO DISK = '/backup/istore.bak'
WITH FORMAT;\""
Arguments
- ... (string): Command you want to run on the SQL Client container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_etl_toolkit_tool_as_i2_etl
Uses an ephemeral ETL toolkit container to run ETL toolkit tasks against the Information Store using the i2 ETL user credentials. For more information about running the ETL Client container and the environment variables required for the container, see ETL Client. For more information about running the ETL toolkit container and the tasks that you can run, see ETL Tools
Example
run_etl_toolkit_tool_as_i2_etl bash -c "/opt/i2/etltoolkit/addInformationStoreIngestionSource --ingestionSourceName EXAMPLE_1 --ingestionSourceDescription EXAMPLE_1"
Arguments
- ... (string): Command you want to run on the ETL Toolkit container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
run_etl_toolkit_tool_as_dba
Uses an ephemeral ETL toolkit container to run ETL toolkit tasks against the Information Store using the DBA user credentials. For more information about running the ETL Client container and the environment variables required for the container, see ETL Client. For more information about running the ETL toolkit container and the tasks that you can run, see ETL Tools
Example
run_etl_toolkit_tool_as_dba bash -c "/opt/i2/etltoolkit/addInformationStoreIngestionSource --ingestionSourceName EXAMPLE_1 --ingestionSourceDescription EXAMPLE_1"
Arguments
- ... (string): Command you want to run on the ETL Toolkit container
Exit codes
- 0: If command was executed successfully.
- 1: If command was NOT executed successfully.
Volume utilities
Uses an ephemeral Red Hat UBI Docker image to update a volume with the contents of another volume directory. For example, to update the the configuration directory in the prometheus_config volume with the content of the config volume /.configuration/prometheus directory, run:
update_volume_from_volume
Uses an ephemeral Red Hat UBI Docker image to update a volume with the contents of another volume directory. For example, to update the the configuration directory in the prometheus_config volume with the content of the config volume /.configuration/prometheus directory, run:
Example
update_volume_from_volume "config" "prometheus" "prometheus_config" "/prometheus"
Arguments
- $1 (string): From volume name.
- $2 (string): From volume location.
- $3 (string): To volume name.
- $4 (string): To volume location.
- $5 (string): Optional, defaults to true. Flag to determine if we should follow symlinks.
Volume utilities
Uses an ephemeral Red Hat UBI Docker image to update a volume with the contents of a local directory. For example, to update the the run/secrets directory in the liberty1_secrets volume with the content of the local /environment-secrets/simulated-secret-store directory, run:
update_volume
Uses an ephemeral Red Hat UBI Docker image to update a volume with the contents of a local directory. For example, to update the the run/secrets directory in the liberty1_secrets volume with the content of the local /environment-secrets/simulated-secret-store directory, run:
Example
update_volume "/environment-secrets/simulated-secret-store" "liberty1_secrets" "/run/secrets"
Arguments
- $1 (string): The local directory on your machine.
- $2 (string): The volume name.
- $3 (string): The directory inside the volume.
- $4 (string): Optional, defaults to current user id. The user id.
- $5 (string): Optional, defaults to current user id. The group id.
- $6 (string): Optional, defaults to true. Flag to determine if we should follow symlinks.
get_volume
Uses an ephemeral Red Hat UBI Docker image to update a local directory with the contents of a specified volume. For example, to get the contents from the run/secrets directory in the liberty1_secrets volume into your local /environment-secrets/simulated-secret-store directory, run:
Example
get_volume "/environment-secrets/simulated-secret-store" "liberty1_secrets" "/run/secrets"
Arguments
- $1 (string): The local directory on your machine.
- $2 (string): The volume name.
- $3 (string): The directory inside the volume.