Adding connectors to your development environment

To add a connector to your deployment of i2 Analyze, you must develop an i2 Connect connector.

Ensure that the DEPLOYMENT_PATTERN variable in the <config_name>/utils/variables.conf file is set to a pattern that includes the i2 Connect gateway.
For example:

DEPLOYMENT_PATTERN="i2c_istore"

For information about developing your own connectors, see i2 Connect gateway connector development.

Note: You can use Connector Designer to create simple connectors through a graphical user interface. When you use Connector Designer, you do not need to follow the instructions below. To open Connector Designer, navigate to https://i2analyze.eia:9443/connector-designer/connectors. For more information about using it, see Connector Designer.

Process overview:

Create a custom connector image
Update the configuration to reference connectors
Build connector images and redeploy

Creating a connector image

Create a connector image from one of the image templates. You can deploy connectors using the following templates:

Spring Boot for Java based connectors.
- For information about developing connectors in Java, see analyze-connect.
i2 Connect server for Node v18 based connectors developed using the i2 Connect server SDK.
- For information about developing connectors using the SDK, see analyze-connect-node-sdk.
NodeJS for Node v18 based connectors.
User managed for connectors that are not managed by the config development environment. The connectors are managed manually outside of the environment.

Connector runtime expectations

The connector must be secured and listen on port 3443. For more information about securing your system, see Securing i2 Analyze.

Connectors that are not user-managed will be run in a Docker container with the following environment:

The identifier of the connector (as defined in the connector-definition.json file) will be provided in environment variable CONNECTOR_ID.
If the connector should log additional debug information then environment variable DEBUG will be set to true.
Environment variables SSL_... will be set to configure SSL for the connector.
- SSL_ENABLED will be set to true
- SSL_PRIVATE_KEY, SSL_CERTIFICATE, SSL_CA_CERTIFICATE, SSL_ADDITIONAL_TRUST_CERTIFICATES and SSL_GATEWAY_CN will be set to the contents of the respective files.
- SSL_SERVER_PORT will be set to 3443

Connectors serving up resources via HTTP

Some connectors' functionality depends on their ability to construct URLs that are answered by the connector itself. This requires the connector to know the URL that the proxy is looking for to route to the connector. The following environment variables are provided to connectors to support this:

Environment variable FRONT_END_URI_PATH may be set to the path portion of the URL that will pass through the proxy and arrive at the connector. The connector must determine the host, port and protocol parts of the URL from HTTP headers it receives.

e.g. /connectors/$CONNECTOR_ID/api.
Environment variable FRONT_END_URI may be set to complete a URL that will pass through the proxy and arrive at the connector. If this is set then the connector should ignore FRONT_END_URI_PATH.
Environment variable JWT_CERTIFICATE_FILE will be set to the contents of the JWT certificate that the connector can use to validate JWT tokens sent by the gateway.

In most cases, the FRONT_END_URI_PATH will be sufficient for the system to work; the haproxy will provide the X-Forwarded-... headers that the connector can use to determine the original protocol, host and port without the need for the deployer to set them explicitly.

If the proxy chain between client and connectors does not permit this then the deployer should set the CONNECTORS_BASE_URI variable in the variables.conf file to a base URL. If this variable is set then the connector ID and /api will be appended and the result passed in to connectors as the FRONT_END_URI variable.

e.g. if CONNECTORS_BASE_URI is set to https://mycompany.com:1234/connectors/ then the connector will receive FRONT_END_URI set to https://mycompany.com:1234/connectors/$CONNECTOR_ID/api ... and the deployer must ensure that this URL is routed to the connector.

Connectors persisting data

Some connectors may require persistent storage that is not lost when the connector container is recreated. In this case the deployer must either set the CONNECTOR_STORAGE_DIRECTORY or set the CONNECTOR_STORAGE_VOLUME variable in the variables.conf file.

CONNECTOR_STORAGE_DIRECTORY

This is a path on the host machine's filesystem.

If this is set then the $CONNECTOR_ID subdirectory within that directory will be mounted as a volume into the connector container and the CONNECTOR_STORAGE_ROOT_LOCATION environment variable will be passed in to provide the path where the volume is mounted within the container.

If the directory (or subdirectory) does not exist they will be created during deployment.

Note: post-deployment, this area should be treated as "owned" by the connectors. If the containers run (internally) as a different user-id to the one being used to run the deployment scripts then you may encounter file permission/ownership issues if you try to access these files directly. e.g. you may need to use sudo to delete them.
CONNECTOR_STORAGE_VOLUME

This is the name of a Docker volume.

If this is set then the $CONNECTOR_ID subdirectory within that volume will be mounted in the connector container and the CONNECTOR_STORAGE_ROOT_LOCATION environment variable will be passed in to provide the path where the volume is mounted within the container.

If the volume or subdirectory does not exist they will be created during deployment.

Spring Boot based connector image

Copy the /templates/springboot-connector-image directory (including the sub-directories) to the /connector-images directory.
Rename the springboot-connector-image folder to a custom connector name. The name of the directory is used to determine which connectors to deploy. Do not use spaces or special characters in the name.
The directory structure is:
```
- connector-images
    - <connector_name>
        - target
        - connector-definition.json
        - connector-version.json
        - ...
```
Copy your connector code into the target directory of the connector directory. For example, copy your .jar file into the /connector-images/<connector_name>/target directory.

Next, configure your connector.

i2 Connect server based connector image

Copy the /templates/i2connect-server-connector-image directory (including the sub-directories) to the /connector-images directory.
Rename the i2connect-server-connector-image folder to a custom connector name. The name of the directory is used to determine which connectors to deploy. Do not use spaces or special characters in the name.
The directory structure is:
```
- connector-images
    - <connector_name>
        - app
        - connector-definition.json
        - connector-version.json
        - ...
```
Copy the i2 Connect server connector distributable into the connector directory. For example, copy your .tar.gz or .tgz file into the /connector-images/<connector_name> directory.

Next, configure your connector.

NodeJS based connector image

Copy the /templates/node-connector-image directory (including the sub-directories) to the /connector-images directory.
Rename the node-connector-image folder to a custom connector name. The name of the directory is used to determine which connectors to deploy. Do not use spaces or special characters in the name.
The directory structure is:
```
- connector-images
    - <connector_name>
        - app
        - connector-definition.json
        - connector-version.json
        - ...
```
Copy your connector code into the app directory of the connector directory. For example, copy the files into the /connector-images/<connector_name>/app directory. By default, the image will start node with the file app.js.

Next, configure your connector.

User managed connector

Copy the /templates/user-managed-connector directory to the /connector-images directory.
Rename the user-managed-connector folder to a custom connector name. The name of the directory is used to determine which connectors to deploy. Do not use spaces or special characters in the name.
The directory structure is:
```
- connector-images
    - <connector_name>
        - connector-definition.json
```
To configure SSL on the external connector, you must do the following:

Below is the example configuration for the i2 Connect server based connector running externally.

Option 1: Use certificates generated by ADT.

You can run manage-environment -t connectors command to generate the required certificates and keys.

The script will generate the required certificates and keys in the ./environment-secrets/generated-secrets/certificates/<connector_name> directory. Once the certificate and the key are generated update your external configuration with the following values:
```
SSL_ENABLED="true"
SSL_SERVER_PORT="<port>"
SSL_PRIVATE_KEY_FILE="./environment-secrets/generated-secrets/certificates/<connector_name>/server.key"
SSL_CERTIFICATE_FILE="./environment-secrets/generated-secrets/certificates/<connector_name>/server.cer"
SSL_CA_CERTIFICATE_FILE="./environment-secrets/generated-secrets/certificates/externalCA/CA.cer"
SSL_GATEWAY_CN='gateway.user'
```
Option 2: Generate the required certificate and key using your own Certificate Authority (CA).

Once the certificate and the key are generated update your external configuration with the following values:
```
SSL_ENABLED="true"
SSL_SERVER_PORT="<port>"
SSL_PRIVATE_KEY_FILE="<path_to_your_private_key>"
SSL_CERTIFICATE_FILE="<path_to_your_certificate>"
SSL_CA_CERTIFICATE_FILE="<path_to_you_certificate_authority>"
SSL_GATEWAY_CN='["gateway.user","<cn_of_the_client_cert_used_outside_of_ADT>"]'
```
Note: With this option you will need to add additional trust certificates to each configuration that uses this external connector.

Important:

Ensure SSL_GATEWAY_CN includes the ADT gateway username (gateway.user).

Ensure SSL_CA_CERTIFICATE_FILE includes the ./environment-secrets/generated-secrets/certificates/externalCA/CA.cer file contents and, if using option 2, append your own CA certificate.

This is to ensure that the connector trusts the ADT gateway client certificate.

Next, configure your connector.

Configuring a connector

Populate the values in the <connector_name>/connector-definition.json file with information about your connector.

Key	Description
`id`	An identifier that is unique for all connectors that will be deployed.
`name`	A name for the connector that is displayed to users in the client.
`configurationPath`	The full path to the configuration endpoint of the connector. By default, it is `config`.
`gatewaySchema`	The short name of an optional gateway schema. When no gateway schema is used, do not provide a value. Add the referenced schema and charting scheme to the `gateway-schemas` directory.
`type`	For i2 Connect server connectors, set to `i2connect-server`. For any other connectors, the `type` key is not required.
`userManaged`	Set to `true` for connectors that are managed outside of the environment. For any other connectors, the `userManaged` key is not required.
`baseUrl`	Used only for connectors with `userManaged` set to `true`. The `baseUrl` value is the URL address of the connector. For any other connectors, the `baseUrl` key is not required.
`sendSensitiveHeaders`	This setting effectively disables connectors that employ user-specific configuration.
`imageName`	When specified, the connector image is tagged with a custom name during the build process and uses this name, along with the tag specified in the version file, when deploying the image. Ensure to include the registry name if different than default DockerHub.

For example:

{
    "id": "connector1",
    "name": "Connector 1",
    "description": "First connector",
    "configurationPath": "/config",
    "gatewaySchema": "template",
    "sendSensitiveHeaders": "false",
    "type": "i2connect-server"
}

{
    "id": "connector2",
    "name": "Connector 2",
    "description": "Second connector",
    "configurationPath": "/config",
    "gatewaySchema": "template",
    "sendSensitiveHeaders": "false",
    "userManaged": true,
    "baseUrl": "https://example-connector:3443" 
}

Notes:

If your user managed connector is hosted in a Docker container on a different network to your config development environment, connect the Liberty container to the same network as your user managed connector container. For example, run docker network connect <connector-network-name> liberty1.<config-name>.
If your connector is running on your host machine's operating system, use the host.docker.internal as the host in the baseUrl value. For example, "http://host.docker.internal:3000"
If your connector is running in WSL, use the IP address for WSL as the host in the baseUrl value. For example, "http://172.23.0.164:3000".
- You can find the IP address of your current WSL distribution by running ip addr show eth0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}'. Note that the IP address changes every time that WSL is shutdown and restarted.

Provide the version of the connector in the <connector_name>/connector-version.json file. For user managed connectors, this file is not required. The version value specifies the version of the connector.

For example:
```
{
    "version": "0.0.1",
    "tag": "0-0-1"
}
```

For more information about configuring connectors, see Managing connectors.

Adding connectors to a config

In the configuration you want to deploy your connector with, update the configs/<config_name>/configuration/connector-references.json file and add the directory name for your connector in the connectors array.
For example, to add the connector in /connector-images/example-connector the connector-references.json file contains the following connectors array:
```
{
    "connectors": [
        {
            "name": "example-connector"
        }
    ],
...
}
```

Adding additional trust certificates for external connectors (if SSL is enabled with custom CA)

If you have enabled SSL on the external connector and used your own Certificate Authority (CA), you must provide this CA in the configuration. To do that, create configs/<config_name>/configuration/secrets/additional-trust-certificates.cer file and copy the CA certificate into it. This file is a text file in OpenSSL PEM format, i.e. certificates should start "-----BEGIN CERTIFICATE-----" and can hold multiple certificates.

Defining a gateway schema (Optional)

You can develop the gateway schema and associated charting schemes by following the instructions in Developing schemas in your configuration development environment. After you develop the schemes, complete the following steps to deploy them with your configuration:

Copy the schema and charting scheme files to the /gateway-schemas directory.
To define the short name of the gateway schema, the prefix of the file name is used. The convention is defined as: <short>-<name>-schema.xml and <short>-<name>-schema-charting-scheme.xml.
For example, files with the names law-enforcement-schema.xml and law-enforcement-schema-charting-schemes.xml have the short name law-enforcement.
Add the short name of your schema to the configs/<config_name>/configuration/connector-references.json file in the gatewaySchemas array.
For example, to add a gateway schema with the short name law-enforcement, the file contains the following gatewaySchemas array:
```
{
    ...
    ],
    "gatewaySchemas": [
        {
            "shortName": "law-enforcement"
        }
    ]
}
```
Note: If you are not using any gateway schemas, do not remove the gatewaySchemas key from the file.
Set the short name of your schema in the <connector_name>/connector-definition.json as the value for the gatewaySchema key of the connector that will use the gateway schema.

Building the connector images and redeploying

Run the deploy command to generate the secrets for your connectors, build the connector images, and deploy your environment.
```
deploy
```
If you have multiple configs, specify the config name with -c <config-name>.

After you add connectors to your environment, you can configure the rest of the configuration.