Support of X-Pack for Elasticsearch

This applies to: Visual Data Discovery

Symphony allows you to connect to your Elasticsearch cluster and data set secured by X-Pack.

X-Pack is an add-on offering for Elasticsearch 7 that aims at securing the data on your cluster, and is included in Elasticsearch 8. Learn more about X-Pack.

Configure Cluster or Index Privileges for a User

To connect to the Elasticsearch cluster, you need to create an Elasticsearch user and configure the access privileges for this user.

The access permissions for the Elasticsearch user determine the scope of the data available for querying by Symphony users.

To work with Elasticsearch data, use X-Pack to grant the following minimal access privileges to the Elasticsearch user:

Monitor privileges for Elasticsearch Cluster
Manage (to get the metadata) and Read (to read data) privileges for Index
Reference Support Matrix

After the Elasticsearch user permissions are configured, you can proceed with connecting to a data source.

Added Libraries Required to Connect Using a Transport Protocol

Symphony extracts specific libraries needed to support secured connections to Elasticsearch clusters over a transport protocol. The Symphony Elasticsearch connector starts and works normally without these libraries, except when you want to use secured transport connections. To use secured transport connections, you must download and enable these libraries. If you attempt to establish a secured transport connection to an Elasticsearch cluster without the required libraries, an error will occur when you try to validate the connection.

To download and enable the transport-required libraries:

Download the required libraries to /opt/zoomdata/lib/edc-elasticsearch-<x.x>/ for Linux, and for <install-path>/lib/edc-elasticsearch-<x.x>/ Windows, where <x.x> is the version of Elasticsearch.

mkdir -p /opt/zoomdata/lib/edc-elasticsearch-<x.x>
wget -P /opt/zoomdata/lib/edc-elasticsearch-<x.x> <library URL>

The following table provides a list of the required libraries for each supported Elasticsearch version, with URLs:

Symphony Connector	Library Name	Version	License	URL
Elasticsearch 7.0	org.elasticsearch.client:x-pack-transport	7.0.0	Commercial Software End User License Agreement (https://www.elastic.co/eula)	https://artifacts.elastic.co/maven/org/elasticsearch/client/x-pack-transport/7.0.0/x-pack-transport-7.0.0.jar
	org.elasticsearch.plugin:x-pack-core	7.0.0		https://artifacts.elastic.co/maven/org/elasticsearch/plugin/x-pack-core/7.0.0/x-pack-core-7.0.0.jar
	com.unboundid:unboundid-ldapsdk	3.2.0	GPLv2, LGPLv2.1, or UnboundID Free Use License (https://docs.ldap.com/ldap-sdk/docs/LICENSE-UnboundID-LDAPSDK.txt)	https://mvnrepository.com/artifact/com.unboundid/unboundid-ldapsdk/3.2.0

For example:

mkdir -p /opt/zoomdata/lib/edc-elasticsearch-7.0
wget -P /opt/zoomdata/lib/edc-elasticsearch-7.0 https://artifacts.elastic.co/maven/org/elasticsearch/client/x-pack-transport/7.0.0/x-pack-transport-7.0.0.jar

wget -P /opt/zoomdata/lib/edc-elasticsearch-7.0 https://artifacts.elastic.co/maven/org/elasticsearch/plugin/x-pack-api/7.0.0/x-pack-api-7.0.0.jar

wget -P /opt/zoomdata/lib/edc-elasticsearch-7.0 http://central.maven.org/maven2/com/unboundid/unboundid-ldapsdk/3.2.0/unboundid-ldapsdk-3.2.0.jar

Update the datasource.driver-config.jar-path property value in the appropriate Elasticsearch connector property file to point to the directory path containing all the libraries you downloaded or to a comma-separated list combining all library paths. For example:
```
datasource.driver-config.jar-path=/opt/zoomdata/lib/edc-elasticsearch-7.0
```
or
```
datasource.driver-config.jar-path=/opt/zoomdata/lib/edc-elasticsearch-7.0/x-pack-transport-7.0.0.jar,
/opt/zoomdata/lib/edc-elasticsearch-7.0/x-pack-api-7.0.0.jar,
/opt/zoomdata/lib/edc-elasticsearch-7.0/unboundid-ldapsdk-3.2.0.jar
```
See Connector Properties and Property Files to determine the correct Elasticsearch property file to use and where to save it.

insightsoftware discourages changing properties in the /opt/zoomdata/conf directory (Linux) or <install-path>/conf (Windows). Copy the files you want to change to the /etc/zoomdata directory (Linux) or <install-path>/conf-modify (Windows) and change them there. This will ensure that your changes are not overwritten when Symphony is next upgraded.

Quickly determine what changes you've made to a properties file using diff in Linux. For example:
```
diff /opt/zoomdata/conf/edc-<connector-name>.properties /etc/zoomdata/<edc-<connector-name>.properties
```
or
```
diff /opt/zoomdata/conf/zoomdata.properties /etc/zoomdata/zoomdata.properties
```
For Windows environments, use your preferred diff utility to compare the differences between your original and updated property files.

Restart the Elasticsearch connector microservice. For example:

sudo systemctl restart zoomdata-edc-elasticsearch-7.0

./bootstrap-composer.ps1 -ServicesAction restart

Connection Via HTTP or Transport Protocol and Using SSL

You can connect to your Elasticsearch cluster using either HTTP or transport protocols. SSL is optional for the HTTP connection but is required for transport connections when connecting to an X-Pack secured Elasticsearch cluster.