Opensearch Configuration

dotCMS requires an OpenSearch node or cluster to index and search the stored content, the file metadata and also in order to perform site wide searches. It is important to note that dotCMS does not automatically manage the OpenSearch cluster - you must perform management functions yourself to ensure proper operation of the OpenSearch infrastructure. That said, generally speaking, an OpenSearch cluster that is dedicated to dotCMS does not require much care or feeding aside from consuming memory and disk space and many dotCMS instances run happily for years against a single node installation or container given enough disk space.

Configuration#

dotCMS reads two sets of connection settings, one per cluster:

If you have not started the migration, only the first set is read and nothing about your configuration needs to change. The migration is switched on with DOT_FEATURE_FLAG_OPEN_SEARCH_PHASE, which is absent by default; an absent or unrecognised value means the migration has not started.

Your Existing Cluster#

To connect dotCMS to your current cluster you should specify at least 3 environmental variables:

DOT_ES_ENDPOINTS=https://es1.dotcmssite.com:9200
DOT_ES_AUTH_BASIC_USER=admin
DOT_ES_AUTH_BASIC_PASSWORD={yourSecretPassword}

• DOT_ES_ENDPOINTS takes a comma separated list of urls that it will connect and load balance search requests to. The property can look like this

"DOT_ES_ENDPOINTS": 'https://es1.dotcmssite.com:9200'

or contain multiple entries, such as

"DOT_ES_ENDPOINTS": 'https://es1.dotcmssite.com:9200,https://es2.dotcmssite.com:9200,https://es3.dotcmssite.com:9200'

• DOT_ES_AUTH_BASIC_USER is the basic auth username that will be sent and used to perform search and indexing operations (defaults to admin)

• DOT_ES_AUTH_BASIC_PASSWORD is the basic auth password. Best practices are to use a secret store mechanism when providing these to your container.

Other, less well used options include

• DOT_ES_AUTH_TYPE, which defaults to BASIC but can also be set to JWT for JWT based authentication.
• DOT_ES_AUTH_JWT_TOKEN which can be used to pass the JWT auth token (again, use secrets).

If DOT_ES_ENDPOINTS is not set, dotCMS assembles one endpoint from DOT_ES_PROTOCOL, DOT_ES_HOSTNAME and DOT_ES_PORT, which default to https, localhost and 9200.

The OpenSearch 3.x Cluster#

The new cluster is configured the same way, under DOT_OS_*:

DOT_OS_ENDPOINTS=https://os1.dotcmssite.com:9200
DOT_OS_AUTH_BASIC_USER=dotcms
DOT_OS_AUTH_BASIC_PASSWORD={yourSecretPassword}

DOT_OS_ENDPOINTS behaves exactly as its counterpart does: a comma separated list, load balanced across. If it is not set, dotCMS assembles one endpoint from DOT_OS_PROTOCOL, DOT_OS_HOSTNAME and DOT_OS_PORT, with the same https / localhost / 9200 defaults.

DOT_OS_AUTH_TYPE defaults to BASIC and also accepts JWT and CERT. Under CERT, dotCMS authenticates with a client certificate rather than a credential, and you supply DOT_OS_TLS_CLIENT_CERT and DOT_OS_TLS_CLIENT_KEY instead of a username and password. Only one authentication method may be configured at a time — basic, JWT or certificate — and dotCMS rejects a configuration that supplies more than one.

TLS for the new cluster is configured under DOT_OS_TLS_ENABLED, DOT_OS_TLS_TRUST_SELF_SIGNED, DOT_OS_TLS_CERT_REQUIRED and DOT_OS_TLS_CA_CERT, all of which default to off or empty. Note that trusting a self-signed certificate is a separate setting from enabling TLS; enabling TLS against a self-signed certificate without also trusting it fails, and the error reads like a connection problem rather than a certificate one.

Settings the New Cluster Inherits#

Most DOT_OS_* settings fall back to their DOT_ES_* counterpart when you do not set them, so a working configuration usually only needs the endpoints and whatever differs. Authentication, TLS enablement, index tuning, scroll and bulk sizing, and query behaviour all resolve this way: dotCMS reads the OS key, then the ES key, then the built-in default.

Two consequences are worth knowing before you rely on it.

The endpoints do not inherit. DOT_OS_ENDPOINTS has no fallback, by design — the two clusters have to be different machines, so inheriting the address would defeat the purpose.

Credentials do inherit, and dotCMS ships ES_AUTH_BASIC_USER=admin and ES_AUTH_BASIC_PASSWORD=admin as property defaults. If you set DOT_OS_ENDPOINTS and nothing else, dotCMS will present your existing cluster's credentials to the new one. That is usually what you want. If it is not — if the new cluster has its own service account, which is the recommended arrangement — set DOT_OS_AUTH_BASIC_USER and DOT_OS_AUTH_BASIC_PASSWORD explicitly rather than letting them fall through.

The settings with no Elasticsearch equivalent, and therefore no fallback, are the mTLS certificate paths (DOT_OS_TLS_CLIENT_CERT, DOT_OS_TLS_CLIENT_KEY, DOT_OS_TLS_CA_CERT), the self-signed and certificate-required trust settings, and the connection pool tuning (DOT_OS_CONNECTION_TIMEOUT, DOT_OS_SOCKET_TIMEOUT, DOT_OS_MAX_CONNECTIONS, DOT_OS_MAX_CONNECTIONS_PER_ROUTE). These take their built-in defaults unless you set them.

Supported Versions#

dotCMS asserts the major version of the new cluster when the migration is switched on, and treats a mismatch as fatal rather than degrading. OpenSearch 2.x is not supported on either connection — dotCMS moved from 1.3.x to 3.x directly and 2.x was never a target.

Once the migration is complete, the DOT_ES_* settings are no longer read, and dotCMS no longer requires the old cluster to be reachable or even to exist.

Versioning History#

dotCMS formerly deployed Elasticsearch. As such, many references to Elasticsearch can still be seen in system variables, REST API endpoints, etc.

In 2021 and at version 7.10, Elastic.co changed Elasticsearch's licensing from an Apache 2.0 (pure open source) license to a proprietary license that precludes other companies from offering a hosted/managed version of Elasticsearch software. This was aimed primarily at Amazon, who had been offering an AWS managed Elasticsearch service without paying any royalties to Elastic.co. After Elastic.co changed the Elasticsearch license, Amazon forked the original open source Elasticsearch project and created a competing version of Elasticsearch software licensed under the Apache 2.0 that they have dubbed "OpenSearch" (https://opensearch.org/). Amazon now uses the OpenSearch project in all their managed "Elasticsearch" offerings.

Due to the Elasticsearch licensing change, dotCMS decided to standardize on Amazon's OpenSearch version of Elasticsearch licensed under the Apache 2.0 license rather than Elastic.co's proprietary Elasticsearch version. In doing this, dotCMS upgraded the Elasticsearch drivers in our software to the last open source version of Elasticsearch provided by Elastic.co, version 7.10.2. As of this writing, we believe dotCMS continues to be compatible with all known versions of Elasticsearch > 7.1.0 whether they are offered by Elastic.co or Amazon but test and support only Amazon's OpenSearch. Moving forward, dotCMS will continue to be compatible with OpenSearch as the OpenSearch versions progress but cannot guarantee that we will maintain version compatibility with Elastic.co's Elasticsearch offerings if they choose to include breaking changes.

That 7.10.2 driver is what the DOT_ES_* connection still uses. The DOT_OS_* connection uses OpenSearch's own client library instead, which is why the two have separate settings and why the new cluster's version is checked rather than assumed.

Current Version#

dotCMS currently uses OpenSearch 1.3.x, and is migrating to OpenSearch 3.x. OpenSearch 2.x is not supported.