The source code and default configuration of the Building Block is available in our code.syseleven.de. Infos on release notes and new features please follow Release notes OpenSearch
With this Building Block SysEleven provides OpenSearch, an open source search and analytics suite that delivers full-text search, real-time indexing, and advanced analytics for scalable data insights. It consists of
Kubernetes resources are defined using the official Helm charts.
Proceed with the following prerequisite description to use the Building Block out of the box.
For the basic OpenSearch cluster deployed by the Building Block as default a resource overview is listed in the table below.
| CPU/vCPU | Memory |
|---|---|
| 5 | 8.5 GB |
Adjust available resources based on the resource requirements of the OpenSearch cluster nodes. For reliability reasons OpenSearch instances of the same kind should not reside on the same Kubernetes node.
Add the directory opensearch to your control repository. Add a .gitlab-ci.yml to the directory with the following content:
include:
- project: syseleven/building-blocks/helmfiles/opensearch
file: JobDevelopment.yaml
ref: 0.15.0
- project: syseleven/building-blocks/helmfiles/opensearch
file: JobStaging.yaml
ref: 0.15.0
- project: syseleven/building-blocks/helmfiles/opensearch
file: JobProduction.yaml
ref: 0.15.0Remove environments you are not using by removing their include.
To choose between major version 2.x and 3.x of OpenSearch and OpenSearch Dashboards set the environment variables OPENSEARCH_MAJOR_VERSION and OPENSEARCH_DASHBOARDS_MAJOR_VERSION to the desired version, the default is 2 for both. This will install the latest version of the chosen branch that the Building Block provides.
No configuration is required.
When the Building Block is used with its defaults, it deploys an OpenSearch cluster with 3 nodes with the role "cluster_manager" and "data". Each has resource requests of 500m CPU and 2 GB memory (request and limit) and 10 GB volume size. As this cluster size is sufficient for general overview and to test the capabilities of the Operator, it is probably undersized. To meet performance requirements a cluster is customized by overwriting the nodePools in values-opensearch-cluster.yaml
cluster:
nodePools:
- component: "nodes"
diskSize: "10Gi"
replicas: 3
roles:
- "cluster_manager"
- "data"
resources:
requests:
memory: "2Gi"
cpu: "500m"
limits:
memory: "2Gi"
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
opster.io/opensearch-cluster: "opensearch-cluster"
opster.io/opensearch-nodepool: "nodes"
topologyKey: kubernetes.io/hostname
By default the OpenSearch cluster is created with default admin credentials "admin:admin". To set strong passwords, the internal_users.yaml of the Security plugin have to be overwritten to create admin and dashboards accounts. Also, the OpenSearch operator needs to know about those credentials to authenticate API requests therefor it is required to prepare secrets containing them. This is done in the values file values-opensearch-extension.yaml of the opensearch extension chart
securityconfig:
internal_users.yml: |-
_meta:
type: "internalusers"
config_version: 2
admin:
hash: "$2a$12$rJeFw143s15v.92cSCPyJu.bhtcym0Q7oY024P8llRIk0sFP0bTwS"
reserved: true
backend_roles:
- "admin"
description: "Admin user"
dashboards:
hash: "$2a$12$wv8p4gfwSruQ1E3qD62kZOqgEutXPDjkIMYvXzYFDJrdW8pZKwK/O"
reserved: true
backend_roles:
- "admin"
description: "Dashboard user to communicate with OpenSearch"
opensearchAuthSecret:
admin:
password: strong-password-here
dashboards:
password: another-strong-password-hereThe Security plugin uses bcrypt hashes which can be created with this command (requires Python with bcrypt module installed)
python -c 'import bcrypt; print(bcrypt.hashpw("strong-password-here".encode("utf-8"), bcrypt.gensalt(12, prefix=b"2a")).decode("utf-8"))'Now, to actually use the secrets, pass them through the Opensearch cluster chart's values file values-opensearch-cluster.yaml
cluster:
security:
config:
adminCredentialsSecret:
name: "opensearch-auth-admin"
securityConfigSecret:
name: "opensearch-securityconfig"
dashboards:
opensearchCredentialsSecret:
name: "opensearch-auth-dashboards"Further information can be found in the official OpenSearch Operator User Guide
We assume, an S3-comapatible object store is used for taking snapshots, so the necessary plugin "repository-s3" is enabled by default in the Building Block.
Taking snapshots requires a snapshot repository which can be specified in the cluster values file values-opensearch-cluster.yaml as
cluster:
general:
snapshotRepositories:
- name: "snapshots-s3"
type: s3
settings:
endpoint: objectstorage.fes.cloud.syseleven.net
bucket: bucket-name
region: fes
base_path: os-snapshot
keystore:
- secret:
name: keystore-snapshots-s3-credentialsThe secret referenced in the keystore key has to be present in the cluster and can be created with the help of the extension chart's values file values-opensearch-extension.yaml
s3SnapshotRepositoryCredentials:
'snapshots-s3':
accessKey: ACCESS_KEY_ID
secretKey: SECRET_ACCESS_KEY
The naming schema of the resulting secret is keystore-<name-of-key>-credentials (here, keystore-snapshots-s3-credentials)
It is possible to configure a default snapshot policy through the extension chart, therefor add this to its values file values-opensearch-extension.yaml
snapshots:
enabled: true
defaultSnapshotPolicy:
snapshot_policy_config.json: |
{
"description": "Daily snapshot policy",
"creation": {
"schedule": {
"cron": {
"expression": "0 5 * * *",
"timezone": "Europe/Berlin"
}
},
"time_limit": "1h"
},
"deletion": {
"schedule": {
"cron": {
"expression": "0 1 * * *",
"timezone": "Europe/Berlin"
}
},
"condition": {
"max_age": "14d",
"max_count": 21,
"min_count": 7
},
"time_limit": "1h"
},
"snapshot_config": {
"date_format": "yyyy-MM-dd-HH:mm",
"timezone": "Europe/Berlin",
"indices": ["*", "-.opendistro_security"],
"repository": "snapshots-s3",
"ignore_unavailable": "true",
"include_global_state": "false",
"partial": "true"
}
}This would creates a snapshot policy which takes a snapshot daily at 5 am of all indices without the global state and deletes snapshots older than 14 days. Here the repository snapshots-s3 is used which must be present.
The policy shown above is based on this example adjust it to your needs.
The official documentation describes how to create users, roles, role bindings, etc. via custom resources. See Managing security configurations with kubernetes resources
The user definition references a secret which contains the password. A secret with the name opensearch-user-passwords can be created through the extension with the parameter userPasswords. The key is the username of the user that should be created, the value is the user's password. With the definitions in the following example an OpenSearch user, a role and a role binding that connects both is created
# values-opensearch-extension.yaml
userPasswords:
myuser: secure-password# values-opensearch-cluster.yaml
roles:
- name: myindex-crud
clusterPermissions:
- 'cluster:monitor/state'
- 'cluster_composite_ops'
- 'indices:monitor/settings/get'
indexPermissions:
- indexPatterns:
- myindex*
allowedActions:
- 'indices:admin/get'
- 'indices:monitor/stats'
- 'create_index'
- 'crud'
- 'manage_aliases'
users:
- name: myuser
secretName: opensearch-user-passwords
secretKey: myuser
usersRoleBinding:
- name: myuser-myindex-rw
users:
- myuser
roles:
- myindex-crudkubectl -n syseleven-opensearch port-forward svc/opensearch-cluster 9200Obtain basic auth credentials (e.g. admin)
# Username
kubectl -n syseleven-opensearch get secret opensearch-cluster-admin-password -o jsonpath='{.data.username}' | base64 -d
# Password
kubectl -n syseleven-opensearch get secret opensearch-cluster-admin-password -o jsonpath='{.data.password}' | base64 -dHTTP GET to /_cat/health
curl -s -k -uusername:password 'https://localhost:9200/_cat/health?v'