How to Migrate Indices from Elasticsearch wth Minio.

Hello, in this tutorial I will describe the steps needed to be follow in order to migrate Indices from Elasticsearch to another Elasticsearch Instance. To accomplish this goal I will use this fantastic cli called elasticsearch-dump to perform the necessary operations (PUT/GET) and Minio S3 Compatible Object Storage to temporarily store the ES indices.

Lets Dive in...

Step1. Clone the repository and build the image.

1. git clone https://github.com/elasticsearch-dump/elasticsearch-dump

2. cd elasticsearch-dump 

3. Add any missing packages like curl nano e.t.c
### Start DockerFile 
FROM node:14-buster-slim
LABEL maintainer="ferronrsmith@gmail.com"
ARG ES_DUMP_VER
ENV ES_DUMP_VER=${ES_DUMP_VER:-latest}
ENV NODE_ENV production

RUN npm install elasticdump@${ES_DUMP_VER} -g
RUN apt-get update \
    && apt-get install -y curl \
    && rm -rf /var/lib/{apt,dpkg,cache,log}
COPY docker-entrypoint.sh /usr/local/bin/

ENTRYPOINT ["docker-entrypoint.sh"]

CMD ["elasticdump"]

4. docker build . -t <docherhub-user>/<image-name>:<release-version> 

5. docker login -u <username> 

6. docker push <docherhub-user>/<image-name>:<release-version>

Step2. Create a Persistent Volume for the Kubernetes job

cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app.kubernetes.io/name: elasticdumpclient-pvc
  name: elasticdumpclient-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: <depends on the Cloud Provider>
  volumeMode: Filesystem
EOF

Step3. Create a new Kubernetes Job and mount the previously provisioned Persistent Volume.

cat << EOF | kubectl apply -f -
apiVersion: batch/v1
kind: Job
metadata:
  name: elasticdumpclient
  labels:
    app: elasticdumpclient
spec:
  template:
    spec:
      initContainers:
        - name: fix-data-dir-permissions
          image: alpine:3.16.2
          command:
            - chown
            - -R  
            - 1001:1001
            - /data
          volumeMounts:
            - name: elasticdumpclient-pvc
              mountPath: /data
      containers:
        - name: elasticdumpclient
          image:  thvelmachos/elasticdump:6.68.0-v1
          command: ["/bin/sh", "-ec", "sleep 460000"] 
          imagePullPolicy: Always
          volumeMounts:
            - name: elasticdumpclient-pvc
              mountPath: /data
      restartPolicy: Never
      volumes:
        - name: elasticdumpclient-pvc
          persistentVolumeClaim:
            claimName: elasticdumpclient-pvc
  backoffLimit: 1
EOF

Step3. Upload a Script to the Kubernetes Job

kubectl cp <local-path>/migrate-es-indices.sh <pod-name>:/data/migrate-es-indices.sh

# cat migrate-es-indices.sh
#!/bin/sh
arr=$(curl -X GET -L 'http://elasticsearch-master.default.svc.cluster.local:9200/_cat/indices/?pretty&s=store.size:desc' | grep -E '[^access](logs-)' | awk '{print $3}')

for idx in $arr; do
  echo "Working: $idx"
  elasticdump --s3AccessKeyId "<access-key>" --s3SecretAccessKey "<secret-key>" --input=http://http://elasticsearch-master.default.svc.cluster.local:9200/$idx --output "s3://migrate-elastic-indices/$idx.json" --s3ForcePathStyle true --s3Endpoint https://<minio-endpoint>:<port>
done

Step4. Execute the Script.

chmod +x /data/migrate-es-indices.sh
# Send job to the Background with nohup
nohup sh /data/migrate-es-indices.sh  > /data/migrate.out  2>&1  &

That's it, Enjoy!!!

I hope you like the tutorial, if you do give a thumps up! and follow me in Twitter, also you can subscribe to my Newsletter in order to avoid missing any of the upcoming tutorials.

Media Attribution

I would like to thank Clark Tibbs for designing the awesome photo I am using in my posts.