Updating serviced on the master host for release 1.9.0

Use this procedure to update the serviced binary on the Control Center master host. For delegate hosts, see Updating serviced on delegate hosts for release 1.9.0.

In multi-host deployments, stop serviced on the master host first.

This procedure is longer and more complex than previous versions; please review the entire procedure before proceeding. In particular, it includes an optional migration of Resource Manager log data, which could take a long time. The migration is optional because Control Center does not require historical logs.

The serviced RPM includes a script that requires uid/gid 1001 for its own use. If 1001 is already in use for another user/group, change it temporarily before starting this procedure.

In testing, data migration speeds averaged about 1 megabyte per second. However, migration time is much longer and impossible to estimate when the file systems involved are mounted and the host is experiencing network performance issues. This procedure includes steps that help you estimate the amount of time required to migrate your log data.

To perform the optional log migration, you need the migrate_cc_services_data.sh script, which is available to customers on delivery.zenoss.io.

If you are using the TMPDIR variable in /etc/default/serviced, you must determine whether the specified path is mounted noexec before proceeding. Step 5 provides a template for doing so.

Follow these steps:

Log in to the host as root or as a user with superuser privileges.

Start the docker service, if necessary.

systemctl is-active docker || systemctl start docker

Disable SELinux temporarily, if necessary.
1. Determine the current mode.
```
sestatus | awk '/Current mode:/ { print $3 }'
```
  • If the result is not enforcing, proceed to step 4.
  • If the result is enforcing, perform the next substep.
2. Disable SELinux temporarily.
```
setenforce 0
```
Copy the current serviced configuration file to a backup and set the backup's permissions to read-only.
```
cp /etc/default/serviced /etc/default/serviced-pre-1.9.0
chmod 0440 /etc/default/serviced-pre-1.9.0
```
Create a temporary directory, if necessary.
The RPM installation process writes a script to TMP and starts it with exec to migrate Elasticsearch.
1. Determine whether the /tmp directory is mounted on a partition with the noexec option set.
```
awk '$2 ~ /\/tmp/ { if ($4 ~ /noexec/) print "protected" }' < /proc/mounts
```
  If the command returns a result, perform the substeps. Otherwise, proceed to step 6.
2. Create a temporary storage variable for TMP, if necessary.
```
test -n "$TMP" && tempTMP=$TMP
```
3. Create a temporary directory.
```
mkdir $HOME/tempdir && export TMP=$HOME/tempdir
```

Set variables for Control Center data paths.

myIsvcs=$(test -f /etc/default/serviced && grep -E '^[[:space:]]*SERVICED_ISVCS_PATH' /etc/default/serviced | cut -f2 -d= | tr -d \"\')
myIsvcs=$(echo $myIsvcs | awk '{ print $NF }')
export HOST_ISVCS_DIR="${myIsvcs:-/opt/serviced/var/isvcs}"
myBackups=$(test -f /etc/default/serviced && grep -E '^[[:space:]]*SERVICED_BACKUPS_PATH' /etc/default/serviced | cut -f2 -d= | tr -d \"\')
myBackups=$(echo $myBackups | awk '{ print $NF }')
myBackups="${myBackups:-/opt/serviced/var/backups}"

Prepare for log data migration.
Note: Even if you are not planning to migrate log data, you must ensure space is available for a backup of the data.
1. Determine the size of the log data to migrate, if desired.
```
du -sh ${HOST_ISVCS_DIR}/elasticsearch-logstash | cut -f1
```
  The result includes a numeral and a units character (M for megabytes, G for gigabytes).
2. Use the result of the previous step to estimate the amount of time required to migrate log data.
  Under ideal conditions, migration speed is approximately 1 megabyte per second.
3. Determine the amount of space available in the backups directory.
```
du -sh ${myBackups} | cut -f1
```
4. Compare the size of the data to migrate (step 7a) with the amount of space available in the backups directory (step 7c).
  If the amount of space available is not at least 4 times greater than the size of the data to migrate, choose an alternate path that does have enough space, and then update the temporary variable:
```
myBackups=<AlternatePath>
```
  You may need to mount additional space on your host.

Set up variables and paths for storing backup copies of your historical data.

export HOST_LS_WORKING_DIR="${myBackups}/logstash" && mkdir -p $HOST_LS_WORKING_DIR
export HOST_BACKUP_DIR="${myBackups}/serviced" && mkdir -p $HOST_BACKUP_DIR

Install the new serviced RPM package.
The installation includes a migration of internal services data, which takes some extra time.
```
yum install --enablerepo=zenoss-mirror /opt/zenoss-repo-mirror/serviced-1.9.0-1.x86_64.rpm
```
If yum returns an error due to dependency issues, see Resolving package dependency conflicts for potential resolutions.

(Optional) Migrate historical log data, if desired.
Reminder: After the upgrade is complete and serviced is restarted, create an Elasticsearch index for the migrated data.

<PathToScript>/migrate_cc_services_data.sh

The script does not complete when messages similar to the following are displayed for a long time:

2021-05-12 15:30:52,967 [INFO] >>> Documents processed: 11680777/24441034; current speed 0 doc's/sec <<<
2021-05-12 15:31:12,971 [INFO] GET http://127.0.0.1:9101/_cat/count?h=count [status:200 request:0.003s]
 |XXXXXXXXXXXXXXXXXXXXXXX---------------------------| 47.000%
2021-05-12 15:31:12,971 [INFO] >>> Documents processed: 11680777/24441034; current speed 0 doc's/sec <<<
2021-05-12 15:31:32,975 [INFO] GET http://127.0.0.1:9101/_cat/count?h=count [status:200 request:0.003s]
 |XXXXXXXXXXXXXXXXXXXXXXX---------------------------| 47.000%
2021-05-12 15:31:32,975 [INFO] >>> Documents processed: 11680777/24441034; current speed 0 doc's/sec <<<
2021-05-12 15:31:52,983 [INFO] GET http://127.0.0.1:9101/_cat/count?h=count [status:200 request:0.004s]

To work around this issue, perform the following substeps:

Log in to a new terminal session on the Control Center master host as root or as a user with superuser privileges.

Stop all running containers.

docker container stop $(docker container list -q)

Stop the Elasticsearch migration process.

kill $(ps aux | awk '/[e]s_cluster.ini/ { print $2 }')

Exit the new terminal session and return to the original session.
Verify that the required environment variables are set.
```
env | grep -E 'HOST_(BACKUP|ISVCS|LS_WORKING)_DIR'
```
If the command returns no result, return to steps 7 and 8 to reset the HOST_BACKUP_DIR, HOST_ISVCS_DIR, and HOST_LS_WORKING_DIR variables.

Delete the update attempt.

pushd ${HOST_LS_WORKING_DIR}
rm -rf ./elasticsearch-logstash-new
pushd ${HOST_ISVCS_DIR}
rm -rf ./elasticsearch-logstash
popd

Restore the backup file.

pushd ${HOST_BACKUP_DIR}
cp -ar elasticsearch-logstash.backup ${HOST_ISVCS_DIR}/elasticsearch-logstash

Return to step 10 and start the migration script again.

Enable SELinux, if necessary.
Perform this step only if you disabled SELinux in step 3:
```
setenforce 1
```
Restore the previous value of TMP, if necessary.
```
test -n "$tempTMP" && export TMP=$tempTMP
```

Make a backup copy of the new configuration file and set permissions to read-only.

cp /etc/default/serviced /etc/default/serviced-1.9.0-orig
chmod 0440 /etc/default/serviced-1.9.0-orig

Compare the new configuration file with the configuration file of the previous release.
1. Identify the configuration files to compare.
```
ls -l /etc/default/serviced*
```
  The original versions of the configuration files should end with orig, but you may have to compare the dates of the files.
2. Compare the new and previous configuration files.Replace New-Version with the name of the new configuration file, and replace Previous-Version with the name of the previous configuration file:
```
diff New-Version Previous-Version
```
  For example, to compare versions 1.9.0 and the most recent version, enter the following command:
```
diff /etc/default/serviced-1.8.0-orig /etc/default/serviced-1.9.0-orig
```
- If the command returns no result, restore the backup of the previous configuration file.
```
cp /etc/default/serviced-pre-1.9.0 /etc/default/serviced && chmod 0644 /etc/default/serviced
```
- If the command returns a result, restore the backup of the previous configuration file, and then optionally, use the results to edit the restored version.
```
cp /etc/default/serviced-pre-1.9.0 /etc/default/serviced && chmod 0644 /etc/default/serviced
```
  For more information about configuring a host, see Configuration variables.
Reload the systemd manager configuration.
```
systemctl daemon-reload
```