Resource Manager 6.3.2
Zenhub scaling enhancements
(ZEN-30832, originally identified as defect ZEN-29812). Previously the zenhub service delegated work to subprocesses, which limited its scale to the resources available on a single host. Now, the zenhub service can delegate (non-invalidation related) work to a separate service, zenhubworker, and you can run as many zenhubworker services as required, on any host in the resource pool. The upgrade process will automatically reconfigure zenhub appropriately for horizontally scaling. However. if you have previously had a "superhub" configuration (characterized by having single zenhub instance in a dedicated Control Center resource pool) put in place by Zenoss Services or Support to help mitigate scaling issues this enhancement now addresses, you may also wish to adjust its resource pool configuration manually, after the upgrade. For more information about updating "superhub" configurations, see Migrating zenhub service pool configuration.
Zope deadlock improvements
This release, along with the Control Center 1.6.3 release, provide the following enhancements to Zope-based services:
- (CC-3559) Automatic restart when a service fails 3 health checks in a row. This feature ensures deadlocked services are restarted after 3 failed health checks (2.5 minutes). The
zenapiservices benefit from this update.
- (ZEN-30871) The default number of
zauthservice instances is now two, so that services which depend on authentication are not blocked when
- (ZEN-29781) A supporting library was upgraded to avoid a thread lock issue in a previous version.
Debug option for device modeling
(ZEN-30344) The browser interface now includes an option to model a device with debug messages enabled.
Configurable query limit
(ZEN-29877) The Solr index service now returns a default limit of 10,000 results. This can be changed by editing global.conf, if desired. The variable name is
Default "HostPolicy" for several RM services has been changed to PREFER_SEPARATE
(ZEN-22178) The default host policy for zenhub, zeneventd, and zope services has been change from the default of "none" to PREFER_SEPERATE. The default policy for the new "zenhubworker" service added by ZEN-30832, has also been set to PREFER_SEPARATE. This HostPolicy setting has no effect for zenhub since it instance maximum was set by ZEN-30832 to one, but has been set this way explicitly for future proofing in case that maximum is raised in the future. This change is also automatically applied by upgrades if the HostPolicy is not already set to PREFER_SEPARATE.
Considerations and workarounds
Compatibility with Zenoss Service Impact
This version of Resource Manager is compatible with Zenoss Service Impact version 5.2.3 or later.
Regular expression matching limitation
(ZEN-29376) Resource Manager uses the Apache Solr search platform, which tokenizes fields. Currently, searching a tokenized field with a regex matches only a single token, limiting complex regex searches.
For example, on the INFRASTRUCTURE page, IP Address is a tokenized field. Regex search ^192.168 will not work to filter only IP addresses that start with 192.168. The Device column is untokenized; regex search on the device name works as expected.
Load time for component graphs
(ZEN-29300) Load time might exceed 10 seconds for more than 200 component graphs when you have activated the All on same graph check box.
NFS client 4.1 is not supported
A file locking defect might exist in NFS 4.1 with RHEL/CentOS 7.4, which could cause zeneventserver to crash and other DFS-related problems. For more information, see Configuring NFS 4.0 or knowledge base article Potential Issues Running With RHEL 7.4 Or CentOS 7.4.
CentralQuery maximum memory allocation pool
(ZEN-15907) The size of the CentralQuery maximum memory allocation pool is set by the RAMCommitment variable in the CentralQuery service definition. The default value is 1024MB. Do not change the value to anything less than 1024MB.
Customers who have deployed Control Center and Resource Manager in a high-availability configuration are still supported in this release. For upgrade instructions, refer to the Control Center Upgrade Guide for High-availability Deployments.
- (ZEN-29807) Upgrading from Resource Manager 5.x to 6.x includes a required step to reindex all cataloged objects into the new Solr index. To speed this process, before upgrading, you can increase the CPU core count to use more workers. For more information, see knowledge base article Reducing 5.X To 6.X Upgrade Times By Increasing Workers For Indexing.
- Beginning with release 6.2.1, support for upgrading versions earlier than 5.3.3 was removed from the upgrade instructions. For assistance with planning for upgrading versions prior to 5.3.3, please contact Zenoss Support. In short, if you are on a version prior to that, we'll recommend you should plan to upgrade to version 5.3.3 and then to the latest 6.x version from there.
- The upgrade process might reinstall an older version of the Catalog Service ZenPack. To avoid this issue, manually remove older versions of the Catalog Service ZenPack .egg file from the ZenPack backups directory.
- (ZEN-28375) Beginning with Resource Manager 6.1.0, for counter/derive (rate) data points, the calculated rate value is stored instead of raw counters. The rate is calculated at the collector daemon as it collects data. If a graph (or API) request for a derive of counter data point spans the upgrade data, the system automatically queries the data correctly. However, slight anomalies might occur in the data that is captured immediately before and after an upgrade from Resource Manager 6.0.1 or earlier.
- (ZEN-29100) No data is collected for the ZenossRM device after an upgrade if the localhost collector is moved off the master host. If you have moved the localhost collector off the Control Center master host, to enable data collection, set properties for the ZenossRM device as follows:
- Navigate to the ZenossRM device overview page and select Configuration Properties.
- Set the following zProperties:
zRMMonCCHost - Enter the IP address of the Control Center master host.
zRMMonCCUser - Enter the name of the ccuser account, which is the default account for gaining access to the Control Center browser interface.
zRMMonCCPassword - Enter the password of the ccuser account.
Migrating zenhub service pool configuration
As a result of enhancement ZEN-30832, the zenhub service can now scale horizontally; that is, zenhub is no longer limited by the resources available to a single Control Center delegate. Your current configuration will be automatically migrate to take advantage of this capability.
If you have previously had a "superhub" configuration (characterized by having single zenhub instance in a dedicated Control Center resource pool) put in place by Zenoss Services or Support to help mitigate scaling issues this enhancement now addresses, you should plan to manually adjust the pool this service is in. In short, there is no longer a need to a separate pool, and you should follow the follow manual process post upgrade to revert such configuration to make best use of deploy resources.
- In the Control Center browser interface, edit the "zenhub" service and note the pool is it configured for. Typically this is something like "zenhub".
- In the Control Center browser interface, note the pool the top level service is configured for (edit service right at the top of the Zenoss application page". Typically this is something like "default" or "resource manager"
- Determine whether the pool in step 2 is currently properly resourced to accept the load from zenhub. Check the pool as a whole can support the increased typical memory usage for zenhub, and that the pool as a whole can support the additional CPU load for zenhub (approximately 1+(number of zenhubworker instances)+(0.5 * number of invalidation workers listed in zenhub.conf) cores of work).
- Stop the zenhub and zenhubworker services.
- In the Control Center browser interface, edit the zenhub service and change the pool configuration from the pool noted in step 1, to the pool noted in step 2.
- Repeat step 5 for the zenhubworker service.
- If you need to move the existing delegate from the pool in step 1 to the pool in step 2, based on the results of the analysis in step 3, do so now.
- Start the zenhub and zenhubworker services.
- If you need did NOT need to move the existing delegate from the pool in step 1 to the pool in step 2, you can deregister the delegate from the pool now.
- Delete the pool identified in step 1.
|ZEN-31101||The zenhub service is performing tasks that should be performed by the zenhubworker service.|
Default "HostPolicy" for several RM services has been changed to PREFER_SEPARATE
|ZEN-30966||On the INFRASTRUCTURE page, an error occurs when filtering devices by priority.|
|ZEN-30945||When making a local copy of a template, the original template is displayed.|
|ZEN-30892||Modeling jobs initiated in the browser interface are not given priority and may be delayed.|
|ZEN-30871||The default number of Zauth instance is too low. See Zope deadlock improvements.|
|ZEN-30866||No debug data is written to the ZODB |
|ZEN-30835||SNMP discovery is not parallel.|
|The zenhub service cannot scale horizontally. See Zenhub scaling enhancements.|
|ZEN-30803||The LDAP plugin does not support adding trusted certificates.|
|ZEN-30795||SNMP traps sent through a NAT do not include source IP address.|
|ZEN-30790||Devices added with the default device class ( / ) can't be found by class.|
|ZEN-30789||Zenbatchload displays manageIp warning on all devices.|
|ZEN-30774||The Ping Status Issues and SNMP Status Issues reports don't work on new installations.|
|ZEN-30695||The modeler plugins for the /Server/SSH/Linux class are SNMP, not command.|
|ZEN-30579||The defaults of the Add maintenance window dialog box do not match its most frequent use case.|
|ZEN-30540||Reports run automatically when selected.|
|ZEN-30438||Modeling and monitoring with SNMP v1 does not work properly.|
|ZEN-30437||API method allows non-privileged users to view privileged data.|
|ZEN-30388||Unable to lock Dashboard portlets from updates.|
|The version of OpenSSL included in Resource Manager (1.0.1) is no longer supported.|
|ZEN-30361||The warning messages from the zenmib utility are not readable (one character per line).|
|ZEN-30344||Unable to view debug information about a modeling attempt from the browser interface. See Debug option for device modeling.|
|ZEN-30325||Moving devices from /Discovered to another class fails, leaving devices in an inconsistent state.|
|ZEN-30024||The zenmib utility is unable to process overlapping ranges.|
|ZEN-29982||The reportmail feature does not work for graph or multi-graph reports.|
|ZEN-29978||AWS account password shown in plain text on Firefox browser.|
|ZEN-29929||The password field in ServiceNow notifications loses the entries in the password fields when other fields are subsequently filled.|
|The zenperfsnmp daemon does not close sockets after collections and eventually stops working.|
|ZEN-29878||The Solr service logs too many INFO messages.|
|ZEN-29877||The Solr service does not limit the number of results that can be returned for a query. See Configurable query limit.|
|ZEN-29856||The zencatalogscan utility times out and eventually fails.|
|ZEN-29838||In browser interfaces, flares that should include hyperlinks have only HTML elements.|
|ZEN-29796||The Edit Trigger dialog does not resize properly.|
|ZEN-29781||Zope services can become deadlocked. See Zope deadlock improvements.|
|ZEN-29775||The Zope self-monitoring script hangs when the instance is unresponsive but still listening on TCP. See Zope deadlock improvements.|
|ZEN-29764||Default configuration of maximum connections for MariaDB does not match recommended configuration.|
|ZEN-29762||The default minimum RAM requirement for the |
|ZEN-29702||Upgrade script reads the FROM version from the wrong location after an initial upgrade failure.|
|ZEN-29654||The zenmail service is unable to parse the hostname from an email address.|
|ZEN-29646||Silent failure when re-identifying a device that has no performance.|
|ZEN-29553||When the SSH password is incorrect, monitoring fails and the resulting event flaps (auto-clear and re-appear).|
|ZEN-29542||Asynchronous jobs for moving devices fail with |
|ZEN-29469||(MultiRealm ZenPack) The IP addresses associated with a device do not move to a new realm until the device is remodeled.|
|ZEN-29452||When adding multiple devices, the duration field starts at a large negative number instead of zero.|
|ZEN-29430||Monitoring template name handling does not support case-sensitive names.|
|ZEN-29393||The default minimum RAM requirement for the |
|ZEN-29308||The upgrade log does not include a summary of the migration scripts that were run during an upgrade.|
|ZEN-29264||Large multi-graph reports can't open in new tab due to long URLs.|
|ZEN-29220||MariaDB does not perform well with the default memory allocator |
|ZEN-29062||Changes made to the LDAP user object classes value outside of the browser interface are not maintained when the LDAP properties form is saved in the browser interface.|
|ZEN-28887||No option to disable the removal of HTML/XML tags from plaintext email notifications.|
|ZEN-28676||The page notification type is unavailable, so integrations with external paging systems is not possible.|
|ZEN-28619||Time values are used and displayed inconsistently in the browser interface.|
|ZEN-25930||Unable to specify fields to include/exclude when exporting events.|
The following list describes the known issues in this release:
(ZPS-4986, ZenPacks.zenoss.HttpMonitor, v3.0.4) PythonCollector crashes when the IP Address or Proxy Address field is blank. After the crash, the
zenpythondebug log includes content similar to the following example:
To avoid the crashes, provide content for the appropriate field(s).
- (ZPS-4473, ZenPacks.zenoss.Microsoft.Windows, v2.9.1) When a Windows device reboots, data from PerfMon data sources do not resume collection. The workaround is to remodel the device.
- (ZEN-31194) This release includes ZenPacks.zenoss.LDAPAuthenticator v3.3.3, which no longer includes an option to skip SSL certificate verification during installation or upgrade. If you are using a self-signed certificate, upload it before upgrading. For more information, see Adding an SSL certificate.
- (ZEN-30457) When processing messages from large devices, zenhub invalidation workers can get disconnected from Rabbit MQ due to heartbeat timeouts. If you are seeing the invalidations queue grow without also coming back down, or the number of consumers on the invalidations queue less than the number of running workers, please contact Zenoss Support.
- This release includes an update of the Python
urllib3library, from version 1.10.2 to version 1.22. The new version checks TLS/SSL certificates by default. If you are using a self-signed certificate, upload it before updating Resource Manager or ZenPacks that rely on
urllib3. This change also affects Zenoss Analytics (ZEN-31334). For more information, refer to the Zenoss Support Knowledgebase.
This section lists ZenPacks that are automatically installed, those that are packaged but not installed, and those that are obsolete and should be uninstalled. For more information about ZenPacks, see the ZenPack Catalog.
This release of Resource Manager installs the following ZenPacks at the current version listed in the table.
The following ZenPacks are packaged with Resource Manager, but not automatically installed:
ZenPacks.zenoss.ZenMailTX is obsolete. If this ZenPack is installed at your site, uninstall it.
Zenoss JSON API changes
The Zenoss JSON API includes the following changes from version 6.2.0:
- (ZEN-29558) The
getDevicesmethod of ModelQuery router is updated to support offsets. The method requires additional fields, and the default limit on the number of devices it returns is 200.
importDevicesmethods of the DeviceDumpLoadRouter class now require ZenManager privileges.
- The DeviceRouter class includes the following enhancements:
getInfomethod now includes improved security for password fields.
- The setCollector method includes improved handling of debug messages.
- The addDevice method includes a bug fix in the duplicate device message.