I was recently experimenting with Zenoss Core and noticed that some odd behavior was occuring with my Linux devices. I’m using SNMP v3 and the devices would model correctly and Components would show up, however graphing and performance data never showed up.
There’s an article at https://support.zenoss.com/hc/en-us/articles/204643769?input_string=how+to+recover that shows how to nuke HBase and OpenTSDB for data corruption, which is normally the nuclear option in this case, however this is a new build. There are several other articles around this subject, but none had the solution that I needed.
First off, I logged into my Zenoss instance and verified the modeling via the UI works correctly. You can manually model a device in debug mode by attaching to the zenmodeler container and manually modeling a specific device as shown below:
- SSH into your Control Center
- Find the ID of your zenmodleler container: “serviced service list“
- Then attach to it: “serviced service attach <6u5e0yjw0o9xxmwxa83jtbvq2>”
- Change to the zenoss user: “su – zenoss“
- Start the model: “zenmodeler run -v10 -d <device name> &> /tmp/<device name.log>”
This process shows no modeling errors, so we move on to check OpenTSDB to see if my metrics are actually in the database. You can find the vhost/port information for OpenTSDB inside of Control Center, but it’s typically https://<opentsdb>.<FQDN of Control Center>. Using the OpenTSDB UI I was able to verify that there are no metric values, however the metric points exist.
That brings us to zenperfsnmp which actually collects the SNMP based performance data. The logs for each Zenoss service are usually under /opt/zenoss/logs in their respective container, and in this case it’s /opt/zenoss/logs/zenperfsnmp.log. Doing a tail -f on the file makes it readily apparent that not much is happening, at all…Odd… let’s kick of a manual collection of SNMP Performance data. After logging into the zenperfsnmp container (same process as above) we login as the zenoss user and run
zenperfsnmp run -v10 -d <device name>
The resulting log shows that zenperfsnmp is polling the OIDs correctly, furthermore, I can check OpenTSDB and see that the metrics I just manually polled are in the TSDB. This makes we wonder, what triggers this process and why it’s not happening automatically for all devices? Turns out, if you do a ps -eaf | grep zenperfsnmp command, it is missing the flags to enable workers to actually do the work. Doh! So next, we hop into the zenperfsnmp service via Control Center and click on “Edit Service”. There is a Startup Command that is originally listed as:
su – zenoss -c “/opt/zenoss/bin/zenperfsnmp run -c –logfileonly –monitor {{(parent .).Name}} “
However, for this to work, it needs to look more like:
su – zenoss -c “/opt/zenoss/bin/zenperfsnmp run -c –logfileonly –workers <Integer> –monitor {{(parent .).Name}} “
After changing this value and restarting the service I was able to follow the zenperfsnmp.log file and verify that it was now collecting data as intended. Confirming that data was now getting entered into OpenTSDB also verified that this was the right path. It takes a while for the graphs in Zenoss to start showing data; however after about 30 minutes they were working as they should.