I ran into an issue where metrics were not showing up in Wavefront and wanted to share the solution. Bottom line, if you send any metrics to the Wavefront Proxy, they absolutely must be in nanoseconds. Here’s why:
I have an application that sends telemetry via Telegraf’s Socket Listener Input Plugin, and the telemetry timestamp was in milliseconds. What happened is that none of the data was recorded in Wavefront. Instead, inside of the /var/log/wavefront/wavefront.log there were entries stating that “[WF-402: Point outside of reasonable timeframe“. There doesn’t appear to be any published mention of this error number. Later on in the error message it goes on to state that the timestamp value was “1506000”. However, my original timestamp was 1506628301128 so there is a huge discrepancy of about 47 years…
The root cause of this can be seen in the Wavefront Output Plugin’s code in the plugins/outputs/wavefront/wavefront.go file on line 214:
It appears that without checking to verify that the values received by the Wavefront Output Plugin are in nanoseconds, it just divides the value down (attempting to get to a second value for legacy system support). Hence, when a value is passed to the output plugin in a value smaller than nanoseconds (in our case: milliseconds), the division makes the timestamp a value too small to be valid and it’s rejected by the proxy as such.