Playing with Wavefront – Network Packet Loss

Playing with Wavefront – Network Packet Loss

Now that we know when an agent goes offline, let’s create a query to detect when our devices experience an increased rate of dropped packets. To do that we’ll create 2 queries, the first is our data on all dropped packets per source.
[code language=”bash”]
sum(mavg(5m,ts("net.drop.*", source="FQDN,sub.domain.com" )),sources)
[/code]
This value is represented with the blue line in the below chart.

That’s great, but we want to detect a change in trends, not just alert on a static threshold. To do that we’re going to create a query that uses moving averages. This query is reflected in the above chart as the orange line.
[code language=”bash”]
sum(mavg(2m,ts("net.drop.*", source="FQDN,sub.domain.com")), sources) – sum(lag(5m,mavg(2m,ts("net.drop.*", source="FQDN,sub.domain.com"))),sources)
[/code]
As you can see, it handles the upticks rather nicely so we’re going to create an alarm off of it using the value of 10 as our threshold.
[code language=”bash”]
sum(mavg(2m,ts("net.drop.*", source="sg01-0-jnks1")), sources) – sum(lag(5m,mavg(2m,ts("net.drop.*", source="sg01-0-jnks1"))),sources) > 10
[/code]
You can see the alert condition triggers in the Alert Backtesting match what we expected from our above research; every time the orange value was over 10 we receive an alert.

There we go…
 

Leave a Reply

Your email address will not be published. Required fields are marked *