VyOS High CPU Utilization with SNMP checks via Zabbix

I was working with VyOS 1.2 and 1.3 routers in a lab test. I wanted to wanted to monitor the underlying Linux/Debian objects and various routing process statuses using Zabbix. Basic items like BGP peers, OSPF neighborship, and some counters. Nothing excessive like dumping full peer tables every 30 seconds. About 24 checks in total every 60-120 seconds.

After about a week of monitoring, I discovered the router would become unresponsive, lock up, and reboot. Digging into a router during an event revealed the SNMP process was utilizing 100% of the CPU. This utilization had starved the underlying routing daemons (zebra, bgpd, ospfd), so they didn’t have the resources to process route updates.

Luckily, this problem has an easy-to-implement workaround found in the VyOS forums. It also appears this bug was resolved in later versions of VyOS.

edit /etc/default/snmpd

and replace

SNMPDOPTS='-LSed -u snmp -g snmp -p /run/snmpd.pid'

with (or whatever options you prefer)

SNMPDOPTS='-LSed -u snmp -g snmp -I -ipCidrRouteTable,inetCidrRouteTable -p /run/snmpd.pid'

Restart snmpd

/etc/init.d/snmpd restart

The critical item to see here is the insertion of this text into the startup options.

-I -ipCidrRouteTable,inetCidrRouteTable

I don’t believe Zabbix had anything to do with this other than tickling an existing bug within VyOS. Maybe my next project should be to update these lab routers.++