It’s no secret that we at WAN Dynamics are big fans of Arista switches. There is a definite feeling when using them that the Arista Extensible Operating System (EOS) was written by engineers for engineers in order to make the network operator’s life easier. In this post, we will discuss three features that are great on their own, however when coupled together, you get an amazing amount of flexibility to solve real world problems. The sum of these features is greater than each of their parts.
Feature #1 - CLI Output to Email
The first cool feature that not many people know about is sending CLI output via email. That’s right, Arista EOS can be configured to pipe the output of commands to be sent to an email address. Sounds simple enough but it can be very useful when grabbing output from a switch that you need or have to share with someone else. Another use case is perhaps you are remoted into a VDI jump box where the copy/paste function doesn’t work properly.
The first thing to do to enable this functionality is to configure the switch to send emails. Your setup may look something like this:
email
from-user network-admin@wandynamics.com
server vrf MGMT smtp.gmail.com:587
auth username network-admin
auth password 7 my-password-hash
tls
Example of sending interface Management1 info:
Arista-720XP-CS1# show interfaces management 1 | email kgiusti@wandynamics.com -i -s "[Management1 Output] From $HOSTNAME"
Enter your message below. Hit ^D to finish the message.
-i means to embed the text in the body of the email rather than include it as an attachment.
-s lets you specify the subject line of the email.
$HOSTNAME is just a variable to reference the current hostname configured on the switch.
Look at that, we got an email:
Feature #2 - QtCat and Qt logs
If you want to see what’s going on under the hood Arista does not stop you. If you want, you can drop to a bash shell and navigate the folder structure just like any other Linux host. One interesting area is the “qt” folder located at /var/log. This where a lot of information regarding logs and agents with the switch are stored. If you want to view some of this you can use a utility called “qtcat”. (much like the Linux utility “cat” but specifically for viewing the contents of .qt files)
Arista-720XP-CS1# bash
Arista Networks EOS shell
[wandyn@Arista-720XP-CS1 ~]$ cd /var/log/qt
For example, if I want information on what the BGP Routing Information Base (RIB) is doing I can take a look at Rib-bgp.qt with the following command:
[wandyn@Arista-720XP-CS1 qt]$ qtcat Rib-bgp.qt
2021-04-05 17:29:19.528202 7 00992.347179038, +35327384829 "BGP NORMAL: bgp_recv_established: peer: 192.168.30.1 received 19 bytes type KeepAlive"
2021-04-05 17:29:19.528314 7 00992.347291475, +269844 "BGP TASK: process_one_peer_msgq: 192.168.30.1 processed 1 messages"
2021-04-05 17:29:19.528315 7 00992.347292167, +1660 "BGP TASK: bgp_recv_msg_job: 192.168.30.1: Initiating socket read, work_count=1"
2021-04-05 17:29:19.528319 3 00992.347295861, +8866 "BGP TASK: bgp_recv_msg_job: processed 1 messages. yield_now no"
2021-04-05 17:29:34.808711 9 01007.627687768, +36672376876 "BGP NORMAL: bgp_traffic_timeout: peer 192.168.30.1 last checked 75 last sent 30 last keepalive 30"
2021-04-05 17:29:34.808717 9 01007.627694285, +15642 "BGP NORMAL: bgp_send_keepalive: sending KEEPALIVE to peer 192.168.30.1"
2021-04-05 17:29:34.808788 9 01007.627765406, +170687 "BGP NORMAL: bgp_send_keepalive: peer 192.168.30.1, res 1"
2021-04-05 17:29:36.730403 7 01009.549380506, +4611805351 "BGP NORMAL: bgp_holdtime_expired: peer 192.168.30.1 last checked 77 last recv'd 17.374942 last keepalive 2"
2021-04-05 17:29:44.127937 7 01016.946914418, +17753808491 "BGP NORMAL: bgp_recv_established: peer: 192.168.30.1 received 19 bytes type KeepAlive"
2021-04-05 17:29:44.128052 7 01016.947029473, +276127 "BGP TASK: process_one_peer_msgq: 192.168.30.1 processed 1 messages"
2021-04-05 17:29:44.128053 7 01016.947030017, +1306 "BGP TASK: bgp_recv_msg_job: 192.168.30.1: Initiating socket read, work_count=1
Since this is Linux under the hood the normal filtering tools such as “grep” for example can be applied to parse output:
[wandyn@Arista-720XP-CS1 qt]$ qtcat Rib-bgp.qt | grep 192.168.30.1
2021-04-05 17:13:07.926412 2 00020.745388757, +31995078 "BGP NORMAL: bgp_peer_reinit peer 192.168.30.1,"
2021-04-05 17:13:07.926660 2 00020.745636808, +595314 "BGP ALL: bgp_set_peer_preference: 192.168.30.1 local AS 65020 peer AS 65010 preference 200"
2021-04-05 17:13:07.926677 1 00020.745654467, +0 "BGP TASK: bgp_find_rib_out: bnp 192.168.30.1 async: no"
2021-04-05 17:13:07.926712 1 00020.745689120, +25150 "BGP TASK: bgp_find_rib_out: bnp 192.168.30.1, new bro 0xeb064004 async no"
2021-04-05 17:13:07.983352 2 00020.802329420, +42154 "BGP NORMAL: bgp_peer_reinit peer 192.168.30.1,"
2021-04-05 17:13:07.983357 2 00020.802334065, +11148 "BGP ALL: bgp_set_peer_preference: 192.168.30.1 local AS 65020 peer AS 65010 preference 200"
If you would like you can take the output from qtcat and redirect to somewhere else such as flash to view at a later time you can do that:
[wandyn@Arista-720XP-CS1 qt]$ qtcat ArpMonitor.qt > /mnt/flash/ARP_Output.txt
[wandyn@Arista-720XP-CS1 qt]$ exit
logout
Arista-720XP-CS1# more flash:ARP_Output.txt
2021-04-09 13:01:45.450388 4 00000.000000000, +699755935682 "-------------------------------- level 4 first msg --------------------------------"
2021-04-09 13:01:45.450388 4 00000.000000000, +0 "Setting netlinkFd to socket descriptor 44"
2021-04-09 13:01:45.450447 4 00000.000059257, +142212 "Setting ip6 netlinkFd to socket descriptor 45"
2021-04-09 13:01:45.574356 4 00000.123968029, +297373046 "Setting netlinkFd to socket descriptor 47"
2021-04-09 13:01:45.574374 4 00000.123986141, +43467 "Setting ip6 netlinkFd to socket descriptor 48"
2021-04-09 13:07:35.723521 4 00350.273132378, +840335321547 "Sending netlink request for: 192.168.30.1 on Management1"
2021-04-09 13:07:35.723541 4 00350.273152716, +48811 "sending RTM_GETNEIGH msg for ip 192.168.30.1 dev ma1 seq: 0x0"
Feature #3 - Event Handler
In the most basic terms, an event-handler tells the switch to do something when a specified condition happens. For example if the switch matches a regex pattern in the logs run the listed bash command.
event-handler MY_Handler
action bash command
!
trigger on-logging
regex pattern to match in logs
One use case for the Event Handler feature that comes to mind is a situation where a customer used the “archive” command on another vendor’s platform to backup configs via FTP every time the user issued “write memory”. The customer wondered if there was an Arista equivalent command for this. We tried to convince the customer that there might be better ways to handle this but they were dead set on using this method. Even though there was not exactly an Arista equivalent command we were able to write an event-handler that accomplished exactly what the previous backup solution was doing.
Using Them Together
All of these utilities are useful on their own however if you use them in conjunction with each other you can do some very powerful things. For example, let's say you have a chronic problem where one of your BGP peers sometimes drops at 2am. Maybe you want to grab some additional output at the time the issue happens then have it sent to you via email and not have to stay up to the wee hours of the morning. Well you can do that.
In this example we have a BGP peering configured in our lab switch and we want to know if when BGP goes down, does our ARP entry for that peer goes away too. We also can grab the historical BGP info.
(Color coded to detail what each section does)
event-handler TSHOOT_BGP_ARP action bash sudo qtcat /var/log/qt/Arp.qt | grep "\<192.168.30.1\>" | email kgiusti@wandynamics.com -i -s "[ARP Output] From $HOSTNAME" ! trigger on-logging regex ^.*BGP-5-ADJCHANGE: peer 192.168.30.1.*Idle*$
- in bash and view the Arp.qt file and grep for lines with 192.168.30.1
- Pipe the output to email and send it to my email address with the subject “ARP Output” from the configured hostname of the switch.
- Do a regex search of the log buffer to match the log entry of when the BGP session for peer 192.168.30.1 goes down and changes status.
Similarly we can write another event-handler to grab BGP information when the session fails:
event-handler TSHOOT_BGP_SESSION action bash sudo qtcat /var/log/qt/Rib-bgp.qt | grep "\<192.168.30.1\>" | email kgiusti@wandynamics.com -i -s "[BGP Output] From $HOSTNAME" ! trigger on-logging regex ^.*BGP-5-ADJCHANGE: peer 192.168.30.1.*Idle*$
- In bash, view the Rib-bgp.qt file and grep for lines with 192.168.30.1
- Pipe the output to email and send it to my email address with the subject “BGP Output” from the configured hostname of the switch.
- Do a regex search of the log buffer to match the log entry of when the BGP session for peer 192.168.30.1 goes down and changes status.
In closing this is not intended as a BGP troubleshooting guide but rather to detail how Arista gives you so many tools in EOS to do creative things in order to solve problems.
Thanks for reading and if you like this, check out some of my other articles on Arista EOS and all the cool things you can do with it.
Dual Datacenter DMZ With VXLAN, VARP, VRF & OSPF with Arista Networks
8 Lessons Learned - Arista Datacenter Interconnect (DCI) with VXLAN and vARP