Aruba AOS-CX packet capture including VXLAN decoding

This post shows you how you can capture packets directly on an Aruba CX switch using the linux tcpdump utility.

When you start using the CX CLI you’ll see it is very much built on Linux and allows for a lot of the same commands. If you run the diagnostics command you unlock a set of actual linux features under the diag utilities stanza

  clock-ppm-dump  Display clock frequency
  ddr             Memory utilities
  ethtool         Query or control network driver and hardware settings
  free            Display amount of free and used memory in the system
  iptraf          Network monitoring utility for IP networks.
  lshw            Dumps full hardware configuration information
  lspci           List PCIe devices
  netstat         Print network connections, routing tables
  packet-drop     trace for the packet drop locations in linux kernel
  ps              Report a snapshot of the current processes
  snmpget         Snmpget is an SNMP application that uses the SNMP GET request 
                  to query for information on a network entity.
  snmpwalk        Retrieve a subtree of management values using SNMP GETNEXT requests
  tcpdump         Dump traffic on a network
  top             Display Linux processes
  tshark          Text-based network analyzer

In this post I’m concentrating on tcpdump but related to this is tshark. This is the command line version of wireshark and will decode packets similar to the middle window of the GUI version of wireshark. This can be used live on traffic discussed below but can also be used to decode packets captured with tcpdump that are stored as a file. This is a useful combination since tcpdump has a very granular filtering system compared to tshark but you may need to look out for just a specific flag in a field which only tshark will show.

Alternatively use tcpdump and send the result to a file which can be copied via USB or SCP to somewhere with the full wireshark for deeper analysis. Some times though you just need to know that a certain packet was received by a box. The use case I researched this feature for was to understand which of the two members of a VSX pair packets were being received on. The network was VXLAN so there was a need to capture UDP4789 packets.

A critical point to understanding packet capture on an ASIC based switch is that by default most traffic doesn’t touch the CPU. You can’t packet capture what doesn’t touch the CPU. So you need to take a step to redirect traffic. This is done with a mirror session (span port) but with the destination set to cpu. I’d recommend looking at ACLs at this stage if this is anything other than a lab network because clearly the CPU won’t handle 100G of traffic. The basic example below captures traffic from two interfaces and the both keyword captures in and out packets.

Leaf1(config-mirror-1)# show run cur
mirror session 1
destination cpu
source interface 1/1/4 both
source interface 1/1/3 both
enable

Next you need a standard tcpdump command. Note the command keyword which allows the traditional linux syntax (more flexibility). The example below grabs VXLAN packets in both directions. The -t strips the timestamps to make a much clearer display. Note that the filter must match the raw packet on the wire. I’m looking for pings between two hosts but I knew they would be encapsulated so filtered accordingly. If a packet is encapsulated in VXLAN you are able to see the contents including headers because the entire original IP PDU is there and tcpdump knows how to display. Anything encrypted or a propriety protocol will just show as binary.

diag utilities tcpdump command -t udp dst port 4789 or udp src port 4789 -A

Example 2-way ping from another subnet (L3VNI)

Here you see the initial VXLAN packet (UDP 4789) which is then decapsulated to show the ping packet and reply. The destination of the ping was the switch itself so expect different results if the host is off box.

IP 10.51.100.0.21484 > 10.51.100.6.4789: VXLAN, flags I, vni 20001

IP 10.0.113.68 > 10.0.113.5: ICMP echo request, id 10889, seq 1, length 108
E………b(.3d..3d.S…………N!………sZ]…E…..@.?..I..qD..q…..…~..a………………………. !"#$%&'()+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_abc IP 198.51.100.6.1894 > 198.51.100.7.4789: VXLAN, flags [I] (0x08),vni 20001

IP 10.0.113.5 > 10.0.113.68: ICMP echo reply, id 10889, seq 1, length 108 E....<.......3d..3d..f...........N!.........sZ:.......E...E)..@.. ..q...qD....*...~..a............................ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_abc

Layer 2 packet traversing VXLAN

The following shows an ARP packet that was received encapsulated by VXLAN. Note the inside packet shows as ARP and not IP at the start of the line and there is no IP header.

IP 10.51.100.7.23065 > 10.51.100.6.4789: VXLAN, flags I, vni 10017
ARP, Request who-has 203.0.113.5 tell 203.0.113.11, length 46
E..`..@…(..3d..3d.Z….L…….'!…….tF…………..tF……q………q……………….