Troubleshooting: Connectivity Issues Between Versa VOS CPE and Versa Analytics Nodes
This article covers the connectivity issues between a Versa SDWAN CPE and Versa Analytics Nodes which might result in the SDWAN CPE’s logs not being visible on the Analytics cluster.
We will look at a step-by-step troubleshooting process to identify and fix this issue.
Troubleshooting
The SDWAN CPE does not establish the connection with the Analytics cluster directly. Instead, the SDWAN branches establish TCP (or UDP) connection with Versa Controller which has ADC (Application Delivery Controller) that load-balances the LEF (log export functionality) connection to one of the nodes in the Analytics cluster. So, we break this connectivity check into two parts.
- Reachability between SDWAN CPE and controller
- Reachability between controller and Analytics cluster
Reachability between SDWAN branch and controller:
Step 1:
– Check the connectivity between the SDWAN CPE and the controller. There must be an established connection between the CPE and the Controller over the SDWAN management IP(tvi-0/3).
Run the command show orgs or-service <organization-Name> lef collectors <collector-name> status
admin@Ind-Spoke-1-cli> show orgs org-services Versa lef collectors LEF-Collector-log_collector1 status VSN SOURCE DESTINATION DESTINATION DESTINATION PENDING LAST ID SOURCE IP PORT IP PORT FQDN ROUTING INSTANCE STATUS MSGS FLAPS FLAPPED ------------------------------------------------------------------------------------------------------------------------ 15 12.0.0.20 1154 12.0.0.8 1234 - Versa-Control-VR Established 0 1 00:15:17
Step 2:
If the connection is in either a ‘Reconnect‘ or ‘Init‘ state, ping the destination IP using the source IP.
CLI> ping <controller’s mgmt ip> routing-instance <tenant-Control-VR> source <branch’s mgmt ip>
If the Ping fails, we need to check if there is a route to the destination IP on the routing instance <tenant-Control-VR>.
admin@Ind-Spoke-1-cli> show route routing-instance Versa-Control-VR | match 12.0.0.8 BGP N/A +0.0.0.0/0 12.0.0.8 00:07:22 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.0/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.1/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a SDWAN N/A +12.0.0.8/32 0.0.0.0 00:07:26 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.8/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.9/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.10/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.16/32 12.0.0.8 00:07:19 ptvi1537 Ind-Cont-T2a BGP N/A +172.16.100.0/24 12.0.0.8 00:07:25 ptvi1537 Ind-Cont-T2a BGP N/A +172.16.120.0/24 12.0.0.8 00:07:25 ptvi1537 Ind-Cont-T2a
If there is no route, then we need to have the route available in <tenant-Control-VR> for the destination IP and verify if there is a reverse route on the Controller for the source IP.
If route is present in both sides, then check if security policy is allowing this connection on the controller.
Reachability between controller and Analytics cluster:
Step 1:
– ADC service should be up on the controller.
admin@Ind-Cont-T2a-cli> show orgs org-services Versa adc virtual-service detail Virtual Service : VAN-VIP Type : any Address : 12.0.0.8 Port : 1234 Routing inst : Versa-Control-VR Admin state : enabled Oper state : UP SERVER TYPE ADDRESS PORT STATE ---------------------------------------------------- LEF-Collector-Analytics-1 any 172.16.100.203 1234 UP/pre>
Step-2:
– If ADC service is down, then check the connectivity between the controller and Analytics cluster’s southbound network. There must be a route to Analytics cluster’s in <tenant-Control-VR) and ensure there is a reverse route from the Analytics cluster to the controller.
admin@Ind-Cont-T2a-cli> ping 172.16.100.203 routing-instance Versa-Control-VR PING 172.16.100.203 (172.16.100.203) 56(84) bytes of data. 64 bytes from 172.16.100.203: icmp_seq=1 ttl=63 time=3.34 ms 64 bytes from 172.16.100.203: icmp_seq=2 ttl=63 time=3.30 ms ^C --- 172.16.100.203 ping statistics --- packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 3.303/3.322/3.341/0.019 ms admin@Ind-Cont-T2a-cli> show route routing-instance Versa-Control-VR | match 172.16.100. BGP N/A +172.16.100.0/24 172.16.120.204 03:36:50 vni-0/2.0
Step-3:
– Ensure ICMP and TCP (or UDP depending on the connection type) is allowed between controller and Analytics Cluster nodes.
Step-4:
– If all the above looks fine, then the controller should have a session created and its being NAT ’ed to one of the nodes in the Analytics cluster.
admin@Ind-Cont-T2a-cli> show orgs org Versa sessions nat brief | select source-ip 12.0.0.20
NAT NAT
VSN VSN SESS DESTINATION SOURCE DESTINATION NAT SOURCE DESTINATION
ID VID ID SOURCE IP IP PORT PORT PROTOCOL NATTED SDWAN APPLICATION NAT SOURCE IP DESTINATION IP PORT PORT
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0 2 29 12.0.0.20 12.0.0.8 1154 1234 6 Yes Yes Analytics/(userdef) 172.16.120.205 172.16.100.203 34858 1234
admin@Ind-Cont-T2a-cli> show orgs org Versa sessions nat detail | select source-ip 12.0.0.20
sessions nat detail 0 2 29
source-ip 12.0.0.20
destination-ip 12.0.0.8
source-port 1154
destination-port 1234
protocol 6
natted Yes
sdwan Yes
application Analytics/(userdef)
nat-source-ip 172.16.120.205
nat-destination-ip 172.16.100.203
nat-source-port 34858
nat-destination-port 1234
forward-pkt-count 253
forward-byte-count 16006
reverse-pkt-count 252
reverse-byte-count 10092
dropped-forward-pkt-count 0
dropped-forward-byte-count 0
dropped-reverse-pkt-count 0
dropped-reverse-byte-count 0
session-age 00:19:31
idle-for 00:00:03
idle-timeout 240
pbf-enabled false
forward-egress-vrf Versa-Control-VR
reverse-egress-vrf Versa-Control-VR
session-provider-zone 0
forward-offload false
reverse-offload false
forward-ingress-interface dtvi-0/79
forward-egress-interface vni-0/2.0
reverse-ingress-interface vni-0/2.0
reverse-egress-interface dtvi-0/79
forward-fc fc_be
reverse-fc fc_be
forward-plp low
reverse-plp low
external-service-chaining false
is-child No
parent-sess-id 0
device ""
source-sgt 0
destination-sgt 0
Step 5:
Verify the log collector configuration of the specified destination address and port by executing the command show log-collector-exporter local collectors
admin@Analytics% show log-collector-exporter local collectors
collector1 {
address 172.16.100.203;
port 1234;
max-connections 512;
storage {
directory /var/tmp/log;
format syslog;
}
}
Step 6:
If the LEF connection status is now ‘Established‘ on the SDWAN CPE and ADC service is up on the Controllers, then check the status of the connections on the Versa Analytic nodes by executing the CLI command.
show log-collector-exporter local collectors connection
admin@Analytics> show log-collector-exporter local collectors connections
COLLECTOR CLIENT CLIENT IP CLIENT CLIENT MESSAGES MESSAGES
NAME SOCKET ADDRESS PORT PROTOCOL TENANT APPLIANCE IN QUEUE PARSED
----------------------------------------------------------------------------------------------------
collector1 16 172.16.120.201 28437 ipfix Versa T1-Controller-a 0 164
23 172.16.120.201 57302 ipfix Cargill T1-Controller-a 0 115
24 172.16.120.205 31272 unknown unknown unknown 0 0
25 172.16.120.205 31203 unknown unknown unknown 0 0
29 172.16.120.205 34858 ipfix Versa Ind-Spoke-1 0 199
28 172.16.120.205 19839 ipfix Versa Ind-Hub
