Troubleshoot – Versa Appliance to Analytics Connectivity Issue

What are your feelings

Troubleshooting: Connectivity Issues Between Versa VOS CPE and Versa Analytics Nodes #

This article covers the connectivity issues between a Versa SDWAN CPE and Versa Analytics Nodes which might result in the SDWAN CPE’s logs not being visible on the Analytics cluster.

We will look at a step-by-step troubleshooting process to identify and fix this issue.


Troubleshooting #

The SDWAN CPE does not establish the connection with the Analytics cluster directly. Instead, the SDWAN branches establish TCP (or UDP) connection with Versa Controller which has ADC (Application Delivery Controller) that load-balances the LEF (log export functionality) connection to one of the nodes in the Analytics cluster. So, we break this connectivity check into two parts.

  1. Reachability between SDWAN CPE and controller
  2. Reachability between controller and Analytics cluster

Reachability between SDWAN branch and controller:


Step 1:

– Check the connectivity between the SDWAN CPE and the controller. There must be an established connection between the CPE and the Controller over the SDWAN management IP(tvi-0/3).

Run the command show orgs or-service <organization-Name> lef collectors <collector-name> status

admin@Ind-Spoke-1-cli> show orgs org-services Versa lef collectors LEF-Collector-log_collector1 status
VSN             SOURCE  DESTINATION  DESTINATION  DESTINATION                                 PENDING         LAST
ID   SOURCE IP  PORT    IP           PORT         FQDN         ROUTING INSTANCE  STATUS       MSGS     FLAPS  FLAPPED
------------------------------------------------------------------------------------------------------------------------
15   12.0.0.20  1154    12.0.0.8     1234         -            Versa-Control-VR  Established  0        1      00:15:17

Step 2:

If the connection is in either a ‘Reconnect‘ or ‘Init‘ state, ping the destination IP using the source IP.


CLI> ping <controller’s mgmt ip> routing-instance <tenant-Control-VR> source <branch’s mgmt ip>

If the Ping fails, we need to check if there is a route to the destination IP on the routing instance <tenant-Control-VR>.


admin@Ind-Spoke-1-cli> show route routing-instance Versa-Control-VR | match 12.0.0.8 BGP N/A +0.0.0.0/0 12.0.0.8 00:07:22 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.0/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.1/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a SDWAN N/A +12.0.0.8/32 0.0.0.0 00:07:26 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.8/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.9/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A +12.0.0.10/32 12.0.0.8 00:07:24 ptvi1537 Ind-Cont-T2a BGP N/A 12.0.0.16/32 12.0.0.8 00:07:19 ptvi1537 Ind-Cont-T2a BGP N/A +172.16.100.0/24 12.0.0.8 00:07:25 ptvi1537 Ind-Cont-T2a BGP N/A +172.16.120.0/24 12.0.0.8 00:07:25 ptvi1537 Ind-Cont-T2a

If there is no route, then we need to have the route available in <tenant-Control-VR> for the destination IP and verify if there is a reverse route on the Controller for the source IP.

If route is present in both sides, then check if security policy is allowing this connection on the controller.


Reachability between controller and Analytics cluster:


Step 1:


– ADC service should be up on the controller.

admin@Ind-Cont-T2a-cli> show orgs org-services Versa adc virtual-service detail

Virtual Service  : VAN-VIP
   Type          : any
   Address       : 12.0.0.8
   Port          : 1234
   Routing inst  : Versa-Control-VR
   Admin state   : enabled
   Oper state    : UP

   SERVER             TYPE  ADDRESS         PORT  STATE
   ----------------------------------------------------
   LEF-Collector-Analytics-1 any   172.16.100.203  1234  UP/pre>

Step-2:


– If ADC service is down, then check the connectivity between the controller and Analytics cluster’s southbound network. There must be a route to Analytics cluster’s in <tenant-Control-VR) and ensure there is a reverse route from the Analytics cluster to the controller.


admin@Ind-Cont-T2a-cli> ping 172.16.100.203 routing-instance Versa-Control-VR PING 172.16.100.203 (172.16.100.203) 56(84) bytes of data. 64 bytes from 172.16.100.203: icmp_seq=1 ttl=63 time=3.34 ms 64 bytes from 172.16.100.203: icmp_seq=2 ttl=63 time=3.30 ms ^C --- 172.16.100.203 ping statistics --- packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 3.303/3.322/3.341/0.019 ms admin@Ind-Cont-T2a-cli> show route routing-instance Versa-Control-VR | match 172.16.100. BGP N/A +172.16.100.0/24 172.16.120.204 03:36:50 vni-0/2.0


Step-3:


– Ensure ICMP and TCP (or UDP depending on the connection type) is allowed between controller and Analytics Cluster nodes.

Step-4:


– If all the above looks fine, then the controller should have a session created and its being NAT ’ed to one of the nodes in the Analytics cluster.

admin@Ind-Cont-T2a-cli> show orgs org Versa sessions nat brief | select source-ip 12.0.0.20
                                                                                                                                           NAT     NAT
VSN  VSN  SESS             DESTINATION  SOURCE  DESTINATION                                                                NAT             SOURCE  DESTINATION
ID   VID  ID    SOURCE IP  IP           PORT    PORT         PROTOCOL  NATTED  SDWAN  APPLICATION          NAT SOURCE IP   DESTINATION IP  PORT    PORT
----------------------------------------------------------------------------------------------------------------------------------------------------------------
0    2    29    12.0.0.20  12.0.0.8     1154    1234         6         Yes     Yes    Analytics/(userdef)  172.16.120.205  172.16.100.203  34858   1234

admin@Ind-Cont-T2a-cli> show orgs org Versa sessions nat detail | select source-ip 12.0.0.20
sessions nat detail 0 2 29
 source-ip                  12.0.0.20
 destination-ip             12.0.0.8
 source-port                1154
 destination-port           1234
 protocol                   6
 natted                     Yes
 sdwan                      Yes
 application                Analytics/(userdef)
 nat-source-ip              172.16.120.205
 nat-destination-ip         172.16.100.203
 nat-source-port            34858
 nat-destination-port       1234
 forward-pkt-count          253
 forward-byte-count         16006
 reverse-pkt-count          252
 reverse-byte-count         10092
 dropped-forward-pkt-count  0
 dropped-forward-byte-count 0
 dropped-reverse-pkt-count  0
 dropped-reverse-byte-count 0
 session-age                00:19:31
 idle-for                   00:00:03
 idle-timeout               240
 pbf-enabled                false
 forward-egress-vrf         Versa-Control-VR
 reverse-egress-vrf         Versa-Control-VR
 session-provider-zone      0
 forward-offload            false
 reverse-offload            false
 forward-ingress-interface  dtvi-0/79
 forward-egress-interface   vni-0/2.0
 reverse-ingress-interface  vni-0/2.0
 reverse-egress-interface   dtvi-0/79
 forward-fc                 fc_be
 reverse-fc                 fc_be
 forward-plp                low
 reverse-plp                low
 external-service-chaining  false
 is-child                   No
 parent-sess-id             0
 device                     ""
 source-sgt                 0
 destination-sgt            0

Step 5:


Verify the log collector configuration of the specified destination address and port by executing the command show log-collector-exporter local collectors

admin@Analytics% show log-collector-exporter local collectors
collector1 {
    address         172.16.100.203;
    port            1234;
    max-connections 512;
    storage {
        directory /var/tmp/log;
        format    syslog;
    }
}


Step 6:


If the LEF connection status is now ‘Established‘ on the SDWAN CPE and ADC service is up on the Controllers, then check the status of the connections on the Versa Analytic nodes by executing the CLI command.

show log-collector-exporter local collectors connection

admin@Analytics> show log-collector-exporter local collectors connections
COLLECTOR   CLIENT  CLIENT IP       CLIENT  CLIENT                              MESSAGES  MESSAGES
NAME        SOCKET  ADDRESS         PORT    PROTOCOL  TENANT   APPLIANCE        IN QUEUE  PARSED
----------------------------------------------------------------------------------------------------
collector1  16      172.16.120.201  28437   ipfix     Versa    T1-Controller-a  0         164
            23      172.16.120.201  57302   ipfix     Cargill  T1-Controller-a  0         115
            24      172.16.120.205  31272   unknown   unknown  unknown          0         0
            25      172.16.120.205  31203   unknown   unknown  unknown          0         0
            29      172.16.120.205  34858   ipfix     Versa    Ind-Spoke-1      0         199
            28      172.16.120.205  19839   ipfix     Versa    Ind-Hub