Troubleshooting OSPF Neighbor Adjacencies
by Radu Pavaloiu
When the OSPF neighbor adjacency is not in the full state then it is in one of the other states:
- There’s no OSPF neighbor at all
- It’s stuck in ATTEMPT
- It’s stuck in INIT
- It’s stuck in 2-WAY
- It’s stuck in EXSTART/EXCHANGE
- It’s stuck in LOADING
1. OSPF is not enabled on the interface
Let’s start with our first scenario. We have two OSPF routers, one of them is a Versa appliance:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR routing-instance Tenant1-LAN-VR [ok][2020-06-26 01:28:32]As you can see Versa box does not report the other router as a neighbor. So, what could be wrong? Let’s take a look on the Versa appliance and on the peer:
admin@branch6-cli> show ospf interface brief Codes for operation state (State): BDR - backup designated router; DR - designated router; ODR - other designated router; PTP - point to point dwn - down; lbk - loopback; wtg - waiting Interface Area DR ID BDR ID State --------- ---- ----- ------ ----- IOU1#show ip ospf 1 interface brief | include Et0/1.10 Et0/1.10 1 0.0.0.0 192.168.254.10/30 10 P2P 0/0We can see that OSPF is not enabled on vni-0/4.10 of the Versa appliance but it’s running on its peer. After this is fixed, the OSPF neighborship is established without any problem:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up
2. Layer 2 is down
OSPF adjacency is reported as down:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR routing-instance Tenant1-LAN-VR State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 down 1.1.1.1 1 downChecking the interface operational state and OSPF operational state we can see that both are reported as being down:
admin@branch6-cli> show ospf interface extensive 192.168.254.9 Codes for operation state (State): BDR - backup designated router; DR - designated router; ODR - other designated router; PTP - point to point dwn - down; lbk - loopback; wtg - waiting Interface Area DR ID BDR ID State --------- ---- ----- ------ ----- vni-0/4.10 0.0.0.0 none none dwn Interface ID: 12 Admin state: enabled; MTU: 1500 Interface op state: down; OSPF op state: down <<<<<< Link IP address: 192.168.254.9; Mask: 255.255.255.252 Area: 0.0.0.0; Router ID: 6.6.6.6; Network type: point-to-point Instance ID: 3014; Cost: 1; Priority: 1 Number of OSPF interface state changes or error: 2 LSA count: 0; Checksum: 0x00000000 Timer intervals configured Hello: 10 secs; Dead: 40 secs Transit delay: 1 sec; Retransmit: 5 secsTroubleshooting should start with checking unicast connectivity from the peer router:
IOU1#ping vrf Tenant1 192.168.254.9 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.254.9, timeout is 2 seconds: ..... Success rate is 0 percent (0/5)After the L2 issue is fixed the unicast reachability between the peers is OK and the OSPF adjacency should come up fine:
IOU1#ping vrf Tenant1 192.168.254.9 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 192.168.254.9, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR routing-instance Tenant1-LAN-VR State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up
3. OSPF Passive Interface
Let’s look at the next issue, same two routers but a different problem. Let’s check it out:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief % No entries found.As you can see there is no OSPF neighbor. By checking if OSPF is enabled on the interface everything looks OK:
admin@branch6-cli> show ospf interface brief Codes for operation state (State): BDR - backup designated router; DR - designated router; ODR - other designated router; PTP - point to point dwn - down; lbk - loopback; wtg - waiting Interface Area DR ID BDR ID State --------- ---- ----- ------ ----- vni-0/4.10 0.0.0.0 none none PTPHowever, if OSPF traffic is sniffed we can see that only one router is sending OSPF hellos:
admin@branch6-cli> tcpdump vni-0/4 filter "host 224.0.0.5" | grep "vlan 10" 03:54:30.992335 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 94: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 56 03:54:40.020335 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 94: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 56 03:54:49.844133 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 94: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 56 03:54:58.968134 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 94: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 56 03:55:08.124232 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 94: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 56If an interface is configured as passive, then its network will still be advertised but it won’t send any OSPF hello packets. This way it’s impossible to form an OSPF neighbor adjacency. After the issue is fixed, we can see OSPF hellos sent by both routers and the adjacency coming up:
04:07:20.624153 0c:e9:f6:f0:7f:05 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 86: vlan 10, p 0, ethertype IPv4, 192.168.254.9 > 224.0.0.5: OSPFv2, Hello, length 48 04:07:22.452133 aa:bb:cc:00:01:10 > 01:00:5e:00:00:05, ethertype 802.1Q (0x8100), length 98: vlan 10, p 0, ethertype IPv4, 192.168.254.10 > 224.0.0.5: OSPFv2, Hello, length 60
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.11 vni-0/4.10 full 1.1.1.1 1 up
4. OSPF Multicast Filtering
Next scenario, same two routers but different issue. Again, the two routers are not establishing the neighborship:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 init 1.1.1.1 1 up IOU1#show ip ospf 1 neighborInteresting, Versa device is showing the OSPF neighbor to be in the INIT state, while its peer is showing nothing. Since the Versa appliance is showing the INIT state we can draw the conclusion that it’s receiving something from R2. Its peer isn’t showing anything so it’s probably not receiving anything. OSPF uses hello packets to establish the OSPF neighbor adjacency and these are sent using the 224.0.0.5 multicast address. Let’s check on the Versa device if this address is reachable over vni-0/4.10:
admin@branch6-cli> ping 224.0.0.5 routing-instance Tenant1-LAN-VR interface vni-0/4.10 count 1 PING 224.0.0.5 (224.0.0.5) from 192.168.254.9 vni-0_4.10: 56(84) bytes of data. --- 224.0.0.5 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0msHowever, the peer is reachable via its unicast address:
admin@branch6-cli> ping 192.168.254.10 routing-instance Tenant1-LAN-VR count 1 PING 192.168.254.10 (192.168.254.10) 56(84) bytes of data. 64 bytes from 192.168.254.10: icmp_seq=1 ttl=255 time=1.58 ms --- 192.168.254.10 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.582/1.582/1.582/0.000 msChecking the Versa’s peer we can see that the OSPF multicast address is filtered out:
interface Ethernet0/1.10 encapsulation dot1Q 10 ip vrf forwarding Tenant1 ip address 192.168.254.10 255.255.255.252 ip access-group 100 in <<<<<< ip ospf network point-to-point ip ospf 1 area 0.0.0.0
IOU1#show access-lists 100 Extended IP access list 100 10 deny ip any host 224.0.0.5 log-input (556 matches) <<<<<< 20 permit ip any any (6230 matches)After the filtering issue is solved the OSPF multicast address is reachable on the Versa appliance and the neighborship is coming up:
admin@branch6-cli> ping 224.0.0.5 routing-instance Tenant1-LAN-VR interface vni-0/4.10 count 1 PING 224.0.0.5 (224.0.0.5) from 192.168.254.9 vni-0_4.10: 56(84) bytes of data. 64 bytes from 192.168.254.10: icmp_seq=1 ttl=255 time=1.77 ms
--- 224.0.0.5 ping statistics --- 1 packets transmitted, 1 received, 0% packet loss, time 0ms rtt min/avg/max/mdev = 1.778/1.778/1.778/0.000 ms admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up
5. Mismatching OSPF network type
This time Versa appliance interface is configured as OSPF Network type broadcast and the peer router has its interface configured as point-to-point. Now the OSPF state is reported as 2-WAY. This means that the peers are seeing each other. The 2-WAY state is attained when the router receiving the hello packet sees its on Router ID within the received hello packet’s neighbor field. At this state, a router decides whether to become adjacent with this neighbor. However, this decision is based on media type:- Broadcast (become adjacent only with the DR and BDR)
- Point-to-Point or Point-to-Multipoint (become adjacent with all the connected routers)
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 2-way 1.1.1.1 1 upIt is easy to spot the issue by checking the OSPF network type on both routers. As you can see on the Versa side the interface network type is broadcast and DR/BDR systems were elected:
admin@branch6-cli> show ospf interface detail 192.168.254.9 Codes for operation state (State): BDR - backup designated router; DR - designated router; ODR - other designated router; PTP - point to point dwn - down; lbk - loopback; wtg - waiting Interface Area DR ID BDR ID State --------- ---- ----- ------ ----- vni-0/4.10 0.0.0.0 192.168.254.9 192.168.254.10 DR Interface ID: 12 Admin state: enabled; MTU: 1500 Interface op state: up; OSPF op state: designated router Link IP address: 192.168.254.9; Mask: 255.255.255.252 Area: 0.0.0.0; Router ID: 6.6.6.6; Network type: broadcast Instance ID: 3014; Cost: 1; Priority: 1On the peer side the network type is point-to-point:
IOU1#show ip ospf 1 interface brief Interface PID Area IP Address/Mask Cost State Nbrs F/C Et1/3.10 1 0.0.0.0 30.30.30.1/24 10 DR 0/0 Et0/2.10 1 0.0.0.0 192.168.254.14/30 10 P2P 0/1 Et0/1.10 1 0.0.0.0 192.168.254.10/30 10 P2P 1/1 <<<<<<After the network type mismatch is solved the adjacency is fully coming up. As a note, neighbors reported in 2-WAY state is normal in a broadcast network type for normal routers (if it is not elected as a DR or a BDR).
6. OSPF Subnet Mask mismatch
Once more the OSPF adjacency is having issues:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR routing-instance Tenant1-LAN-VRTwo peers are checking if the subnet is matching via the OSPF hello packets. To troubleshoot this issue OSPF debugging for hello packets should be performed:
admin@branch6-cli> show configuration routing-instances Tenant1-LAN-VR protocols ospf 3014 debug | display set set routing-instances Tenant1-LAN-VR protocols ospf 3014 debug flags [ hello ] set routing-instances Tenant1-LAN-VR protocols ospf 3014 debug level debugThe log events are saved on the bellow path:
/var/log/versa/ rtdtrc.logIn the log we can see the “network mask” mismatch and also that the peer OSPF hello packets are discarded.
20-06-26 07:09:32.00: qonmhllo.c 0075(Tenant1-LAN-VR): hello buf 0x7f478f785fb0, if_cb 0x7f4787d57030, router_id 0x7ffeccd3fc20 20-06-26 07:09:32.00: qon2hllo.c 0067(Tenant1-LAN-VR): qonm_verify_hello_packet 20-06-26 07:09:32.00: qon2hllo.c 0094(Tenant1-LAN-VR): hello interval 10 expected 10 20-06-26 07:09:32.00: qon2hllo.c 0180(Tenant1-LAN-VR): Network masks do not match 20-06-26 07:09:32.00: qon2hllo.c 0300(Tenant1-LAN-VR): Invalid packet received - log problem 3 20-06-26 07:09:32.00: qonmhllo.c 0350(Tenant1-LAN-VR): Mismatched network masks 20-06-26 07:09:32.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:09:32.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 7 20-06-26 07:09:32.00: qonmhllo.c 0145(Tenant1-LAN-VR): Bad packet - discardingAs a note, the subnet mask is checked only on the multi-access interfaces and is ignored on point-to-point links. The source of this seemingly strange behavior is Section 10.5 of RFC 2328 which says: “However, there is one exception to the above rule: on point-to-point networks and on virtual links, the Network Mask in the received Hello Packet should be ignored.” Versa OS conforms strictly to the RFC and allows OSPF neighbors to form adjacency over a point-to-point link even when the subnet masks do not match. After fixing the “network mask” mismatch the OSPF adjacency is coming up.
7. OSPF Hello & Dead Interval mismatch
The check of hello & dead interval timers matching is also happening via OSPF hello packets exchange. On the Versa appliance peer the default values of OSPF hello and dead timers were changed:interface Ethernet0/1.10 encapsulation dot1Q 10 ip vrf forwarding Tenant1 ip address 192.168.254.10 255.255.255.252 ip ospf network point-to-point ip ospf dead-interval 44 <<< default value is 40 ip ospf hello-interval 11 <<< default value is10 ip ospf 1 area 0.0.0.0 endChecking the OSPF debug log on the Versa appliance we’ll see that the received OSPF hello packet is discarded as the hello interval is not matching the local one. The dead interval is not even checked as the decision was already taken. The dead interval is checked only in case of hello interval matching for the peers.
20-06-26 07:37:07.00: qon2hllo.c 0094(Tenant1-LAN-VR): hello interval 11 expected 10 20-06-26 07:37:07.00: qon2hllo.c 0107(Tenant1-LAN-VR): Hello intervals do not match 20-06-26 07:37:07.00: qon2hllo.c 0300(Tenant1-LAN-VR): Invalid packet received - log problem 1 20-06-26 07:37:07.00: qonmhllo.c 0300(Tenant1-LAN-VR): Mismatched hello intervals 20-06-26 07:37:07.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:37:07.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 8 20-06-26 07:37:07.00: qonmhllo.c 0145(Tenant1-LAN-VR): Bad packet - discardingAfter the hello & dead intervals are fixed the OSPF adjacency is coming up:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up
8. OSPF Area mismatch
One more time the adjacency is not coming up:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going upIntf address Interface State Neighbor ID Pri Op ———— ——— —– ———– — — 192.168.254.10 vni-0/4.10 down 1.1.1.1 1 up In this case the Versa peer is part of OSPF area 0.0.0.1:
interface Ethernet0/1.10 encapsulation dot1Q 10 ip vrf forwarding Tenant1 ip address 192.168.254.10 255.255.255.252 ip ospf network point-to-point ip ospf 1 area 0.0.0.1 <<<<<< endVersa log is not very explicit this time however we can see that every 10 sec (default OSPF hello interval time) a packet is dropped:
20-06-26 07:56:23.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:56:23.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 2 20-06-26 07:56:32.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:56:32.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 2 20-06-26 07:56:42.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:56:42.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 2 20-06-26 07:56:51.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 07:56:51.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 2OSPF adjacency is coming up after the area mismatch is fixed on the Versa peer:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up
9. OSPF authentication mismatch
OSPF adjacency is in down state between Versa and its peer as the OSPF authentication key is different:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR routing-instance Tenant1-LAN-VR State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 down 1.1.1.1 1 upIn the Versa log we can see the same kind of output as for the previous issue, however the reason code for dropping the OSPF hello packets is different (Reason code: 6 vs Reason code: 2)
20-06-26 08:20:03.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 08:20:03.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 6 20-06-26 08:20:13.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 08:20:13.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 6 20-06-26 08:20:22.00: qonmhllo.c 0503(Tenant1-LAN-VR): qonm_packet_dropped_event 20-06-26 08:20:22.00: qonmhllo.c 0522(Tenant1-LAN-VR): Reason code: 6After the OSPF authentication is fixed the adjacency is coming up:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up [ok][2020-06-26 08:33:09]The same troubleshooting method could be used if only one peer has authentication configured or if the authentication method is different (clear text vs MD5)
10. IP MTU mismatch on the interface connecting the peers
In this situation the OSPF state is reported as EXSTART:admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up
Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 exst 1.1.1.1 1 upThe troubleshooting should start by checking the local interface MTU and trying to send ICMP echo requests (ping) using the payload size reported in the MTU – L2 encapsulation overhead. Also, the DF bit should be set to prevent IP fragmentation.
admin@branch6-cli> show interfaces detail vni-0/4.10 | grep MTU MTU : 1500 admin@branch6-cli> ping 192.168.254.10 routing-instance Tenant1-LAN-VR count 1 packet-size 1472 df-bit enable PING 192.168.254.10 (192.168.254.10) 1472(1500) bytes of data. --- 192.168.254.10 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0msInterface MTU check between two peers happens during the Link State Database Synchronization process. There is the interface MTU field in the OSPF Database Description message. Interface MTU specifies the largest unfragmented packet that can be sent from the originating interface. If the MTUs do not match, the peers do not become adjacent. Otherwise, one neighbor might send messages larger than the other neighbor can receive. The troubleshoot should continue by enabling the OSPF database description packet tracing:
set routing-instances Tenant1-LAN-VR protocols ospf 3014 debug flags [ database-description ] set routing-instances Tenant1-LAN-VR protocols ospf 3014 debug level debugChecking the log we’ll see that the OSPF state is not moving over the EXSTART state:
20-06-29 02:05:45.00: qoamddsc.c 0106(Tenant1-LAN-VR): qoam_rcv_db_description 20-06-29 02:05:45.00: qoamddsc.c 0309(Tenant1-LAN-VR): Neighbor is in state 5 20-06-29 02:05:45.00: qoamddsc.c 0362(Tenant1-LAN-VR): INIT or EXCHANGE_START 20-06-29 02:05:45.00: qoamddsc.c 0422(Tenant1-LAN-VR): State is EXCHANGE_START 20-06-29 02:05:45.00: qoamddsc.c 0457(Tenant1-LAN-VR): dd_flags 0X07, dd data_len 8 20-06-29 02:05:45.00: qoamddsc.c 0459(Tenant1-LAN-VR): nbr rtr ID 0X01010101 our rtr ID 0X06060606 20-06-29 02:05:45.00: qoamddsc.c 0461(Tenant1-LAN-VR): dd seq no 8554 nbr dd seq no 1613411 20-06-29 02:05:45.00: qoamddsc.c 0478(Tenant1-LAN-VR): packet empty is True 20-06-29 02:05:45.00: qoamddsc.c 0525(Tenant1-LAN-VR): Remaining in EXCHANGE_START, flags 0X00000000 20-06-29 02:05:45.00: qoamddsc.c 0839(Tenant1-LAN-VR): NBR NM flags now 0X00000000^Cand also we can see the MTU mismatch between the peers (Local MTU = 1500B (05 DC in HEX), remote MTU = 1400B (05 78 in HEX)):
20-06-29 02:05:43.00: qonmuser.c 0511(Tenant1-LAN-VR): qonm_user_pkt_sent 20-06-29 02:05:43.00: qonmuser.c 0549(Tenant1-LAN-VR): DATABASE DESCRIPTION packet 20-06-29 02:05:43.00: qonmuser.c 0630(Tenant1-LAN-VR): Area id 0.0.0.0 20-06-29 02:05:43.00: qonmuser.c 0633(Tenant1-LAN-VR): Neighbor router ID 1.1.1.1 20-06-29 02:05:43.00: qonmuser.c 0636(Tenant1-LAN-VR): Destination IP Address 224.0.0.5 20-06-29 02:05:43.00: qonmuser.c 0642(Tenant1-LAN-VR): Interface index 0X0000000C 20-06-29 02:05:43.00: qonmuser.c 0644(Tenant1-LAN-VR): Interface IP address 192.168.254.9 20-06-29 02:05:43.00: qonmscks.c 1395(Tenant1-LAN-VR): 45 c0 00 34 00 00 00 00 01 59 00 00 c0 a8 fe 09 e0 00 00 05 20-06-29 02:05:43.00: qonmscks.c 1395(Tenant1-LAN-VR): 02 02 00 20 06 06 06 06 00 00 00 00 0b 73 00 00 00 00 00 00 20-06-29 02:05:43.00: qonmscks.c 1395(Tenant1-LAN-VR): 00 00 00 00 05 dc 42 07 00 18 9e 63 20-06-29 02:05:45.00: qonmuser.c 0310(Tenant1-LAN-VR): qonm_user_pkt_received 20-06-29 02:05:45.00: qonmuser.c 0348(Tenant1-LAN-VR): DATABASE DESCRIPTION packet 20-06-29 02:05:45.00: qonmuser.c 0425(Tenant1-LAN-VR): Area id 0.0.0.0 20-06-29 02:05:45.00: qonmuser.c 0428(Tenant1-LAN-VR): Neighbor router ID 1.1.1.1 20-06-29 02:05:45.00: qonmuser.c 0431(Tenant1-LAN-VR): Destination IP Address 192.168.254.10 20-06-29 02:05:45.00: qonmuser.c 0437(Tenant1-LAN-VR): Interface index 0X0000000C 20-06-29 02:05:45.00: qonmuser.c 0439(Tenant1-LAN-VR): Interface IP address 192.168.254.9 20-06-29 02:05:45.00: qonmscks.c 1395(Tenant1-LAN-VR): 45 c0 00 40 84 69 00 00 01 59 b6 d6 c0 a8 fe 0a c0 a8 fe 09 20-06-29 02:05:45.00: qonmscks.c 1395(Tenant1-LAN-VR): 02 02 00 20 01 01 01 01 00 00 00 00 82 f2 00 00 00 00 00 00 20-06-29 02:05:45.00: qonmscks.c 1395(Tenant1-LAN-VR): 00 00 00 00 05 78 52 07 00 00 21 6a ff f6 00 03 00 01 00 04 20-06-29 02:05:45.00: qonmscks.c 1395(Tenant1-LAN-VR): 00 00 00 01After MTU mismatch is fixed the adjacency is coming up:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR brief State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up [ok][2020-06-29 02:18:07]
11. OSPF Adjacency is Stuck in Loading
This is a quite rare problem in OSPF neighbor relationship. It could be a result of an event like LS request is being made and neighbor is sending bad packet or memory corruption exists. Also, the LS update could become corrupted on the way if there are issues with the transmission links. In the bellow output the Versa appliance peer is in OSPF Loading state:IOU1#show ip ospf 1 neighbor Neighbor ID Pri State Dead Time Address Interface 7.7.7.7 0 FULL/ - 00:00:36 192.168.254.13 Ethernet0/2.10 6.6.6.6 0 LOADING/ - 00:00:38 192.168.254.9 Ethernet0/1.10Versa device reports a full OSPF adjacency:
admin@branch6-cli> show ospf neighbor org Tenant1 routing-instance Tenant1-LAN-VR extensive State codes: atmpt - attempt, exchg - exchange, exst - exchange start, load - loading, 2-way - two-way, full - full Op codes: gdown - going down, gup - going up Intf address Interface State Neighbor ID Pri Op ------------ --------- ----- ----------- --- -- 192.168.254.10 vni-0/4.10 full 1.1.1.1 1 up Area: 0.0.0.0 DR: none BDR: none Admin state: up Relationship state with neighbor: full Neighbor priority: 1; Options: 0x52; Re-transmission queue length: 5 <<<<<< Number of neighbor relationship state changes or error: 144 Permanence: dynamic Hellos suppressed: no Requested LSAs: 0 Dead timer due in: 00:00:22 (hrs:mins:secs:n/a) Hitless restart status: not helping Remaining hitless restart interval: none Hitless restart result: noneHowever, something is wrong as the retransmission-queue length always has OSPF updates waiting to be sent on the wire. This is most likely a result of these updates being corrupted on the wire or when the LSAs are kept/inserted in the LS database (memory corruption).
