APs unable to fail to join vSZ from remote location over WAN

Summary

Why APs unable to fail to join vSZ from remote location

Question

Why APs unable to fail to join vSZ from remote location over WAN?

Customer Environment

vSZ,SZ,SCG, AP, Aruba Switch, Firewall

Root Cause

The problem was isolated to ArubaMobility Switch Denying FTP traffic on this site.

Troubleshooting Steps

Customer had a newly installed setup of vSZ and APs deployed at different locations. Few sites were working ok, however from this new site, none of the AP join vSZ. We confirmed that there is no local firewall in this site.  To isolate the problem, we connected one AP in the customers network and in parallel we collected tcpdump on the vSZ. We found that traffic from AP’s public IP was hitting vSZ and we send a response to it. From AP TCPdump we saw LWAPP packets to the vSZ's IP, but no response back.

We suspected that the LWAPP packets are being dropped since, bringing up an AP from a different network works and if it is connected to problematic network, it connects fine to vSZ, the whole point of suspicion was the stage of LWAPP2SCG conversion. To rule this out, we upgraded another AP to 3.0.4 version and it came up fine on vSZ. This concreted our belief that there is some firewall rule/restriction on the network which drops LWAPP packets at the AP end. End customer however need proof to prove that.

Further LWAPP2SCG log analysis, we found that vSZ did receive the LWAPP packets and responded back, however firewall seems to be dropping it:

2016-08-03T10:22:13+00:00 RuckusDev lwapp2scg[38177]: [38177:lwappHandler:185] Info: Packet is coming ::ffff:80.78.71.220 10001 89f2c1b6 1~<<<<<<<<<<<
2016-08-03T10:22:13+00:00 RuckusDev lwapp2scg[38177]: [38177:sendLWAPPPacket:65] Info: Discovery Response sent ~<<<<<<<<<<
2016-08-03T10:22:15+00:00 RuckusDev lwapp2scg[38177]: [38177:signal_handler:342] Info: SIGTERM! ...<<<<<<<<<<
2016-08-03T10:22:15+00:00 RuckusDev lwapp2scg[58239]: [58239:loadSettings:260] Info: Work with FTP aware firewall!<<<<<<<<<<<<<
2016-08-03T10:22:15+00:00 RuckusDev lwapp2scg[58239]: [58239:loadSettings:318] Info: 'deny' <<<<<<<<<<<


Logs from AP  shows a Discovery Request but never gets a response back:

Oct 21 00:59:11 RuckusAP local2.info syslog: Send Discovery Request message to 40.122.171.143(0), seq_num = 2
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type Discovery type(58) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type WTP Descriptor(3) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type WTP Radio Information(4) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type WTP Static IP Address Info(82) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type WTP Board Data(50) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type Vendor Specific(104) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.debug syslog: LWAPP_MSG_DISC_REQ radio id=0, max_vaps=8, mac=2c:c5:d3:0c:c6:08 country code=21843, environment=32
Oct 21 00:59:11 RuckusAP local2.debug syslog: LWAPP_MSG_DISC_REQ radio id=1, max_vaps=8, mac=2c:c5:d3:0c:c6:0c country code=21843, environment=32
Oct 21 00:59:11 RuckusAP local2.notice syslog: --- add message element 37 (model num ext), length 18 : r500
Oct 21 00:59:11 RuckusAP local2.debug syslog: apgroup name is
Oct 21 00:59:11 RuckusAP local2.info syslog: *** Add ME *** type WTP Static IPv6 Address Info(250) to Discovery Request
Oct 21 00:59:11 RuckusAP local2.info syslog: Start Discovery timer, timeout 5 seconds


We asked access to the intermediate switch noticed that the return traffic from vSZ at Aruba Switch had No Sync flag for LWAPP.

(ArubaS1500-24P) #show datapath session table 40.122.171.143

Datapath Session Table Entries
------------------------------
Flags: F - fast age, S - src NAT, N - dest NAT
       D - deny, R - redirect, Y - no syn
       H - high prio, P - set prio, T - set ToS
       C - client, M - mirror, V - VOIP
       Q - Real-Time Quality analysis
       I - Deep inspect, U - Locally destined
       E - Media Deep Inspect, G - media signal
       u - User Index
Source IP/     Destination IP  Prot SPort DPort  Cntr Prio ToS Age Destination TAge UsrIdx UsrVer Flags
Destination MAC
--------------  --------------  ---- ----- -----  ---- ---- --- --- ----------- ---- ------ ------ -----
40.122.171.143  172.16.6.57     17   12223 12223  0/0     0 0   0   0/0/8       d    0      0      FNY  <<<<<<<<
192.168.99.251  40.122.171.143  17   12223 12223  0/0     0 0   0   0/0/8       d    0      0      FSC


Further, customer bypassed their original router and connected Ruckus AP to a Linksys  Router through Aruba1500 (Linksys-Aruba1500-AP). This allowed AP to pass the LWAPP however it was failing to download the firmware. While looking into the datapath session for the client IP, we found that switch was Denying the FTP traffic.

(ArubaS1500-24P) #show datapath session table 192.168.99.254

Datapath Session Table Entries
------------------------------
Flags: F - fast age, S - src NAT, N - dest NAT
       D - deny, R - redirect, Y - no syn
       H - high prio, P - set prio, T - set ToS
       C - client, M - mirror, V - VOIP
       Q - Real-Time Quality analysis
       I - Deep inspect, U - Locally destined
       E - Media Deep Inspect, G - media signal
       u - User Index
Source IP/     Destination IP  Prot SPort DPort  Cntr Prio ToS Age Destination TAge UsrIdx UsrVer Flags Destination MAC
--------------  --------------  ---- ----- -----  ---- ---- --- --- ----------- ---- ------ ------ -----
192.168.99.254  40.122.171.143  6    51130 21     0/0     0 0   0   0/0/8       25   0      0      SDCI <<<<<<<<<<<<<<

 

Resolution

We ruled out that it could be the global firewall configuration or an uplink ACL on the switch denying FTP ingress/egress.
To isolate the switch issue, we bypass the switch and connected AP directly to the Linksys Router. Moment we connected the
AP it joined the vSZ in few minutes. Subsequently, all AP’s joined to the vSZ from this site.

 

“For more information on this topic—including video tutorials—visit the Ruckus Support How-To Hub at https://support.ruckuswireless.com/how-to-hub.”

Article Number:
000005989

Updated:
June 11, 2019 04:53 PM (8 days ago)

Tags:
Configuration, Troubleshooting, SmartCell Gateway, virtual SmartCell Gateway

Votes:
1

This article is:
helpful
not helpful

Working...Please wait

This is here to prevent you from accidentally submitting twice.

The page will automatically refresh.