Wireless 802.1X Troubleshooting

This post covers multi-layer troubleshooting of 802.1X authentication on wireless devices. The problem station in this post is running Windows 10, trying to authenticate to the “Sharp House” SSID, and authenticates against my Windows server configured with NPS. We’ll start at layer 1 and look at 802.11 frames and the state machine, tools in Windows 10 to show client-side information, Windows NPS Server configuration, and the available Cisco 9800 WLC troubleshooting tools.

My home lab contents:

  • Dell Poweredge T610 running ESXi 6.7
  • Windows Server 2012R2
    • Domain Services
    • Certificate Authority
    • NPS/RADIUS
  • Windows 10 Pro with Orinoco 802.11ac USB Adapter
  • Cisco 9800-CL Virtual Wireless Controller
  • Cisco 9120 Wireless Access Point

Layer 1

We’ll start from the bottom at layer 1. Be sure you are typing your password correctly! Also, from the wireless perspective, make sure there isn’t so much interference that your device can’t even transmit frames without collisions or corruption. You can see this by using a spectrum analyzer or by reviewing a packet capture and seeing a very large percentage of retries.

Layer 2 – Authentication

Now we’ll look at layer 2 and the 802.11 state machine. I am using Wireshark to view captured authentication, association, and EAP/TLS frames. Starting at state 1 of the 802.11 state machine, we look within the capture to ensure that the device has completed the open system authentication process. This is a simple authentication request/response exchanged between the station and the AP.

802.11 State Machine

Below we can see each of these frames side-by-side. Sequence #1 is the authentication request from the station and sequence #2 is the authentication response from the AP. In our case, each show successful. If either of these frames show unsuccessful, this means that there is something configured on the access point or controller that is limiting clients from joining for load balancing or if WEP is in use but not supported/configured on the client side. Open system authentication supports the use of WEP, but it has been deprecated since the release of 802.11i-2004.

Authentication Request/Response

Layer 2 – Association

The next step of the 802.11 state machine, still at layer 2, is association. After completing the open system authentication process, the station will send an association request frame to the AP. The AP will respond with an association response frame containing a status code and an association ID (AID) that is unique to the client.

Association Response

If the association response returns “unsuccessful”, the station either isn’t compatible with the BSS, doesn’t meet the minimum-security requirements for the BSS, or the AP is rejecting for one of many other reasons. The 802.11-2016 standard provides a list of status codes that can be included in the association response frame.

Status codeNameMeaning
0SUCCESSSuccessful.
1REFUSED,Unspecified failure.
REFUSED_REASON_UNSPECIFIED
2TDLS_REJECTED_ALTERNATIVE_PROVIDEDTDLS wakeup schedule rejected but alternative schedule provided.
3TDLS_REJECTEDTDLS wakeup schedule rejected.
4Reserved.
5SECURITY_DISABLEDSecurity disabled.
6UNACCEPTABLE_LIFETIMEUnacceptable lifetime.
7NOT_IN_SAME_BSSNot in same BSS.
8-9Reserved.
10REFUSED_CAPABILITIES_MISMATCHCannot support all requested capabilities in the Capability Information field.
11DENIED_NO_ASSOCIATION_EXISTSReassociation denied due to inability to confirm that association exists.
12DENIED_OTHER_REASONAssociation denied due to reason outside the scope of this standard.
13UNSUPPORTED_AUTH_ALGORITHMResponding STA does not support the specified authentication algorithm.
14TRANSACTION_SEQUENCE_ERRORReceived an Authentication frame with authentication transaction sequence number out of expected sequence.
15CHALLENGE_FAILUREAuthentication rejected because of challenge failure.
16REJECTED_SEQUENCE_TIMEOUTAuthentication rejected due to timeout waiting for next frame in sequence.
17DENIED_NO_MORE_STASAssociation denied because AP is unable to handle additional associated STAs.
18REFUSED_BASIC_RATES_MISMATCHAssociation denied due to requesting STA not supporting all of the data rates in the BSSBasicRateSet parameter, the Basic HT-MCS Set field of the HT Operation parameter, or the Basic VHT-MCS and NSS Set field in the VHT Operation parameter.
19DENIED_NO_SHORT_PREAMBLE_SUPPORTAssociation denied due to requesting STA not supporting the short preamble option.
26 Reserved.
27DENIED_NO_HT_SUPPORT Association denied because the requesting STA does not support HT features.
28R0KH_UNREACHABLE R0KH unreachable.
29DENIED_PCO_TIME_NOT_SUPPORTED Association denied because the requesting STA does not support the phased coexistence operation (PCO) transition time required by the AP.
30REFUSED_TEMPORARILYAssociation request rejected temporarily; try again later.
31ROBUST_MANAGEMENT_POLICY_VIOLATIONRobust management frame policy violation.
32UNSPECIFIED_QOS_FAILURE Unspecified; QoS-related failure.
33DENIED_INSUFFICIENT_BANDWIDTHAssociation denied because QoS AP or PCP has insufficient bandwidth to handle another QoS STA.
34DENIED_POOR_CHANNEL_CONDITIONSAssociation denied due to excessive frame loss rates and/or poor conditions on current operating channel.
35DENIED_QOS_NOT_SUPPORTEDAssociation (with QoS BSS) denied because the requesting STA does not support the QoS facility.
36 Reserved.
37REQUEST_DECLINEDThe request has been declined.
38INVALID_PARAMETERSThe request has not been successful as one or more parameters have invalid values.
39REJECTED_WITH_SUGGESTED_CHANGESThe allocation or TS has not been created because the request cannot be honored; however a suggested TSPEC/DMG TSPEC is provided so that the initiating STA can attempt to set another allocation or TS with the suggested changes to the TSPEC/DMG TSPEC.
40STATUS_INVALID_ELEMENTInvalid element i.e. an element defined in this standard for which the content does not meet the specifications in Clause 9.
41STATUS_INVALID_GROUP_CIPHERInvalid group cipher.
42STATUS_INVALID_PAIRWISE_CIPHERInvalid pairwise cipher.
43STATUS_INVALID_AKMPInvalid AKMP.
44UNSUPPORTED_RSNE_VERSIONUnsupported RSNE version.
45INVALID_RSNE_CAPABILITIESInvalid RSNE capabilities.
46STATUS_CIPHER_OUT_OF_POLICYCipher suite rejected because of security policy.
47REJECTED_FOR_DELAY_PERIODThe TS or allocation has not been created; however the HC or PCP might be capable of creating a TS or allocation,  in response to a request, after the time indicated in the TS Delay element.
48DLS_NOT_ALLOWEDDirect link is not allowed in the BSS by policy.
49NOT_PRESENTThe Destination STA is not present within this BSS.
50NOT_QOS_STAThe Destination STA is not a QoS STA.
51DENIED_LISTEN_INTERVAL_TOO_LARGEAssociation denied because the listen interval is too large.
52STATUS_INVALID_FT_ACTION_FRAME_COUNTInvalid FT Action frame count.
53STATUS_INVALID_PMKIDInvalid pairwise master key identifier (PMKID).
54STATUS_INVALID_MDEInvalid MDE.
55STATUS_INVALID_FTEInvalid FTE.
56REQUESTED_TCLAS_NOT_SUPPORTEDRequested TCLAS processing is not supported by the AP or PCP.
57INSUFFICIENT_TCLAS_PROCESSING_RESOURCESThe AP or PCP has insufficient TCLAS processing resources to satisfy the request.
58TRY_ANOTHER_BSS The TS has not been created because the request cannot be honored; however the HC or PCP suggests that the STA transition to a different BSS to set up the TS.
59GAS_ADVERTISEMENT_PROTOCOL_NOT_SUPPORTEDGAS Advertisement Protocol not supported.
60NO_OUTSTANDING_GAS_REQUESTNo outstanding GAS request.
61GAS_RESPONSE_NOT_RECEIVED_FROM _SERVERGAS Response not received from the Advertisement Server.
62GAS_QUERY_TIMEOUT STA timed out waiting for GAS Query Response.
63GAS_QUERY_RESPONSE_TOO_ LARGEGAS Response is larger than query response length limit.
64REJECTED_HOME_WITH_SUGGESTED_CHANGESRequest refused because home network does not support request.
65SERVER_UNREACHABLEAdvertisement Server in the network is not currently reachable.
66 Reserved.
67REJECTED_FOR_SSP_PERMISSIONSRequest refused due to permissions received via SSPN interface.
68REFUSED_UNAUTHENTICATED_ACCESS_NOT_SUPPORTEDRequest refused because the AP or PCP does not support unauthenticated access.
69-71 Reserved.
72INVALID_RSNEInvalid contents of RSNE.
73U_APSD_COEXISTANCE_NOT_SUPPORTEDU-APSD coexistence is not supported.
74U_APSD_COEX_MODE_NOT_SUPPORTEDRequested U-APSD coexistence mode is not supported.
75BAD_INTERVAL_WITH_U_APSD_COEXRequested Interval/Duration value cannot be supported with U-APSD coexistence.
76ANTI_CLOGGING_TOKEN_REQUIREDAuthentication is rejected because an Anti-Clogging Token is required.

These status codes indicate whether the issue is related to compatibility, configuration, current network/AP status, or other. Understanding the meaning will help guide you through the troubleshooting steps.

Layer 3 – EAP

The next step when trying to connect to an SSID configured with 802.1x security is to perform the EAP exchange. I detail this process in my 802.11 Frame Exchange post. The EAP packet format, as defined in RFC 3748, is quite simple. There are four different codes: Request, Response, Success, and Failure.

EAP Packet Format

Understanding the exchange of frames below is an important step in understanding what to look for in EAP frames. This is a simplified diagram that shows the basic phase 1 EAP frame exchanges. Secure EAP methods such as PEAP or EAP-TLS follow multiple phases that we’ll discuss later.

Basic EAP Exchange

The AP will request the station for an identity that it sends to the authentication server as an access request.

EAP Request

The station responds with a bogus identity (the real identity is sent in phase 2). In my case SHARP\Jeremy.

EAP Response

The AP will forward this information to the authentication server as a RADIUS Access Request. The server responds with a RADIUS Access Challenge that the AP sends to the station as a RADIUS Challenge Request. This frame also contains the EAP Type.

RADIUS Challenge Request

The frame above shows that the authentication server responded with EAP-MS-AUTH, this is not what we want in this scenario. We expect PEAP in the challenge request frame when NPS/RADIUS is properly configured; it should look like the frame below.

EAP Type: PEAP

The station will send a RADIUS Challenge Response specifying the desired auth type to the AP that it forwards to the authentication server as a RADIUS Access Request. The desired auth type is based on the WLAN profile configuration. If the profile is configured for EAP-MSCHAPv2 then it will respond with that as the desired auth type. So far, we see that the EAP type in the challenge request above and the desired auth type in the challenge response below do not match.

RADIUS Challenge Response

The authentication server will check the contents of the RADIUS Access Request and respond with a RADIUS Access Accept or Reject. The AP forwards the result as an EAP frame, containing a Success or Failure code, to the station.

Final EAP Frame

As we can see above, we’ve got an EAP Failure. Wireless controllers will typically log this result and have it available within the statistics to show how often clients are failing to associate and for what reason. In this case we see a failure because the EAP types configured between the station and the NPS server do not match.

After this failure, the AP sends a deauthentication frame to the station with a reason code, sending it back to state 1 of the 802.11 state machine. This frame shows “Unspecified reason” in the reason code field. The real reason, as discussed already, is that the station requested PEAP but the authentication server is not configured for PEAP.

Deauthentication Frame

Layer 4+ – TLS

If you are troubleshooting user or computer-based authentication, know that EAP-MSCHAPv2 is not a secure method of exchanging authentication information by itself. For this reason, it relies on PEAP to create a TLS tunnel then the station and authentication server communicate the username and password information within the tunnel. Tunnel creation is phase 1 of the PEAP process. When the station sends its identity during phase 1, it sends a bogus username. If you perform a packet capture and don’t also see TLS type traffic, the tunnel hasn’t established. To create the tunnel with this form of authentication, a server-side certificate is required. If the station doesn’t trust the issuer of the certificate, it won’t establish the tunnel. Note that EAP-MSCHAPv2 is its own EAP type but does not support certificates and is not secure by itself.

PEAP Frame Exchange
PEAP Validates Server Certificate

When troubleshooting EAP-TLS, look for the same frames. The difference between the two is that the station will present a certificate of its own to perform mutual authentication. In this case, the TLS tunnel will not be established unless the station trusts the issuer of the server certificate AND the authentication server trusts the issuer of the client’s certificate. When working with Microsoft Server’s CA role, be sure to integrate it with active directory (AD) so that all stations/servers in the domain automatically trust it.

EAP-TLS Mutual Authentication

Below we can see the Phase 1 process unprotected (not encrypted) and phase 2 is encrypted (shows as “Application Data”). Before the data is encrypted, the frame exchanges immediately prior show the station requesting the server’s certificate, the authentication server sending the certificate, and the agreement to use PEAP. The 802.11 State Machine PCAP is linked below for download and review.

PEAP Phase 1 & 2
Phase 2 EAP Frames Encrypted

Station-side

There are occasionally a few issues that point back to the station. Typically, it’s a compatibility issue between the configuration of the BSS and wireless driver. Drivers are often the culprit for many wireless-related issues; it is possible that the drive does not send the correct information during the EAP frame exchange, resulting in failure.

Network shell (netsh) is the go-to tool for viewing anything and everything wireless related. Issue “Netsh WLAN show all” to see:

  • Drivers
  • Interfaces
  • Wireless LAN settings
  • Filtered SSIDs based on GPO/User configuration
  • Profiles
  • Currently visible networks
  • Wireless Device Capabilities

First, I check the profile configuration. Each SSID you have connected to will show as a user profile. The profile list can be very long if the device has connected to many different wireless networks; you can issue “netsh wlan show profiles” to see a quick list and “netsh wlan show profile (profile name)” to look at one profile in detail. If you have deployed the WLAN profile using group policy, it will show as a group policy profile. Here we can see that I have a profile by the name of “Sharp House” that I deployed using group policy and a profile named “HowIWiFi” from a previous manual connection. For each profile we can see the details including: EAP type, credentials expected (user or computer), configured ciphers, SSID, whether it will connect automatically, and if the station will connect when the SSID is hidden. In our case, we can see the profile is configured with EAP Type: Microsoft: Protected EAP (PEAP).

Sharp House Profile

The second step I take is to review the drivers and wireless device capabilities to confirm they match up with the profile in question above as well as the SSID configuration on the AP/Controller.

Driver Support

Netsh also features the ability to generate a robust wlan report of recent connections and their statuses by running “netsh wlan show wlanreport”. This report should give you everything you need to identify an issue from the client side. It runs ipconfig, shows the certs on the computer, shows the state machine process for connections, how long the session lasted, what the result was, event IDs, whether the station is automatically trying to connect, and more.

netsh wlan show wlanreport
WLAN Report Contents

I like to check the statistics at the top to see the primary reasons the connection has failed. Then I’ll scroll to the bottom of the report to find the latest connection attempt, so I am viewing the latest and most accurate information. The report shows all the event IDs and is typically more verbose than using event viewer alone. For this reason, I don’t find that I must check event viewer unless there is some obvious issue with the wireless driver, and it is crashing.

Server Side

There are some obvious mistakes that can be made within the NPS configuration, especially if it wasn’t setup correctly to begin with. Add each AP as a RADIUS client or add the wireless controller, based on your environment. Verify the shared secret matches on both sides. The Connection Request Policy can be as simple as identifying Wireless/802.11 in the NAS Port Type and selecting “Authenticate on this server”.

Connection Request Policy

The network policy should include PEAP in the main “EAP Types” section, not EAP-MSCHAPv2 (shown incorrectly below).

Incorrect EAP Types Configuration

Select “Microsoft: Protected EAP (PEAP)” then “Edit…” and select the server certificate that it is either self-signed, issued by your certificate authority (CA), or trusted third party certificate. In the EAP Types section within PEAP, select EAP-MSCHAPv2. The outer layer is PEAP, which creates the TLS tunnel. The inner EAP version is EAP-MSCHAPv2.

PEAP Properties

Here is the basic configuration of RADIUS Clients, Connection Request Policy, and Network Policy.

Basic NPS Configuration

To check the server-side logs (when using Windows Server NPS role), open the NPS console, select “Accounting”, view the Log File Properties section to find the configured location.

NPS Log File Location

You can open the latest log file and scroll to the bottom to see the latest authentication attempts and the network policy that was matched during the authentication attempt. If the station you are troubleshooting is matching a different policy that is above the one you are working with, you may need to reconfigure or re-order the policies by moving them up/down.

NPS Log File

This file is typically difficult to parse through because it is comma delimited and, even if you make a copy of it and change the file type to CSV, you may not have an application to view it.

The easiest place to view log files is within the server manager console. Select NAP on the left side, scroll down to the events section, and find an event based on the time of the failure. If you want to filter by “access denied”, enter event ID 6273 in the filter text box.

Server Manger Events

This is a quick reference to event viewer. You can view NPS events under Custom Views > Server Roles > Network Policy and Access Services.

Event Viewer – NPS Events
NPS Event Views

WLC Troubleshooting

My controller and AP in use are Cisco 9800CL and Cisco 9120. In the 9800, you don’t use the traditional debug commands to troubleshoot AP join and client authentication. Through the GUI there is a Troubleshooting area where you can enter a client mac address and download a log file based on a selected time period. This is a useful tool if you don’t have access to perform an over-the-air packet capture; it shows a new line for each step through the 802.11 state machine the client takes. More information can be found here.

9800 Troubleshooting Page

The log file shows authentication failed for the client when it used the username SHARP\Jeremy.

9800 Radioactive Tracing File

Validate the WLAN configuration shows 802.1X.

Cisco 9800 WLC – WLAN Configuration

The Culprit

The reason for failures in my screenshots was due to only having EAP-MSCHAPv2 as the primary EAP type in the network policy within NPS. This is a common misunderstanding and misconfiguration.

Logs I saw frequently on the station:
  1. Reason Code 1078067222 – Network authentication failed. Windows doesn’t have the required authentication method to connect to this network.
  2. Error Code: 126 – WLAN Extensibility Module has failed to start. Module Path: C:\Windows\system32\Rtlihvs.dll
  3. Wireless Security Failed. Error 0x525 – Unable to identify a user for 802.1X authentication
  4. Wireless network is blocked due to connection failure. Length of block timer (minutes): 20

The PCAP below shows the attempt to negotiate PEAP as the authentication type from the station and the authentication server returning an EAP failure message.

Example PCAP

The PCAP below shows the entire 802.11 state machine with PEAP-MSCHAPv2.

  • Authentication
  • Association
  • PEAP Phase 1
  • PEAP Phase 2
  • Four-way Handshake
802.11 State Machine – PEAP-MSCHAPv2

Conclusion

I hope these steps and tools allow you to quickly resolve 802.1X/EAP related connectivity issues and better understand the process for next time! I have found that I had the most confusion around EAP, authentication, four-way handshake, and certificates in my earlier days. Even coming from a Microsoft background and being comfortable with configuring certificate authorities, NPS/Radius, and managing active directory for many companies. I find troubleshooting 802.1X to be one of the most frustrating processes since there are often situations where there is an unspecified error, or it isn’t obvious why the station can’t meet the requirements.

Feel free to share some feedback along with any additional tips and tricks! Each wireless system will have its own troubleshooting tools. Thankfully these are quite extensive with the addition of extra radios to APs for packet captures and traffic analytics. Check out my other blog posts to better understand the frame formats, types, and exchanges!

References

One thought on “Wireless 802.1X Troubleshooting

Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Create a website or blog at WordPress.com

Up ↑

%d bloggers like this: