This post covers multi-layer troubleshooting of 802.1X authentication on wireless devices. The problem station in this post is running Windows 10, trying to authenticate to the “Sharp House” SSID, and authenticates against my Windows server configured with NPS. We’ll start at layer 1 and look at 802.11 frames and the state machine, tools in Windows 10 to show client-side information, Windows NPS Server configuration, and the available Cisco 9800 WLC troubleshooting tools.

My home lab contents:

• Dell Poweredge T610 running ESXi 6.7
• Windows Server 2012R2
  • Domain Services
  • Certificate Authority
  • NPS/RADIUS
• Windows 10 Pro with Orinoco 802.11ac USB Adapter
• Cisco 9800-CL Virtual Wireless Controller
• Cisco 9120 Wireless Access Point

Layer 1

We’ll start from the bottom at layer 1. Be sure you are typing your password correctly! Also, from the wireless perspective, make sure there isn’t so much interference that your device can’t even transmit frames without collisions or corruption. You can see this by using a spectrum analyzer or by reviewing a packet capture and seeing a very large percentage of retries.

Layer 2 – Authentication

Now we’ll look at layer 2 and the 802.11 state machine. I am using Wireshark to view captured authentication, association, and EAP/TLS frames. Starting at state 1 of the 802.11 state machine, we look within the capture to ensure that the device has completed the open system authentication process. This is a simple authentication request/response exchanged between the station and the AP.

Below we can see each of these frames side-by-side. Sequence #1 is the authentication request from the station and sequence #2 is the authentication response from the AP. In our case, each show successful. If either of these frames show unsuccessful, this means that there is something configured on the access point or controller that is limiting clients from joining for load balancing or if WEP is in use but not supported/configured on the client side. Open system authentication supports the use of WEP, but it has been deprecated since the release of 802.11i-2004.

Layer 2 – Association

The next step of the 802.11 state machine, still at layer 2, is association. After completing the open system authentication process, the station will send an association request frame to the AP. The AP will respond with an association response frame containing a status code and an association ID (AID) that is unique to the client.

If the association response returns “unsuccessful”, the station either isn’t compatible with the BSS, doesn’t meet the minimum-security requirements for the BSS, or the AP is rejecting for one of many other reasons. The 802.11-2016 standard provides a list of status codes that can be included in the association response frame.

Status code	Name	Meaning
0	SUCCESS	Successful.
1	REFUSED,	Unspecified failure.
	REFUSED_REASON_UNSPECIFIED
2	TDLS_REJECTED_ALTERNATIVE_PROVIDED	TDLS wakeup schedule rejected but alternative schedule provided.
3	TDLS_REJECTED	TDLS wakeup schedule rejected.
4		Reserved.
5	SECURITY_DISABLED	Security disabled.
6	UNACCEPTABLE_LIFETIME	Unacceptable lifetime.
7	NOT_IN_SAME_BSS	Not in same BSS.
8-9		Reserved.
10	REFUSED_CAPABILITIES_MISMATCH	Cannot support all requested capabilities in the Capability Information field.
11	DENIED_NO_ASSOCIATION_EXISTS	Reassociation denied due to inability to confirm that association exists.
12	DENIED_OTHER_REASON	Association denied due to reason outside the scope of this standard.
13	UNSUPPORTED_AUTH_ALGORITHM	Responding STA does not support the specified authentication algorithm.
14	TRANSACTION_SEQUENCE_ERROR	Received an Authentication frame with authentication transaction sequence number out of expected sequence.
15	CHALLENGE_FAILURE	Authentication rejected because of challenge failure.
16	REJECTED_SEQUENCE_TIMEOUT	Authentication rejected due to timeout waiting for next frame in sequence.
17	DENIED_NO_MORE_STAS	Association denied because AP is unable to handle additional associated STAs.
18	REFUSED_BASIC_RATES_MISMATCH	Association denied due to requesting STA not supporting all of the data rates in the BSSBasicRateSet parameter, the Basic HT-MCS Set field of the HT Operation parameter, or the Basic VHT-MCS and NSS Set field in the VHT Operation parameter.
19	DENIED_NO_SHORT_PREAMBLE_SUPPORT	Association denied due to requesting STA not supporting the short preamble option.
26		Reserved.
27	DENIED_NO_HT_SUPPORT	Association denied because the requesting STA does not support HT features.
28	R0KH_UNREACHABLE	R0KH unreachable.
29	DENIED_PCO_TIME_NOT_SUPPORTED	Association denied because the requesting STA does not support the phased coexistence operation (PCO) transition time required by the AP.
30	REFUSED_TEMPORARILY	Association request rejected temporarily; try again later.
31	ROBUST_MANAGEMENT_POLICY_VIOLATION	Robust management frame policy violation.
32	UNSPECIFIED_QOS_FAILURE	Unspecified; QoS-related failure.
33	DENIED_INSUFFICIENT_BANDWIDTH	Association denied because QoS AP or PCP has insufficient bandwidth to handle another QoS STA.
34	DENIED_POOR_CHANNEL_CONDITIONS	Association denied due to excessive frame loss rates and/or poor conditions on current operating channel.
35	DENIED_QOS_NOT_SUPPORTED	Association (with QoS BSS) denied because the requesting STA does not support the QoS facility.
36		Reserved.
37	REQUEST_DECLINED	The request has been declined.
38	INVALID_PARAMETERS	The request has not been successful as one or more parameters have invalid values.
39	REJECTED_WITH_SUGGESTED_CHANGES	The allocation or TS has not been created because the request cannot be honored; however a suggested TSPEC/DMG TSPEC is provided so that the initiating STA can attempt to set another allocation or TS with the suggested changes to the TSPEC/DMG TSPEC.
40	STATUS_INVALID_ELEMENT	Invalid element i.e. an element defined in this standard for which the content does not meet the specifications in Clause 9.
41	STATUS_INVALID_GROUP_CIPHER	Invalid group cipher.
42	STATUS_INVALID_PAIRWISE_CIPHER	Invalid pairwise cipher.
43	STATUS_INVALID_AKMP	Invalid AKMP.
44	UNSUPPORTED_RSNE_VERSION	Unsupported RSNE version.
45	INVALID_RSNE_CAPABILITIES	Invalid RSNE capabilities.
46	STATUS_CIPHER_OUT_OF_POLICY	Cipher suite rejected because of security policy.
47	REJECTED_FOR_DELAY_PERIOD	The TS or allocation has not been created; however the HC or PCP might be capable of creating a TS or allocation,  in response to a request, after the time indicated in the TS Delay element.
48	DLS_NOT_ALLOWED	Direct link is not allowed in the BSS by policy.
49	NOT_PRESENT	The Destination STA is not present within this BSS.
50	NOT_QOS_STA	The Destination STA is not a QoS STA.
51	DENIED_LISTEN_INTERVAL_TOO_LARGE	Association denied because the listen interval is too large.
52	STATUS_INVALID_FT_ACTION_FRAME_COUNT	Invalid FT Action frame count.
53	STATUS_INVALID_PMKID	Invalid pairwise master key identifier (PMKID).
54	STATUS_INVALID_MDE	Invalid MDE.
55	STATUS_INVALID_FTE	Invalid FTE.
56	REQUESTED_TCLAS_NOT_SUPPORTED	Requested TCLAS processing is not supported by the AP or PCP.
57	INSUFFICIENT_TCLAS_PROCESSING_RESOURCES	The AP or PCP has insufficient TCLAS processing resources to satisfy the request.
58	TRY_ANOTHER_BSS	The TS has not been created because the request cannot be honored; however the HC or PCP suggests that the STA transition to a different BSS to set up the TS.
59	GAS_ADVERTISEMENT_PROTOCOL_NOT_SUPPORTED	GAS Advertisement Protocol not supported.
60	NO_OUTSTANDING_GAS_REQUEST	No outstanding GAS request.
61	GAS_RESPONSE_NOT_RECEIVED_FROM _SERVER	GAS Response not received from the Advertisement Server.
62	GAS_QUERY_TIMEOUT	STA timed out waiting for GAS Query Response.
63	GAS_QUERY_RESPONSE_TOO_ LARGE	GAS Response is larger than query response length limit.
64	REJECTED_HOME_WITH_SUGGESTED_CHANGES	Request refused because home network does not support request.
65	SERVER_UNREACHABLE	Advertisement Server in the network is not currently reachable.
66		Reserved.
67	REJECTED_FOR_SSP_PERMISSIONS	Request refused due to permissions received via SSPN interface.
68	REFUSED_UNAUTHENTICATED_ACCESS_NOT_SUPPORTED	Request refused because the AP or PCP does not support unauthenticated access.
69-71		Reserved.
72	INVALID_RSNE	Invalid contents of RSNE.
73	U_APSD_COEXISTANCE_NOT_SUPPORTED	U-APSD coexistence is not supported.
74	U_APSD_COEX_MODE_NOT_SUPPORTED	Requested U-APSD coexistence mode is not supported.
75	BAD_INTERVAL_WITH_U_APSD_COEX	Requested Interval/Duration value cannot be supported with U-APSD coexistence.
76	ANTI_CLOGGING_TOKEN_REQUIRED	Authentication is rejected because an Anti-Clogging Token is required.

These status codes indicate whether the issue is related to compatibility, configuration, current network/AP status, or other. Understanding the meaning will help guide you through the troubleshooting steps.

Layer 3 – EAP

The next step when trying to connect to an SSID configured with 802.1x security is to perform the EAP exchange. I detail this process in my 802.11 Frame Exchange post. The EAP packet format, as defined in RFC 3748, is quite simple. There are four different codes: Request, Response, Success, and Failure.

Understanding the exchange of frames below is an important step in understanding what to look for in EAP frames. This is a simplified diagram that shows the basic phase 1 EAP frame exchanges. Secure EAP methods such as PEAP or EAP-TLS follow multiple phases that we’ll discuss later.

The AP will request the station for an identity that it sends to the authentication server as an access request.

The station responds with a bogus identity (the real identity is sent in phase 2). In my case “SHARP\Jeremy”.

The AP will forward this information to the authentication server as a RADIUS Access Request. The server responds with a RADIUS Access Challenge that the AP sends to the station as a RADIUS Challenge Request. This frame also contains the EAP Type.

The frame above shows that the authentication server responded with EAP-MS-AUTH, this is not what we want in this scenario. We expect PEAP in the challenge request frame when NPS/RADIUS is properly configured; it should look like the frame below.

The station will send a RADIUS Challenge Response specifying the desired auth type to the AP that it forwards to the authentication server as a RADIUS Access Request. The desired auth type is based on the WLAN profile configuration. If the profile is configured for EAP-MSCHAPv2 then it will respond with that as the desired auth type. So far, we see that the EAP type in the challenge request above and the desired auth type in the challenge response below do not match.

The authentication server will check the contents of the RADIUS Access Request and respond with a RADIUS Access Accept or Reject. The AP forwards the result as an EAP frame, containing a Success or Failure code, to the station.

As we can see above, we’ve got an EAP Failure. Wireless controllers will typically log this result and have it available within the statistics to show how often clients are failing to associate and for what reason. In this case we see a failure because the EAP types configured between the station and the NPS server do not match.

After this failure, the AP sends a deauthentication frame to the station with a reason code, sending it back to state 1 of the 802.11 state machine. This frame shows “Unspecified reason” in the reason code field. The real reason, as discussed already, is that the station requested PEAP but the authentication server is not configured for PEAP.

Layer 4+ – TLS

If you are troubleshooting user or computer-based authentication, know that EAP-MSCHAPv2 is not a secure method of exchanging authentication information by itself. For this reason, it relies on PEAP to create a TLS tunnel then the station and authentication server communicate the username and password information within the tunnel. Tunnel creation is phase 1 of the PEAP process. When the station sends its identity during phase 1, it sends a bogus username. If you perform a packet capture and don’t also see TLS type traffic, the tunnel hasn’t established. To create the tunnel with this form of authentication, a server-side certificate is required. If the station doesn’t trust the issuer of the certificate, it won’t establish the tunnel. Note that EAP-MSCHAPv2 is its own EAP type but does not support certificates and is not secure by itself.

When troubleshooting EAP-TLS, look for the same frames. The difference between the two is that the station will present a certificate of its own to perform mutual authentication. In this case, the TLS tunnel will not be established unless the station trusts the issuer of the server certificate AND the authentication server trusts the issuer of the client’s certificate. When working with Microsoft Server’s CA role, be sure to integrate it with active directory (AD) so that all stations/servers in the domain automatically trust it.

Below we can see the Phase 1 process unprotected (not encrypted) and phase 2 is encrypted (shows as “Application Data”). Before the data is encrypted, the frame exchanges immediately prior show the station requesting the server’s certificate, the authentication server sending the certificate, and the agreement to use PEAP. The 802.11 State Machine PCAP is linked below for download and review.

Station-side

There are occasionally a few issues that point back to the station. Typically, it’s a compatibility issue between the configuration of the BSS and wireless driver. Drivers are often the culprit for many wireless-related issues; it is possible that the drive does not send the correct information during the EAP frame exchange, resulting in failure.

Network shell (netsh) is the go-to tool for viewing anything and everything wireless related. Issue “Netsh WLAN show all” to see:

• Drivers
• Interfaces
• Wireless LAN settings
• Filtered SSIDs based on GPO/User configuration
• Profiles
• Currently visible networks
• Wireless Device Capabilities

First, I check the profile configuration. Each SSID you have connected to will show as a user profile. The profile list can be very long if the device has connected to many different wireless networks; you can issue “netsh wlan show profiles” to see a quick list and “netsh wlan show profile (profile name)” to look at one profile in detail. If you have deployed the WLAN profile using group policy, it will show as a group policy profile. Here we can see that I have a profile by the name of “Sharp House” that I deployed using group policy and a profile named “HowIWiFi” from a previous manual connection. For each profile we can see the details including: EAP type, credentials expected (user or computer), configured ciphers, SSID, whether it will connect automatically, and if the station will connect when the SSID is hidden. In our case, we can see the profile is configured with EAP Type: Microsoft: Protected EAP (PEAP).

The second step I take is to review the drivers and wireless device capabilities to confirm they match up with the profile in question above as well as the SSID configuration on the AP/Controller.

Netsh also features the ability to generate a robust wlan report of recent connections and their statuses by running “netsh wlan show wlanreport”. This report should give you everything you need to identify an issue from the client side. It runs ipconfig, shows the certs on the computer, shows the state machine process for connections, how long the session lasted, what the result was, event IDs, whether the station is automatically trying to connect, and more.

I like to check the statistics at the top to see the primary reasons the connection has failed. Then I’ll scroll to the bottom of the report to find the latest connection attempt, so I am viewing the latest and most accurate information. The report shows all the event IDs and is typically more verbose than using event viewer alone. For this reason, I don’t find that I must check event viewer unless there is some obvious issue with the wireless driver, and it is crashing.

Server Side

There are some obvious mistakes that can be made within the NPS configuration, especially if it wasn’t setup correctly to begin with. Add each AP as a RADIUS client or add the wireless controller, based on your environment. Verify the shared secret matches on both sides. The Connection Request Policy can be as simple as identifying Wireless/802.11 in the NAS Port Type and selecting “Authenticate on this server”.

The network policy should include PEAP in the main “EAP Types” section, not EAP-MSCHAPv2 (shown incorrectly below).

Select “Microsoft: Protected EAP (PEAP)” then “Edit…” and select the server certificate that it is either self-signed, issued by your certificate authority (CA), or trusted third party certificate. In the EAP Types section within PEAP, select EAP-MSCHAPv2. The outer layer is PEAP, which creates the TLS tunnel. The inner EAP version is EAP-MSCHAPv2.

Here is the basic configuration of RADIUS Clients, Connection Request Policy, and Network Policy.

To check the server-side logs (when using Windows Server NPS role), open the NPS console, select “Accounting”, view the Log File Properties section to find the configured location.

You can open the latest log file and scroll to the bottom to see the latest authentication attempts and the network policy that was matched during the authentication attempt. If the station you are troubleshooting is matching a different policy that is above the one you are working with, you may need to reconfigure or re-order the policies by moving them up/down.

This file is typically difficult to parse through because it is comma delimited and, even if you make a copy of it and change the file type to CSV, you may not have an application to view it.

The easiest place to view log files is within the server manager console. Select NAP on the left side, scroll down to the events section, and find an event based on the time of the failure. If you want to filter by “access denied”, enter event ID 6273 in the filter text box.

This is a quick reference to event viewer. You can view NPS events under Custom Views > Server Roles > Network Policy and Access Services.

WLC Troubleshooting

My controller and AP in use are Cisco 9800CL and Cisco 9120. In the 9800, you don’t use the traditional debug commands to troubleshoot AP join and client authentication. Through the GUI there is a Troubleshooting area where you can enter a client mac address and download a log file based on a selected time period. This is a useful tool if you don’t have access to perform an over-the-air packet capture; it shows a new line for each step through the 802.11 state machine the client takes. More information can be found here.

The log file shows authentication failed for the client when it used the username SHARP\Jeremy.

Validate the WLAN configuration shows 802.1X.

The Culprit

The reason for failures in my screenshots was due to only having EAP-MSCHAPv2 as the primary EAP type in the network policy within NPS. This is a common misunderstanding and misconfiguration.

Logs I saw frequently on the station:

1. Reason Code 1078067222 – Network authentication failed. Windows doesn’t have the required authentication method to connect to this network.
2. Error Code: 126 – WLAN Extensibility Module has failed to start. Module Path: C:\Windows\system32\Rtlihvs.dll
3. Wireless Security Failed. Error 0x525 – Unable to identify a user for 802.1X authentication
4. Wireless network is blocked due to connection failure. Length of block timer (minutes): 20

The PCAP below shows the attempt to negotiate PEAP as the authentication type from the station and the authentication server returning an EAP failure message.

Example PCAP

The PCAP below shows the entire 802.11 state machine with PEAP-MSCHAPv2.

• Authentication
• Association
• PEAP Phase 1
• PEAP Phase 2
• Four-way Handshake

Conclusion

I hope these steps and tools allow you to quickly resolve 802.1X/EAP related connectivity issues and better understand the process for next time! I have found that I had the most confusion around EAP, authentication, four-way handshake, and certificates in my earlier days. Even coming from a Microsoft background and being comfortable with configuring certificate authorities, NPS/Radius, and managing active directory for many companies. I find troubleshooting 802.1X to be one of the most frustrating processes since there are often situations where there is an unspecified error, or it isn’t obvious why the station can’t meet the requirements.

Feel free to share some feedback along with any additional tips and tricks! Each wireless system will have its own troubleshooting tools. Thankfully these are quite extensive with the addition of extra radios to APs for packet captures and traffic analytics. Check out my other blog posts to better understand the frame formats, types, and exchanges!

References

• CWSP Certified Wireless Security Professional Study Guide: Exam CWSP-205
• Banner
• EAP Packet Format
• Amits Cisco Zone
• Netsh Command Syntax, Contexts, and Formatting
• How to check the NPS logs in the Event Viewer
• BRKEWN-3013 – Troubleshoot Catalyst 9800 Wireless Controllers