Introduction
This is the twenty-fourth in my Security Series of Connect articles. For more information on how to keep your enterprise environment secure using often-overlooked capabilities of Symantec Endpoint Protection (and the OS upon which it functions), see Mick's Greatest Hits: Index of Helpful Connect Security Articles. This article was last updated in June 2019.
This article provides tips on how to track down the true source of security incidents or infections that are flagged upstream on a DNS server.
Help! My DNS Server is Badly Infected!
Administrators, monitoring for malicious activity in their network, have been known to panic when Windows Event logs or Symantec Endpoint Protection logs display a storm of Intrusion Prevention System (IPS) entries like:
[SID: 30574] Audit: Malicious Domain Request attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
[SID: 24129] Web Attack: Fake Tech Support Website 235 attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
[SID: 29582] System Infected: Ransom.GandCrab Activity attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
[SID: 30190] System Infected: Trojan.Mdropper Activity 10 attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
[SID: 31450] Web Attack: Malicious Phishing Website 25 attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
[SID: 26940] System Infected: Trojan.Jectin Activity 2 attack blocked. Traffic has been blocked for this application: C:\WINDOWS\SYSTEM32\DNS.EXE
Hundreds of IPS events, especially coming from a Domain Controller, should set the alarm bells ringing. But if the events are all related to the server’s legitimate DNS.exe, breathe a sigh of relief.
There is a security event or infection, but it’s not on this server. SEP’s IPS component is reacting to malicious traffic forwarded to this Domain Name Server.
|
DNS is at the core of network communications: it is used by many protocols. Every network has DNS servers within their environment, and, remember, every Domain Controller in a Windows domain is also a DNS server. ISPs have DNS servers to resolve internet domains once traffic leave the local environment, or there are public DNS servers available from Google and others.
DNS servers can be hijacked, poisoned or compromised by threat actors, or known malicious domains can be sinkholed at the DNS server to prevent infection. A full examination of DNS and security is beyond the scope of this article. Here we will just focus on putting an end to the storm of IPS events!
|
If Not Here…. Where?
If one-off IPS events signify that a Fake Tech Support Website or Phishing site was blocked, no further action is needed (except to schedule in some training for the organization on what not to click!) But if the events warn of a system infected with a malware, and these events occur constantly, there’s malware on a computer somewhere in the organization. That is more serious. It’s time to play detective and track down where those DNS requests for malicious sites are coming from.
Sometimes this job is straightforward... export the "Network Threat Protection" - “Attacks” logs from the Symantec Endpoint Protection Manager. When opened in your favorite spreadsheet program and you filter for DNS events, it's possible to see the Remote Host IP where the traffic came from... (some columns removed for clarity)
In this network, the admin knows that 192.168.1.32 and 192.168.1.33 are the network's DNS servers. The events from 192.168.1.33 are caused by recursive DNS traffic, the remote DNS server asking EXAMPLE-DNS if it has information on a malicious domain. Those can be ignored.
The other entry, though, is an endpoint. The machine with IP 192.168.1.166 turned out to be a computer running SEP with AntiVirus only (no IPS, SONAR or other technologies) and very old definitions. There's the infection. The machine was isolated, cleaned, patched and added back to the network with the full SEP suite installed. IPS storm resolved!
What if this job is not that straightforward?
If the SEPM's IPS logs don't point directly to the infected culprit, dig a little deeper.
Sometimes it’s possible to hunt down the source based on known domain name IoCs (Indicators of Compromise). For example, if a network admin sees events “[SID: 29582] System Infected: Ransom.GandCrab Activity attack blocked” and knows that a certain variant of GandCrab ransomware is making the rounds, perhaps from a well-respected news item….
GandCrab Ransomware: Now Coming From Malspam
https://isc.sans.edu/forums/diary/GandCrab+Ransomware+Now+Coming+From+Malspam/23321/
…
First was the HTTP GET request to butcaketforthen.com for the Word document. Next was an HTTP request to sorinnohoun.com by the Word macro for the PowerShell script. After that was post-infection traffic to nomoreransom.coin (an IP address check followed by callback traffic) caused by the GandCrab DLL.
A good starting point is to check if DNS is fielding requests for those domains, and from where.
Another technique to learn the domain name: set up a packet capture tool like Wireshark and record the DNS traffic. Be sure to set a filter for port 53- that will catch UDP or TCP DNS traffic, and nothing else. When the IPS events occur in SEP, check the timestamp and see what domains were being resolved.
Either way: determine the domain name that is responsible for the IPS events. That’s key.
Any Job is Easy When You Use the Right Tool
There is far more DNS activity going on than most users realize. Run this (free!) tool in the background on any Windows computer for an hour or so to see what I mean.
DNSQuerySniffer v1.76
http://www.nirsoft.net/utils/dns_query_sniffer.html
(Symantec is not affiliated with NirSoft in any way- I just personally like and recommend their tools. This particular tool can help threat hunters spot potential C2 traffic, for instance.)
DNSQuerySniffer shows all of the requests, how long they took, what types of resource records were involved, and much more. A visit to one web page may result in a number of DNS queries, thanks to various trackers and so on, and it’s not just web browsers which are responsible for all this activity.
Now imagine that flood of activity, aggregated for a whole organization. There are hundreds of DNS requests per second hitting the network’s DNS servers even in mid-sized offices. Also, keep in mind that Domain Controllers are recursive DNS servers: when they receive queries and the answer is not in their cache, they will forward the request on to the other DNS servers in the domain. It’s busy.
Luckily, there are a couple of tools to make this job easier….
1. DNS Server Logging
Microsoft Windows Server 2012 and above have excellent built-in DNS logging capabilities.
DNS Logging and Diagnostics
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/dn800669(v=ws.11)
Older releases have debug logging that can be enabled for a short period.
Select and enable debug logging options on the DNS server
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc759581%28v%3dws.10%29
Once a log has been collected, use a text editor to narrow the search just to the traffic related to the domain in question: (some columns removed, for clarity…)
[date] [time] 07FC PACKET UDP Snd 10.x.x.199 A (12)nomoreransom(4)coin(0)
[date] [time] 0DD8 PACKET UDP Rcv 10.x.x.199 A (12)nomoreransom(4)coin(0)
[date] [time] 07FC PACKET UDP Snd 10.x.x.18 A (12)nomoreransom(4)coin(0)
[date] [time] 07FC PACKET UDP Snd 10.x.x.199 A (12)nomoreransom(4)coin(0)
[date] [time] 07FC PACKET UDP Snd 10.x.x.2 A (12)nomoreransom(4)coin(0)
[date] [time] 0U81 PACKET UDP Rcv 10.x.x.2 A (12)nomoreransom(4)coin(0)
[date] [time] 0DD8 PACKET UDP Snd 10.x.x.2 A (12)nomoreransom(4)coin(0)
[date] [time] 07FC PACKET UDP Snd 10.x.x.199 A (12)nomoreransom(4)coin(0)
[date] [time] 0DB4 PACKET UDP Rcv 10.x.x.1 A (12)nomoreransom(4)coin(0)
[date] [time] 0DB4 PACKET UDP Snd 10.x.x.18 A (12)nomoreransom(4)coin(0)
[date] [time] 0DB4 PACKET UDP Rcv 10.x.x.1 A (12)nomoreransom(4)coin(0)
[date] [time] 07FC PACKET UDP Snd 198.x.x.54 A (12)nomoreransom(4)coin(0)
Further narrow it down to just the Rcv records- the IP address is where the query came from.
[date] [time] 0DD8 PACKET UDP Rcv 10.x.x.199 A (12)nomoreransom(4)coin(0)
[date] [time] 0U81 PACKET UDP Rcv 10.x.x.2 A (12)nomoreransom(4)coin(0)
[date] [time] 0DB4 PACKET UDP Rcv 10.x.x.1 A (12)nomoreransom(4)coin(0)
[date] [time] 0DB4 PACKET UDP Rcv 10.x.x.1 A (12)nomoreransom(4)coin(0)
An admin familiar with the network and its infrastructure will be able to recognize what IP addresses are for other DNS servers (recursive queries going from server to server) and what IP addresses are endpoints. In this case, the 10.x.x.1 and 10.x.x.2 were Domain Controllers (with DNS servers built in) but 10.x.x.199 was a Windows XP machine. That was the infected one.
2. Wireshark
What if the DNS server is something other than a Microsoft product? Or if it is not possible to generate and examine sufficient logs… a second option is to use a tool like Wireshark to capture the DNS-related traffic. Filter for DNS and search for a string that contains the domain name you seek….
In that example an old endpoint machine at 10.x.x.82 sent a request for the suspicious domain to DNS server 10.x.x.24.
Use Unmanaged Detector
In both examples above, the endpoint which was infected was one that did not have a functioning SEP client installed. This makes sense: if there had been a SEP client with IPS installed and definitions up to date, that IPS event would have happed right there, not upstream at the DNS server!
As mentioned in the Killing Wannacry: How to Eradicate Ransom.Wannacry for Good article, it’s essential to check the network for any computers where SEP is malfunctioning or completely absent.
Configuring a client to detect unmanaged devices
https://www.symantec.com/docs/HOWTO80763
Unprotected or poorly protected computers are easily infected and can give an attacker a means of causing widespread destruction. With ransomware like the GandCrab used in this article, that can be very costly to clean up, after!
Know What’s Normal for Your Network
Just a final tip: keep an eye on what’s going on in your network even when there’s no outbreak underway. Logging what DNS activity occurs can make it easy to spot what is abnormal, both when hunting for threats or when investigating an incident and trying to figure out what happened. Explore Passive DNS!
Conclusion
Many thanks for reading!
Please leave comments and feedback below.