confusing firewall block results

[Migration User]

Maybe someone can explain this one!
Firewall rule, define IE as the application.
Ports 80 and 443 outgoing as what to block.
(so far, so good) I want to prevent IE from going to specific sites. Since IP addresses are many and vary for the sites we are blocking, I chose to define the block using domain name.
The example SYMANTEC gives to block using the domain method is *.symantec.com  which means block anything in the symantec.com domain be it pictures.symantec.com or forums.symantec.com and so on. Makes sense - and it typically works. HOWEVER...........
I've setup this rule to block:
*.twitter.com
*.facebook.com
and one other. All using the *.domain.com format
Here's the kicker. In the logs, I see CONSTANT blocking for 209.56.124.23  and  209.56.124.24  and  209.56.124.25
Workstation after workstation show up in the log and all showing as having those three addresses BLOCKED and the name of the rule is the one I used to block those domains!
Now here's the trick - sometimes folks complain "I can't get to walmart.com" or "I can't get to bestbuy.com" and when I check teh rules, sure enough they have been blocked by the twitter rule and those addresses show up!
Ah, but it gets BETTER! Yesterday I saw that rule many dozens of times and when I looked to see what was blocked, it was the above 3 addresses and the log said WWW.SYMANTEC.COM !!!

Now how can a rule defined using DOMAINS constantly block the same 3 IP addresees and with each log entry, show a different site, sometimes including SYMANTEC.COM and other good sites??

Below are two examples of the "details" I see when I highlight the log entry and choose "details" and get the detail popup. I am NOT blocking symantec.com NOR am I blocking foxsports.com, but they were blocked - by a rule defined to block only twitter.com and facebook.com and another - none of the sites being blocked are defined in the rule!
Note the rule name (two groups, same rule with slightly different rule name so it's happening with TWO GROUPS!) note the IP, the domain that was blocked and the rule.
This is weird! It also means you can't use SEP to accurately block anything by domain name -   (and should I be using local/remote or source/destinition for best results?)


Event Type: TCP initiated
Event Time: 09/30/2009 16:04:05
Domain Name: IVRS-SEP1
Site Name: IVRS-SEP01
Server Name: VRDSMSEP2
Group Name: My Company\Client Computers\Desktop
Computer Name  
Current: VR006160RQ2H58B
When event occurred: VR006160RQ2H58B
 
IP Address  
Current: 10.252.6.25
When event occurred: 10.252.6.25
 
Severity: Major
Remote Host Name: www.symantec.com
Remote Host IP: 209.56.124.25
Network Protocol: TCP
Local Port:  1567
Remote Port:  80
Traffic Direction: Outbound
Occurrence: 3
Begin Time: 09/30/2009 16:03:54
End Time: 09/30/2009 16:03:59
Application Name: C:/Program Files/Internet Explorer/iexplore.exe
Blocked: Blocked
Rule Name: Block Twitter-Facebook-Scout
Alert: 0
Location Name: Default
User Name: Kristine.xxxxxxxxx
Domain Name: IVRS-SEP1

---------------------------------------

Event Type: TCP initiated
Event Time: 09/30/2009 16:02:23
Domain Name: IVRS-SEP1
Site Name: IVRS-SEP01
Server Name: VRDSMSEP1
Group Name: My Company\Testing
Computer Name  
Current: VR004260HT1H570
When event occurred: VR004260HT1H570
 
IP Address  
Current: 10.252.170.81
When event occurred: 10.252.170.81
 
Severity: Major
Remote Host Name: msn.foxsports.com
Remote Host IP: 209.56.124.25
Network Protocol: TCP
Local Port:  3557
Remote Port:  80
Traffic Direction: Outbound
Occurrence: 8
Begin Time: 09/30/2009 16:01:20
End Time: 09/30/2009 16:02:17
Application Name: C:/Program Files/Internet Explorer/iexplore.exe
Blocked: Blocked
Rule Name: log-block twitter
Alert: 0
Location Name: Default
User Name: Richard.xxxxx
Domain Name: IVRS-SEP1

 

[Migration User]

ShadowsPapa, doing a traceroute to the 3 IP addresses you listed (209.56.124.23, 209.56.124.24 and 209.56.124.25), all terminate at unnamed hosts inside the Iowa network.  Perhaps these are proxies or other filters?

I was able to create block rules without issue by following the steps in the first part of this document:

http://service1.symantec.com/SUPPORT/ent-security.nsf/docid/2009012915443648

I went ahead and added logging, as well as switched the rule to ask (rather than block) at first, then switched it to block.  The difference in my logs were that the IP address returned for both facebook and my personal website, rather than any internal addresses.

What are these three IP addresses?  If these are proxy or filters of any fashion, would it be possible to take a non-business critical machine and position it in such a way in your network as to not need to pass through any filtering?

Just to verify, did you follow the steps in the above document exactly as written?

What does ping -a facebook.com return?  Does it return an external IP, or does it return one of the 209.x.x.x numbers?

[Migration User]

We can't tracert or ping because ITE has that blocked and we have to pass through ITE to get to ICN for the web.......
But here are the results...not even close to the addresses mentioned.... I'll look at that document again to double-check.

(209.56.124.25, 209.56.124.23, 209.56.124.24  don't resolve to anything here - it comes up empty)


J:\>ping -a facebook.com

Pinging facebook.com [69.63.187.17] with 32 bytes of data:

Request timed out.
Request timed out.

Ping statistics for 69.63.187.17:
    Packets: Sent = 2, Received = 0, Lost = 2 (100% loss),
Control-C
^C
J:\>ping -a www.facebook.com

Pinging www.facebook.com [69.63.181.12] with 32 bytes of data:

Request timed out.

Ping statistics for 69.63.181.12:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
^C

[Migration User]

I'd look into what exactly those three IP addresses are...given that the logs I generated showed the correct IP addresses in the SEPM (as well on the local client), I wonder if what those three addresses are ultimately doing something to DNS responses.

[Migration User]

I've been trying for months to figure that out - but without the ability to properly ping, etc.
I guess I could try from home - I agree, I suspect that it's those things doing something to us.
In some cases, well like this week when the fellow said he wasn't able to get to Bestbuy.com and walmart.com and I saw in the logs it was indeed blocked, THOSE address but with the bestbuy.com name, he rebooted his computer and it worked properly then.
So something is messing with the DNS somewhere.

[Migration User]

Well, you could always just randomly start unplugging stuff...and when someone starts screaming, just say "hey, you know, if only we had ping enabled...it might be helpful to figure things out like this..."

(kidding!)

If you want I can send you the traceroute...maybe it'd help you at least narrow down the scope of what you're looking for?

[Migration User]

OK, more info!
See below - but basically, these are normal akaimi servers on our network. Akaimi supplies servers to customers like the State of Iowa.
The issue seems to be that SEP is blocking by IP and not NAME! When I put in *.twitter.com for exmaple, SEP is resolving the address, which ends up being one of those servers, then blocking by IP address, and since other entities also use Akaimi, they end up gettng blocked. This is why Symantec.com and Walmart.com end up getting blocked..........
So he said "basically your firewalls are not blocking by name even though you set it up that way, it's still blocking by IP and since several other things will resolved to the same IP, it will block them as well"

He suggested pointing the firewalls to other DNS servers, but that can't be done since SEP looks at the local computer for name resolution and we must point to our own servers.
So it seems that according to ICN here - anyone using SEP in the name mode, which is really using IP instead, will be seeing this very soon if not already........

Darwin says:

who owns the following ips: 209.56.124.23, 209.56.124.24 and 209.56.124.25

John Borden says:

those are akaimi web caching servers

20% off all traffic on the internet goes to these servers

they host for microsoft google and multiple other big vendors

John Borden says:

ok have him contact our NOC 323-4400 they can give him the info pretty easy

most customers are concerned we explain and they say ok

[Migration User]

We do query, and *something* is telling us "hey, twitter.com is 209.56.124.23".  We're adding that to a table and anything coming from that address is blocked.  Unfortunately, in the case of web proxies, most if not all traffic will appear to be coming from that address, and since the IP addresses match, it's blocked.

This is something I'd like to see go to development for clarification.  I can understand both sides of the equation...on one hand, merely querying an IP address would be way faster than resolving the name every time, but, on the other hand, if you're behind a proxy, if you're NOT checking by name you'll run into this.

Please contact support and get a case set up for this so we can push it through the channels to development.

[Migration User]

Any way to tell SEP to use a DIFFERENT DNS server? If I could possibly point SEP to other servers.....
OK, will have to call them after bit.
This could be huge, according to the folks at ICN, since web proxies are all over.
The kicker, they said that Iowa is thinking of moving all their web stuff to Akaimi, which means all state web sites would be blocked as well!
Basically, if you try to block anything that is associated with Akaimi, and who isn't associatedf with Akaimi now days - including SYMANTEC, it will block other things also using it - which according to them, is 20% of all web traffic today, add Iowa to that in the future.

[Migration User]

Not that I'm aware of, but, again, that'd need to be something that came down from dev. 

Historically...and I can't see this changing...anytime Symantec has needed to do something that wasn't strictly Symantec, we made Windows calls and used existing Windows tools.  A perfect example is the client remote install tool from the SAV days...when you got a list of computers in the network to deploy out to, that was pulled from a query to Windows' My Network Neighborhood.  I think one of the very few Symantec tools we use is the tiny SMTP engine we use to pass email alerts to a mail server from the SEPM.

Given all that, I suspect this will ultimately come back as "well, we're being told by <this Windows item> that twitter is the Akamai address, so we block that IP."  What I'm hoping will be that dev will make changes so that it will query the dns name itself...but that's well out of my hands.

[Migration User]

I ran into a similar issue awhile back.  To resolve it I created custom IDS signatures looking for specific domains that would silently drop the traffic.  Should work in your case.

[Migration User]

That sounds interesting, however, the creation of such a thing is confusing me a bit (hey, it's Friday)
I get to the first screen where it says Signatures, then the tab Signature Groups and I guess I have no clue what that's for - or any of it for that matter!
I've never created one - and the help isn't telling me what the stuff is for so I'm not sure how to use it.
I wonder if this would be abetter way to block a domain until Symantec can get the above answered or dealt with or it's otherwise resolved.
Why did you go this route - issues similar to ours?

[Migration User]

Yeah and when network neighborhood wasn't properly populated (it hasn't been in ANY place I've ever worked, often lacking a good many computers on complex networks) then the deploy failed or you had to resort to manually typing in the computers...........
I never liked that reliance on Windows - Windows is often wrong.......
Here and PFG I tried to select computers to deploy to from the drop-down only to find it was simply looking to my workstation for the list - and the list was only half-correct so I had to use inventory tools to get a fulls list, export to a text file, trim the text file to the computers I wanted to deploy to, then deploy.
Loved the deployment tool, HATED the fact it looked to the local computer for that list..........

[Migration User]

Somewhat similar.  We were looking to simply detect access on certain "non-approved" websites.  I had not worked with the firewall much at that point so i looked into creating an IDS signature.  Once we saw that users were accessing these specific websites we set it to block and for the most part the problem was resolved. 

I will see if I can write up an article to walk you through the custom rule creation this weekend.  It will likely be done late next week though... I know my wife has lots for me to do around the house.  ;)