BrightMail Gateway 9.0.1 - Where does it store the threatcon values?

[JonE]

We are developing a portal from our website that displays the current threatcon level to our customers.  We have seen the threatcon tool that Symantec has that you can attach to your website by copying and pasting some javascript into your page.  We don't want to use that, because it is too big and bulky and doesn't fit our website layout.  I'm not redesigning the website to fit their tool.

We have development access to the MySQL database tables in the brightmail database and can read/write to them.  My question is, where is the value that the control center reads the current threatcon level from? Is it on of the tables in the brightmail MySQL database or do they store it on the filesystem somehwere?  Does anyone know?

Thanks,
Jon

[SSE-JDavis]

We pull this file down to get the information to display:

http://securityresponse.symantec.com/avcenter/threatcon.zip

We don't store it in any MySQL tables that I am aware of.

[JonE]

Ok, thank you.  That should help.  I can auto-download that file every night for the day and parse the file for the content I need.  I saved the .gif files of TC1.gif thru TC4.gif from the BrightMail server.  I will display that based on the xml content, and then link that .gif to your security center link of...

http://www.symantec.com/security_response/threatcon/index.jsp

On another note...

The entire point of having access to the MySQL database started because we have customers complaining that emails are being filtered.  So, our techs have to login to the BrightMail interface, and then run reports against the Message Audit Logs.  So, what we plan on doing is creating some web pages that allow each user to run a report on messages that are to/from them ONLY.

So, when they login to their webmail, they will have a link that will open a new page that allows them to search for emails that went To them or From them within a certain date range.  This will save my techs a TON of time and they won't have access to other email traffic information for other users even within their same domain.

So, long story short... Is the data from the Message Audit Logs stored in the database or on the filesystem?  I assume the filesystem, because I don't see it in the DB anywhere.  Thanks again for your help.

I like the BrightMail web interface btw.  You guys did a good job on it.  It is pretty solid, very functional, and so far... bug free.

[SSE-JDavis]

This seems like a lot of work when we already have a function where users can have a personal spam quarantine they log in to just by setting up Authentication and Recipient Validation with an LDAP source.

 

The MAL is not stored in MySQL but it is indexed and searches are pretty fast, so the index might be in a MySQL table somewhere.

[JonE]

Unforntunately, I'm not sure that will work for us.  The software we use for hosting email is MailSite (www.mailsite.com) and it can store the user's login info via the registry (meant for a single machine) or using a SQL database (used where you have a farm of email servers and one centralized DB).  So, the data is stored in the SQL database for us, because we have multiple email servers for redundancy.  The BrightMail appliance (we are using the VM Image) won't attach to a database for username/password verification and access so the users can login, correct?  It only uses LDAP connectoins.  Or am i missing something?

[SSE-JDavis]

Yes, the source has to be an LDAP server, but it doesn't necessarily have to be the mail server itself. If you have Active Directory you can give your users a mail attribute and we could query that. They would just log in with their Windows credentials.

[JonE]

I'm still not sure how that would work.  We do have Active Directory, but our users aren't in it.  Only our technicians are in Active Directory to administer the servers.  This is on purpose for better security.  The users, and their email settings, are all stored in the database.  The users also have the ability to manually create/edit/remove any user accounts (email accounts) as needed through the MailSite administration screen (a web page).

So, with your method, I would have to rewrite or integrate with the vendor's webmail administration interface so that it would add an active directory account, add mail attributes to it, when they add a new account, update the Active Directory account when they edit it, and make sure I delete when they delete the user's account.

I'm thinking it would still be easier to not worry about integration in BrightMail or in MailSite.  If i create a .NET web-based project and read from the MailSite database to allow login and what report content that is returned, and then read the BrightMail database or filesystem for the actual report content it sounds easier.

If MailSite or Symantec drastically change their format or environment, I "may" not have to change my code if the file system or database don't change.  And since it is outside of their systems, I don't run the risk of 'messing them up'.  And you and I both know that Symantec tries to refuse support for custom integration (which I don't want to even get started on).

Thoughts?  Suggestions?  Ideas?  I don't like doing stuff the hard way, so I'm open for anything you can think of.

[Migration User]

I take it you are a ISP or a service provider that your users, aren't in your LDAP/AD enviroment. Are you hosting mail for mulitple companies? 

Why not an LDAP instance for your e-mail infrastucture.  It should be straight forward to modify your add/remove scripts for mailsite to do the adds to the LDAP instance.

I see a posting in the MailSite forums asking about it's LDAP feature (in 2006).  Are you sure MailSite doesn't have an LDAP service?   This doc references it

http://www.mailsite.com/support/docs/html/1/02/10200.asp

[JonE]

I contacted the manufacturer (Rockliffe) and they confirmed that they only support LDAP V2, which doesn't verify passwords.  It only verifies accounts queries.  They are working on LDAP V3 support, but they have no idea when that will be done. It may be a couple years.

They did say that they support vrfy support. Does BrightMail use this technology? Can we make use of that instead of LDAP so users can access their spam quarantine?

Also, an important question is... even if we got BrightMail to be able to sync with LDAP so users could log into see their spam quarantine, is there a way for us to allow them to report against the incoming/outgoing message logs?  If not, then this doesn't solve our problem anyway.

Another poster recommended using syslog to possibly accomplish my task.  Can I write the transaction data to logs and put them in SQL server so I can run my own reports on the traffic?  The entire point to this is to allow users to search for their incoming/outgoing traffic for their emails EXACTLY like the "Mesage Audit Logs" area does in the BrightMail web interface.

[SSE-JDavis]

Absolutely! We call it Recipient Validation and it would probably drop your unwanted mail flow by around 50% or more.

[Migration User]

We get most from Global Senders, but Rcpt Validation does help, and it seems to cause a downward trend. I think that it suppresses backscatter spam a big plus as well.

[JonE]

I looked at the documentation in BrightMail and there isn't anything about VRFY support.  Can you assist with this task?  I don't know how it works or what settings to enable.  I don't want to do it wrong and have it block emails and have my clients get upset.

Also, i looked at that quick syslog and if that is the only data it gives, then that won't work. It doesn't show what email came from what person at what time and what the result was (filtered, delivered, etc.) or am i missing something?

[JonE]

Cricket, Do you mean that BrightMail catches most through the Symantec Global Bad Senders group in the Reputation->Bad Senders area?

I really need to limit our bandwidth used from spam.  we get about 105,000 connection attempts per day.  Out of those, about 95K are blocked from bad reputation altogether. Another 5K are filtered and deleted because of content we catch through custom filters.  Another 5K are the remaining emails that come through (some are still spam, some are legitimate).  This then results in two more questions...

Using the recipient validation in the proper way should limit phishing and harvest attacks, right?

Any ideas on how I can start limiting the 95K+ connections per day to save bandwidth?

[SSE-JDavis]

Under the Admin tab, you would click on Directory Integration and add a data source. You would choose the Recipient Validation. Choose other for the source and choose then provide us with the IP address of the mail server and what port its running on (Default is 389).

 

Then you would have us either do an anonymous bind or provide us with credentials for your mail server to perform the query.

[Migration User]

I think recipient validation helps a bit (~ 1% for us, your milage varies), but you cut down outbound back scatter too, so it helps twice.  Your stats look somewhat similar to ours.

Yes, once recipient validation is enabled, you can do DHA filter - I gray list for 2 hours.

Reduce bandwidth - how about limiting the max connections per sending IP, max total connections?  Consider looking at connection classification

I don't find Mail to be a big bandwith user.  This is 7 days of one of my 4 scanners:

This was done using SNMP monitoring:

 

Breakdown on filtering:

24 hours for us, one site:

	Content violations:	3,013	0.1
	Viruses:	1,745	0.1
	Invalid recipients:	35,006	1.7
	Bad reputation:	1,990,245	96.0
	Spam:	42,835	2.1

Drilling into Bad reputation:

	Directory harvest attacks:	5,093	0.3
	Virus attacks:	3	< 0.1
	Bad IPs:	749	< 0.1
	Connection Classification:	154,845	7.8
	Symantec Global Bad Senders:	1,829,555	91.9

[Migration User]

Some old data from syslog files to give you a feel.  The facility tag Local_1 is the message audit log data. this tag is specified when you set up the syslog (Remote logging in Symantec speak).

Source IP	Generated	Received	Source Name	Facility	Severity	Tag	Origin	Message
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	mail	Info	ecelerity	Scaner_1	[4956] ML-DELIVERY: Message ID: 92/60-04956-417AAFB4, Audit ID 93766e67-b7b5aae00000135c-14-4bfaa714cc39, Delivery succeeded to host: <147.118.97.217:25>, sender: [e-mail], recipient: [e-mail], Note: [None]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	mail	Info	ecelerity	Scaner_1	[4956] ML-DELIVERY_ATTEMPT: Message ID: 92/60-04956-417AAFB4, Audit ID 93766e67-b7b5aae00000135c-14-4bfaa714cc39, Delivery attempted, sender: [e-mail], recipient: [e-mail]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	mail	Info	ecelerity	Scaner_1	[4956] ML-RECEIVED_RECIPIENT: Message ID: 82/60-04956-417AAFB4, Audit ID: 93766e67-b7b5aae00000135c-14-4bfaa714cc39, recipient: [e-mail]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	mail	Info	ecelerity	Scaner_1	[4956] ML-RECEIVED: Message ID: 82/60-04956-417AAFB4, Audit ID 93766e67-b7b5aae00000135c-14-4bfaa714cc39, Received on: 147.118.110.104:25, from host: 147.118.102.133:1412, sender: [e-mail], Size: 10783, Note: [None]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|DELIVER|147.118.97.217:25|[e-mail]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|IRCPTACTION|[e-mail]|deliver
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|TRACKERID|[e-mail]|AAAAARRH2jw=
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	bmserver	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|VERDICT|[e-mail]|none|ABC|default
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	bmserver	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|UNTESTED|[e-mail]|suspect|safe|opl|content_1272655785899|content_1273272803576|content_1272655306861|content_1272655426102|content_1272655254887|content_1272655065884|content_1272655547739|content_1272655494426|content_1272655350725|content_1272654408544|content_1272654642538|content_1272655731793|fastpass|sys_deny_ip|sys_allow_ip|sys_allow_email|sys_deny_email|dns_allow|dns_deny|user_allow|user_deny|freq_va|freq_dha|freq_sa|connection_class_0|connection_class_1|connection_class_2|connection_class_3|connection_class_4|connection_class_5|connection_class_6|connection_class_7|connection_class_8|connection_class_9|senderauth_fail|senderauth_batv_sign|senderauth_batv_fail|blockedlang|knownlang
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	bmserver	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|SUBJECT| service availability-192.168.16.60-1274717972478-47097
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	bmserver	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|SOURCE|internal
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|SENDER|[e-mail]
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|ACCEPT|147.118.102.133:1412
10.10.10.10	5/24/2010 16:19	5/24/2010 11:19	Scaner_1	local 1	Info	ecelerity	Scaner_1	1274717972|93766e67-b7b5aae00000135c-14-4bfaa714cc39|ORCPTS|[e-mail]

[Migration User]

Syslog Q:  I think you are missing something.  The last column is the details. Each of the "Local 1" items show the message AuditID so they can be tied together.  I've deleted that part for clarity, here is the rest of the data, with comments. These are exactly what's on the Message Audit Log details screen.

DELIVER|147.118.97.217:25|[e-mail]
     Delivered to 147.118.97.217 port 25 to recipient [e-mail]

 

IRCPTACTION|[e-mail]|deliver
     For this recipient, [e-mail], this msg was delivered (vs invalid recipient, etc)

VERDICT|[e-mail]|none|abc|default
     What policies were applied?  In this group ABC, default policy

ACCEPT|147.118.102.133:1412
     Message accepted from 147.118.102.133, port 1412

ORCPTS|[e-mail]
    List of orginal recipient (pre address resolution/alias rewriting)

SUBJECT| service availability-192.168.16.60-1274717972478-47097
    Subject of the message

SOURCE|internal
    Interface the message arrived on

SENDER|[e-mail]
    Sender of the message

[JonE]

When adding a data source in the Directory Integration area, the only items i have available to me in the 'Directory Type' dropdown are

1. Active Directory
2. Active Directory Global Catalog
3. iPlanet/Sun ONE/Java Directory Server
4. Domino
5. Other

The screen itself says 'LDAP Server Configuration'.  Something doesn't seem right to me.  Could it be a version difference? I have V 9.

I did a telnet to my mail server on port 25. I entered 'VRFY <email address>' for both a valid and invalid email and they responded accordingly.  What am I missing?  I don't see anything that allows me to setup VRFY info/connection test in the 'Directory Integration' section.

[Migration User]

SBG doesn't do VRFY, it does an LDAP lookup against a directory source.  VRFY would happen TOO LATE, since it would happen during delivery to your internal mail server. The LDAP based recipient validation happens while the remote SENDER is submitting the mail. SBG rejects mMail for invalid recipients before the message body is even accepted.  This makes it the SENDER MTA's job to handle the reject.  With VRFY, you'd end up generating a back-scatter bounce, which might be aimed at an innocent 3rd party.  You might end up on a block list this way.

You need a box running one of the LDAP servers you listed, listening on port 398 or 636 (over SSL).  You might need to spin up a standalone AD environment, or use an open source LDAP that functions like one of the above. See http://en.wikipedia.org/wiki/List_of_LDAP_software#Server_software for some suggestions.

[Migration User]

Another suggestion about your question on limiting e-mail bandwidth.  Can you, or your ISP implement QOS - Quality of Service controls on inbound port 25?

[SSE-JDavis]

I see, I thought you were just confusing an LDAP lookup with the VRFY command.

We don't act as a proxy so that will not work. We would need an LDAP source to query which you had stated your mail server supports LDAP queries.

[JonE]

I see what you are saying about LDAP, but I disagree.  There is no reason the SBG wouldn't be able to send a vrfy command to our backend email servers to verify email addresses just like using active directly for harvest attack checks.  When the request comes in, instead of querying an LDAP, it would run a VRFY command with the recipent's email address to see if it returns OK or BAD ADDRESS, and a respond appropriately.  Can you explain why this would be a problem? 

I wrote my own syslog server in .NET a couple of minutes ago to capture the data from one of the scanners.  it is spitting out a bunch of data, but I don't know what the delimiter values are or the column values.  Is there a lookup chart for this data so I can parse it properly and put it into the database?

Here are some of the results of the captures....

<142>Oct  5 11:45:14 BrightMail02 ecelerity: 1286293514|0a1e1e71-b7b14ae000001881-28-4cab48089252|DELIVER|10.30.30.69:25|[e-mail]
<142>Oct  5 11:45:14 BrightMail02 ecelerity: 1286293514|0a1e1e71-b7b14ae000001881-2c-4cab480a9262|ACCEPT|122.175.74.185:2316
<142>Oct  5 11:45:16 BrightMail02 ecelerity: 1286293516|0a1e1e71-b7b14ae000001881-31-4cab480cfac5|IRCPTACTION|<none>|msg_reject_other
<142>Oct  5 11:45:17 BrightMail02 ecelerity: 1286293517|0a1e1e71-b7b14ae000001881-32-4cab480df598|ACCEPT|209.47.140.180:38441
<142>Oct  5 11:45:17 BrightMail02 ecelerity: 1286293517|0a1e1e71-b7b14ae000001881-32-4cab480df598|IRCPTACTION|<none>|msg_reject_other
<142>Oct  5 11:45:17 BrightMail02 ecelerity: 1286293510|0a1e1e71-b7b14ae000001881-26-4cab48066495|ACCEPT|69.63.178.170:48714
<142>Oct  5 11:45:17 BrightMail02 ecelerity: 1286293511|0a1e1e71-b7b14ae000001881-26-4cab48066495|SENDER|[e-mail]
<142>Oct  5 11:45:19 BrightMail02 ecelerity: 1286293519|0a1e1e71-b7b14ae000001881-34-4cab480f86fa|ACCEPT|209.47.140.179:38558
<142>Oct  5 11:45:19 BrightMail02 ecelerity: 1286293519|0a1e1e71-b7b14ae000001881-34-4cab480f86fa|IRCPTACTION|<none>|msg_reject_other
<142>Oct  5 11:45:19 BrightMail02 ecelerity: 1286293519|0a1e1e71-b7b14ae000001881-35-4cab480f25a8|ACCEPT|209.47.140.179:38620
<142>Oct  5 11:45:19 BrightMail02 ecelerity: 1286293519|0a1e1e71-b7b14ae000001881-35-4cab480f25a8|IRCPTACTION|<none>|msg_reject_other
<142>Oct  5 11:45:20 BrightMail02 ecelerity: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|ORCPTS|[e-mail]
<142>Oct  5 11:45:20 BrightMail02 bmserver: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|SOURCE|external
<142>Oct  5 11:45:20 BrightMail02 bmserver: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|SUBJECT| exclusive online offer, try an electronic cigarette on us!
<142>Oct  5 11:45:20 BrightMail02 bmserver: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|MSGID| <[e-mail]>
<142>Oct  5 11:45:20 BrightMail02 bmserver: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|UNTESTED|[e-mail]|gray|suspect|content_1265648077507|content_100|content_1281714689890|user_allow|user_deny|freq_dha|freq_sa|connection_class_0|connection_class_1|connection_class_2|connection_class_3|connection_class_4|connection_class_5|connection_class_6|connection_class_7|connection_class_8|connection_class_9|senderauth_fail|senderauth_batv_sign|senderauth_batv_fail|knownlang
<142>Oct  5 11:45:20 BrightMail02 bmserver: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|VERDICT|[e-mail]|spam|default|spam: delete (incoming)
<142>Oct  5 11:45:20 BrightMail02 ecelerity: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|TRACKERID|[e-mail]|AAAAARZCC8g=
<142>Oct  5 11:45:20 BrightMail02 ecelerity: 1286293520|0a1e1e71-b7b14ae000001881-33-4cab480eec86|IRCPTACTION|[e-mail]|delete
<142>Oct  5 11:45:20 BrightMail02 ecelerity: 1286293518|0a1e1e71-b7b14ae000001881-33-4cab480eec86|ACCEPT|66.7.198.225:51705
<142>Oct  5 11:45:20 BrightMail02 ecelerity: 1286293519|0a1e1e71-b7b14ae000001881-33-4cab480eec86|SENDER|[e-mail]
<142>Oct  5 11:45:21 BrightMail02 ecelerity: 1286293521|0a1e1e71-b7b14ae000001881-36-4cab48111d5d|ACCEPT|209.47.140.180:38703
<142>Oct  5 11:45:21 BrightMail02 ecelerity: 1286293521|0a1e1e71-b7b14ae000001881-36-4cab48111d5d|IRCPTACTION|<none>|msg_reject_other
<142>Oct  5 11:45:22 BrightMail02 ecelerity: 1286293522|0a1e1e71-b7b14ae000001881-37-4cab4812fd09|ACCEPT|209.47.140.180:38793
<142>Oct  5 11:45:22 BrightMail02 ecelerity: 1286293522|0a1e1e71-b7b14ae000001881-37-4cab4812fd09|IRCPTACTION|<none>|msg_reject_other

[SSE-JDavis]

Sure, it is possible it is just not a function the Brightmail Gateway can perform.

 

You should post that as an Idea and I would upvote it.

[JonE]

Can you please provide a link or general area of where I can request this?  Your site is very big.  Thanks for your help on the VRFY stuff.

Also, now that I can get some of the log data, any idea where i can get the specs for the data being returned from the syslog info?

[SSE-JDavis]

If you mouse over the Security section at the top of the page you can go down to Security and click Ideas.

 

The delimiter is the pipe (|),  but I don't know what the columns would be named. They seem pretty self explanatory.

[Migration User]

check you message audit log for the same message and you can map them yourself.   one of your examples includes this message AuditID: 0a1e1e71-b7b14ae000001881-33-4cab480eec86

Pull the logs for this message and compare to what your .net tool does.

[JonE]

JDavis, is there anyway you can get the code chart for the syslogs from the dev team?  It appears that items have a code to them, example <142> appears to be message log related, where <30> appears to be errors for the server DNS lookup, etc., but I don't really know.  This would save me a lot of time and guessing.  While it may seem to be self explanatory, it really isnt.

Like, some say ecelerity in them. WTF is that? Some say crond, and others say bmserver.  What would be the difference between ecelerity and bmserver? they look the same.  thanks for your help.

[SSE-JDavis]

Ecelerity is our MTA, bmserver is what downloads the spam definition updates. Crond is the part of Linux that schedules tasks.

[Migration User]

Jon,

 

     I'll be honest with you.  It sounds like with all of the cusomizations you are making you would be much happier with the Brightmail SDK

Title: 'How to obtain the SDK for Symantec Brightmail Message Filter'
Document ID: 2010012810412954
> Web URL: http://service1.symantec.com/SUPPORT/ent-gate.nsf/docid/2010012810412954?Open&seg=ent
 

In reality the appliance is locked down to prevent tampering and possible modifcation that would effect the way it works.  The SDK will allow integration with your MTA as well as extensive configuration options.

 

-John