Messaging Gateway

 View Only
Expand all | Collapse all

Message Audit Logs shows status as "Processing Status"

  • 1.  Message Audit Logs shows status as "Processing Status"

    Posted Sep 26, 2022 09:04 AM
    Hi,

    We have a problem as subject suggests, which I have copied from this link Message Audit Logs shows status as "Processing Status"
    Broadcom remove preview
    Message Audit Logs shows status as "Processing Status"
    Issue/Introduction When an SMTP connection is rejected or deferred by Symantec Messaging Gateway (SMG) mail server, the Message Audit Log (MAL) shows its status as "(Processing Status)" for Sender, Recipients, Verdict, Tracker, Delivery, Untested verdicts. This is confusing as the status may not be updated even several hours after the message was processed.
    View this on Broadcom >

     which suggests that the problem is solved with new version, which we have...
    but the problem is still the same...

    It is not just with this one domain (which is by the way on good senders list), some goes through some don't. So what is exactly the problem with the ones who don't? Because SMG doesn't say anything which makes sense... And obviously the problem isn't solved with newer version, because it's still there.

    Anyone can help?



  • 2.  RE: Message Audit Logs shows status as "Processing Status"

    Broadcom Employee
    Posted Sep 30, 2022 01:16 PM
    The article you mention is referencing the "action taken" column in the MAL.  In the older release that wasn't getting properly updated.
    It is difficult to be certain, but given that fact that the Subject and recipient are not filled in, this really looks like the SMTP conversation was terminated very early.  Almost as if, the connection was accepted, got through the "mail from" phase and then dropped before the "rcpt to" completed.

    The audit log entries are created and initialized with the audit-id at connection time and then subsequent data is filled in as the conversation progresses, the message is actually accepted and filtering is performed.  The "(Processing Status) above is telling you that those fields were never updated because the SMTP conversation never went to completion, hence there is no data to fill in. 

    My suggestion would be to check your maillogs for instances of connections being dropped before the SMTP conversation completes.


  • 3.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Sep 30, 2022 01:18 PM
    Check to see if Cisco fw smtp interpolation is enabled on the infront of the smg.




  • 4.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 03, 2022 09:40 AM
    Hi Thomas,

    So SMG fills data only when smtp conversation is completed... Wouldn't be less painful if smg tells why conversation isn't completed in the first place?

    And where to see/check mail logs? Our exchange is behind smg... so these mails are never accepted by exchange.


  • 5.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 03, 2022 09:42 AM
    Check for firewall smtp interpolation




  • 6.  RE: Message Audit Logs shows status as "Processing Status"

    Broadcom Employee
    Posted Oct 03, 2022 12:26 PM
    The mail logs on the SMG.  
    To clarify:  the mal log is populated incrementally.  
    "Wouldn't be less painful if smg tells why conversation isn't completed in the first place?"
    I agree:  In this case it "looks" like a remote server opened a connection to relay mail to the SMG and then stopped sending data, leaving it listening (i.e. with an active read() outstanding).  Again, this is just a guess from what I can see here.
    Ideally, under these conditions if/when the read finally times out, the mallog record for the message should have some kind of "connection abandoned" indicator.  I know that if it happens in the other direction (i.e. SMG is relaying to the next hop and it closes the connection) the mal entry will have some kind of "abort" indicator.
    I'll look into this and see if I can get more information.

    In the meantime, I suggest you follow Alexander's suggestion and see if there is any filtering/action going on at the router level, as that could be a root cause of the issue you are seeing.






  • 7.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 04, 2022 09:22 AM
    Yes, our guy responsible for router is looking for any kind of odd smtp requests... but he did mention something:

    "I've noticed that the Symantec does not offer the STARTTLS command.
    This might stop some servers from sending mails since they can not be encrypted."

    Could this be the issue here?


  • 8.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 04, 2022 09:23 AM
    Show us the tls settings for smtp




  • 9.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 05, 2022 10:43 AM
    It is off...




  • 10.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 05, 2022 10:46 AM
    Pics please




  • 11.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 05, 2022 10:43 AM




  • 12.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 05, 2022 10:47 AM
    Click tls for all connections




  • 13.  RE: Message Audit Logs shows status as "Processing Status"

    Broadcom Employee
    Posted Oct 05, 2022 11:11 AM
    I don't think this is the reason, not abased on your initial screenshot. If it failed due to no STARTTLS, then you wouldn't get to the point of having the Sender address in the MAL. This address is pulled from the MAIL FROM: during the SMTP handshake, which would be past the point of STARTTLS.

    So, the sender connected, provided a MAIL FROM and likely one or more RCPT TO (this would generate the MAL with Audit ID, Sender, and Connection Classification results). The typical point of failure for a MAL like this is during the DATA section, where the mta sends the contents of the message itself (the headers and the viewable body/attachments).

    As was said before, a likely cause of this is something affecting the network before the SMG, such as a firewall or IPS. Recently we have seen an increase of these types of cases where the sender uses an SMTP feature called PIPELINING but then does not properly wait for all necessary responses before closing the connection. This causes a protocol violation because the sender does not follow the SMTP protocol properly. These articles are related (even if you are not seeing "Abort" in MAL):
    Some messages show "Abort message" in SMG's audit logs.
    "421 esmtp: protocol deviation" and "Abort" action for inbound messages listed in Message Audit Logs

    You likely best bet is to audit your network to determine if anything might be interfering with SMTP, as was said previously. Also, it would be helpful to you to reach out to a sender who has had a problem to see if you can get any information on what they saw as the connection failure. They may be able to see a connection deferral such as the "421 esmtp: protocol deviation" mentioned above, or a network connection failure that would help in investigation.

    If you are using SMG 10.7.4 or newer, you can also run a tcpdump to capture network packets for further investigation:
    TCPDUMP usage on Messaging Gateway 10.7.4 and newer



    ------------------------------
    ---------------------------------------------
    Support Engineer
    * Integrated Cyber Defense Exchange
    * Messaging Gateway
    * Packet Shaper
    Symantec Enterprise Division
    Broadcom Software
    ------------------------------



  • 14.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 05, 2022 11:15 AM
    Use tcmpdump in smg cli and then test with the remote IP smtp server, then report back please.




  • 15.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 06, 2022 10:42 AM
    Here's the section from cap file where it can be seen it almost finished... :)


    So, MAL doesn't recognized recipient, but logs did manage to capture it.
    And am I reading this correct, that their server 195.191 returned this error... their server is unavailable?


  • 16.  RE: Message Audit Logs shows status as "Processing Status"

    Posted Oct 06, 2022 11:00 AM
    Looks like the remote server is the issue




  • 17.  RE: Message Audit Logs shows status as "Processing Status"

    Broadcom Employee
    Posted Oct 06, 2022 11:19 AM

    There's your protocol deviation. Since I'm not actually looking at the packet capture, I can't say exactly why the sender is deviating from the SMTP protocol, but I believe what has been seen in other cases is that the sender is using PIPELINING (which your sender is as well) and they do not wait for all required responses before continuing. If I remember correctly, we've seen that the network communication is attempted to close before the QUIT response has been provided. With PIPELINING, the sender MUST wait for some responses before continuing, including the QUIT response (any response that may change the state of the SMTP transaction).

    "421 esmtp: protocol deviation" and "Abort" action for inbound messages listed in Message Audit Logs

    We've seen this occur more frequently over the last several months. What firewall are you using?  One recipient was using Sonicwall and they confirmed that there was some update within the timeframe of the start of this issue, but that's currently anecdotal until we have more confirmations. Either way, check any device in your environment that might affect SMTP transactions or network communications that include SMTP and:

    1) see if there has been any updates in the last few months

    2) disable any functionality or feature that might affect SMTP transactions

    I'm still hoping we can find a common cause so it can be documented.



    ------------------------------
    ---------------------------------------------
    Support Engineer
    * Integrated Cyber Defense Exchange
    * Messaging Gateway
    * Packet Shaper
    Symantec Enterprise Division
    Broadcom Software
    ------------------------------