Messaging Gateway

 View Only
Expand all | Collapse all

ControlCenter hang quarantine email and queue

  • 1.  ControlCenter hang quarantine email and queue

    Posted Oct 16, 2021 06:30 PM
    Hello
    from week I have a problem with controlcenter
    CPU (8vCPU/16 GB RAM/100GB HDD) highest usage > 20%

    poczta [10.7.4-13]> monitor system
    system: %user %sys %wait memt memu memf
    24.88% 6.88% 0.00% 16247908 8370404 7877504


    Message Queue stuck on
    Control-Center
    61191 421 4.4.0 [internal] Failed to connect: no mail servers for this domain could be reached at this time
    When i try to open any message from spam quarantine the message is not showing and Web-GUI  "freeze" 

    i try 

    cc-config database --status
    cc-config database --check
    cc-config database --optimize

    service controlcenter restart

    poczta [10.7.4-13]> patch list
    SMG patch installation history:
    patch-10.7.4-288 2021-10-13 17:54

    In log (catalina/admin) I don't see anything special

    only restart help fix for 6-7 hours (CPU is less then 10%)

    Daily I have aprox 600 messages (200 Clean), temporary I change resend time to 1-15 minutes (from 15m-4h)

    Any solutions?

    best Regards
    Jerzy


  • 2.  RE: ControlCenter hang quarantine email and queue

    Broadcom Employee
    Posted Oct 22, 2021 06:19 PM

    Hi Jerzy,


    How many emails are stuck in queue and in which queue (Inbound/Outbound/Delivery queue)? Could you share a screen capture of Message Audit Log for an email that might be pending in queue and screen capture of the queue.
    You may want to check connectivity using the telnet command as given in article for Inbound/Outbound emails that you are experiencing issues with -

    SMTP Code: "421 4.4.0 [internal] failed to connect: no mail servers for this domain could be reached at this time"

    Thanks & Regards,
    Isha

     

                                              

     

     



    ------------------------------
    Isha
    ------------------------------



  • 3.  RE: ControlCenter hang quarantine email and queue

    Posted Oct 22, 2021 06:20 PM

    Correct we need a pic?






  • 4.  RE: ControlCenter hang quarantine email and queue

    Broadcom Employee
    Posted Oct 22, 2021 07:03 PM
    You could try to telnet to port 41025 (assuming default configuration) which is the receiving port for email to get put into the spam quarantine.
    If you can connect on that port, then maybe do an ad-hoc SMTP conversation to ensure you can inject mail into the quarantine.
    (this will confirm the quarantine can accept mail)

    The other thing is to double check that your non-local relay host is properly configured.  If the quarantine processor can't access THAT host, mail will just accumulate in the quarantine until it expires and the expunger deletes it. 
    (this will confirm the quarantine can offload/deliver mail)

    One other idea is that maybe the quarantine isn't the primary issue:  that "internal" message may be an indication that the filtering component is crashing.  As Isha pointed out:  are you seeing backup in the inbound, outbound, or delivery queues?  If they are in the inbound or outbound queues, it is more likely that the filtering component is crashing and re-starting. 
    Normal flow is inbound or outbound acceptance -> filtering component -> delivery queue  -> off to the next hop/destination.
    Alternate flow is inbound/outbound acceptance -> filtering component -> quarantine -> quarantine release host -> off to the next hop/destination.

    While you are at it, might as well check the "bad message" queue:  sometimes you get some aberrant email that sucks up resources in the filtering layer, fails, gets re-submitted for filtering, gumming up the works.  If we detect this internally, we will sideline that email in the bad messages queue indefinitely.


  • 5.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 06:48 AM
    Edited by Jerzy Krawczyk Nov 09, 2021 06:58 AM
    Hello
    it this moment I inform IT dep to not display messages with xml attachment in quarantine
    This solve problem with CPU usage and freeze GUI
    but I had only issue in Delivery Queue - we send a lot messages to one domain via internet. Its normally and this domain accept always messages, but if in delivery queue is message with "no Body" 451 4.4.2  then queue with messages to this domain in stuck (421 4.4.0 ). (SMG is IN/OUT gateway). ​​ 
    All others messages was deliver sucesfully. I must restart SMG and all mesagess is released normally.
    of course i check telnet from SMG to this domain and work ok.
    best regards 
    Jerzy


  • 6.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 07:30 AM
    Install 10.7.5. Boom




  • 7.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 10:11 AM
    Update not working

    Error: Package: kmod-microsoft-hyper-v-4.3.2-20190527.smg.x86_64 (installed)
    Requires: kernel(__vsock_core_init) = 0x558bbf63
    Removing: kernel-3.10.0-957.21.3.el7.x86_64 (@symrepo_smg/6)
    kernel(__vsock_core_init) = 0x558bbf63
    Updated By: kernel-3.10.0-1160.36.2.el7.x86_64 (symrepo_smg)
    kernel(__vsock_core_init) = 0x9f5ec719
    You could try using --skip-broken to work around the problem
    You could try running: rpm -Va --nofiles --nodigest
    1


    SMG patch installation history:
    patch-10.7.4-288 2021-10-13 17:54



  • 8.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 10:29 AM

    Not good.  Call support






  • 9.  RE: ControlCenter hang quarantine email and queue

    Broadcom Employee
    Posted Nov 09, 2021 11:16 AM
    Did you do a "2 step" update:  "update download" to stage the new software and then, sometime later do an "update install"?
    This would expose the error you are seeing.   The "fix" is to simply run "update install" again.

    I believe this issue and the mitigation are already documented in the release notes and/or a KB article.

    As you can see from the log, the underlying issue is that the older kernel RPM had a dependency on a couple of Hyper-V specific modules.  The new kernel no longer has those dependencies, hence you see the error message.  Re-running the install forces the issue, and you should be fine.


  • 10.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 11:19 AM

    I approve this message.  Call support






  • 11.  RE: ControlCenter hang quarantine email and queue

    Broadcom Employee
    Posted Nov 09, 2021 11:22 AM
    I approve your approval of my message. ;)


  • 12.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 09, 2021 01:09 PM

    Call support.






  • 13.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 15, 2021 11:53 AM
    Hello
    Patch/new version aplied but not resolve problem
    Messages still sometimes stuck on queue


  • 14.  RE: ControlCenter hang quarantine email and queue

    Posted Nov 15, 2021 12:26 PM

    Yuk.  Call support please






  • 15.  RE: ControlCenter hang quarantine email and queue

    Broadcom Employee
    Posted Nov 15, 2021 12:48 PM
    Yep, call support, but I have to ask:  did you do any of the the things that Isha and myself recommended?
    Perhaps you could save some bit of time with support if you look into those things and have the information available when you interact with customer support.