Hello All,
I'm opening this thread because I really need the community's help in determining how to get our Messaging Gateway back to normal.
For Context:
Back on the 4th of July, our area had a really bad storm that knocked out power for our entire town for about a day. After power was restored, we noticed one of the email scanners in our scanner/control center cluster would not boot. Every single checkpoint we reverted to was corrupted and would not boot, so we had to restore our scanner back to an old image saved back when the scanner was still on version 10.7 or so.
Over the course of a couple days, we reactivated the licensing and upgraded the firmware image back up to 10.9.2. We had multiple email delivery issues during that time, but eventually were able to resolve them by clearing the DNS cache on the scanner so all of the domain names we had in there would resolve properly to where they were supposed to go.
Unfortunately, it seems like multiple issues have remained. Ever since them, we constantly see emails intended for various different customer domains get caught up in the queue with the error "421 4.4.0 [internal] Failed to connect: no mail servers for this domain could be reached at this time", "421 4.4.4 [internal] Domain Does Not Resolve", or "450 4.4.1 [internal] Connection Timed Out".
Nothing else inside our network had changed since that day. The scanner was restored with the exact same IP address, and the exact same MAC address. We should not be seeing these issues, as mail has almost always delivered immediately through these scanners.
Some other things to note are:
1 - This problem is affecting BOTH scanners (not just the one we restored)
2 - We've tried clearing every cached item that is possible to be cached in SMG, and besides, considering it's been two months since the original incident you'd think all involved caches would've been cleared by now
3 - There seems to be a reliable pattern: it doesn't affect EVERY email, only some seemingly at random, and it will attempt to deliver the email 3 times, after which on the 4th try it will go through without fail. Sometimes, if I load the queue and attempt to manually flush it, those emails will also deliver, but only sometimes.
4 - This only affects INBOUND emails, not OUTBOUND
I had a ticket open with TD Synnex on the issue and I could not convince them that there was some sort of issue with the way SMG is processing emails. To me, it seems like there's something wrong with the way SMG is resolving domain names, as if they're resolving to the wrong location and trying to send there, and then after enough attempts it finds the real address and successfully delivers the email. Which is definitely weird, considering SMG has settings in the Protocols section for MANUALLY ASSIGNING resolution addresses to domain names. If SMG has a way to manually resolve names in a way the admin wants it to, why wouldn't it prioritize these settings over attempting a bad DNS resolution.
With the way it looks and how I described it, it screams DNS issue to me. Whenever I've had issues with the way emails have been delivered in the past, including emails being delayed in the queue, it was either because I accidentally had an internal DNS server turned off or the SMG scanner just wasn't reaching it. In this scenario, I've confirmed multiple times that the DNS servers referenced by the scanners and control center are turned on and accessible.
I've also been seeing other weird occurrences, like a lot of emails coming in with some sort of attachment get their connection aborted and delayed by 20 - 30 minutes, and sometimes INBOUND and OUTBOUND emails getting delivered to 365 domains will get indefinitely stuck in the queue with the error: "454 4.7.0 Connection is not TLS encrypted. Recipient organization requires TLS.". And this is with the "Attempt TLS connection" setting turned on for the scanner and for every domain in Protocols.
I know this seems like a hyper niche issue, which is exactly why is causing me so much trouble. So many customers are receiving emails later than they should, or sometimes not at all. It can be annoying, but in cases where you need to receive an email with a confirmation code in it it can be completely debilitating, considering they're getting delivered AFTER THEY EXPIRE. If anybody has any advice for things to test or for settings for me to change, I'm all ears. I just need help getting our Messaging Gateway back into the same working condition it was in for YEARS.