Author: Vittore Zen
A post-mortem analysis of how the behavior of some users and some forgetfulness of the network administrator dragged us into the black hole of the blacklist and how the FlashStart® services helped us to remove the IP address from the blacklist and live happily.
A day of ordinary madness
10:00 | the first phone call
The IT staff is called by the administrative office: they complain because when they scan a document in the multifunction printers the scans (scan to email) do not arrive in their mailbox.
10:05 | the second phone call
The IT staff is called by the marketing department: they complain because for a few days some customers have pointed out that important emails sent to them had been cataloged as Spam and placed in their respective folder.
10:15 | the SMTP server logs
The IT staff checks that the scans are sent correctly to the internal SMTP servers. Then check the correct forwarding of the emails and analyze the files /var/log/mail.log /var/log/mail.info and /var/log/maillog
In the logs of our internal SMTP server on which the scans from the multifunction rely for sending, there are several lines with SMTP 421 error and others with SMTP 550 error.
Recall that 421 errors indicate temporary blocks and the mail server will try to resend the emails.
Example of an SMTP 421 error:
due to the nature of the content and/or the links within. To best protect our users from spam,
the message has been blocked. Please visit https://support.google.com/mail/answer/188131
for more information. u22si16671234pfl.244 – gsmtp
550 errors are permanent failures. You will have to take some action before Gmail removes your server IP address from their blacklist.
SMTP 550 errors indicate permanent failures. You will then need to take some action before, for example, Gmail removes the IP address of the mail server from their blacklist.
Example of an SMTP 550 error:
host gmail-smtp-in.l.google.com [22.214.171.124]:
550-5.7.1 [194.XXX.XXX.181] Our system has detected an unusual rate of
550-5.7.1 unsolicited mail originating from your IP address. To protect our
550-5.7.1 users from spam, mail sent from your IP address has been blocked.
550-5.7.1 Please visit http://www.google.com/mail/help/bulk_mail.html to review
550 5.7.1 our Bulk Email Senders Guidelines. k8si8849879lbl.62
These errors begin to point us towards a problem: the gmail servers are not accepting our traffic because we are on a blacklist.
10:30 | blacklisted
To check if we are on a blacklist, we query the URL https://mxtoolbox.com/blacklists.aspx and indicate the IP address with which our internal SMTP server goes out on the internet.
Oops! We are on the blacklist. Now there are two questions:
» Why are we blacklisted?
» The only solution to solve this problem is to wait?
10:35 | Mikrotik firewall
To answer the first question, the IT staff begins to investigate more deeply. We start by analyzing the perimeter firewall, a Mikrotik device with the latest version of RouterOS on board. There is unusually high traffic coming out of the network. Using the command /tool torch the data tells us that the outgoing traffic comes from the DNS service: our Mikrotik router is resolving names not only for our internal network but also for any client on the internet: Damn, we are an open DNS resolver and we didn’t know it (see the article “Avoid open DNS resolvers on Mikrotik”). We close all DNS ports that we had forgotten open.
Our likely being included in a black list derives from this role we have assumed on the network.
11:00 | the user
The antivirus reports that in a user’s browsing cache there are files attributable to phishing activity. The network administrator calls the user, who confirms that he has clicked on a link that was indicated on an email received, an internet page has opened with a link on which he has clicked and then a notice of the antivirus that does not remember, in any case, has confirmed everything proposed. This too was not needed. What a day! The problems reported at 10 are still not resolved, the emails still do not arrive.
13:00 | advice
The colleague has heard about FlashStart® and its services related to internet filtering and DNS. Objection: but FlashStart® is just a DNS resolver, what does it have to do with blacklists?
13:15 | FlashStart®
You connect to the FlashStart® site and activate a demo version on your network (activate a free trial). FlashStart is an easy to use and efficient Internet Malware and Content Filter, distributed in more than 120 countries.
After a few minutes of configuration, the phishing site is no longer reachable from the users’ workstations: if it had been there before, the user would not have been able to make mistakes by clicking risky clicks. You begin to see the light …
14:00 | the second IP address
On the FlashStart® site (guide) you have read the documentation relating to the removal from the blacklist. The IT staff doesn’t have to do anything, within a few hours the FlashStart® system will do the job for us.
For the moment we change the configuration of the Mikrotik firewall by adding a rule of src-nat to let the internal SMTP server out with a different IP than the public one present in the blacklist:
The mail resumes its normal operation.
The day after
To check if we are still on a blacklist, we query the URL https://mxtoolbox.com/blacklists.aspx and indicate the IP address with which our internal SMTP server went out on the internet.
Hurray! FlashStart® did its job well: we are no longer on the blacklist. Now the problem is definitively solved.