I've been having a number of strange DNS problems lately, and some have not been solvable. Had another one today that we did discover what was happening. The email dispatch was doing DNS lookup for MX records for chicken-willing.com (changed to protect the innocent) and getting nothing. I ran dig on the command line and get this:
;; Truncated, retrying in TCP mode.
something I've never seen before when doing a DNS lookup, and then I get this:
; DiG 9.3.6-P1-RedHat-9.3.6-25.P1.el5_11.2 mx chicken-willing.com ;; global options: printcmd ;; connection timed out; no servers could be reached
If I try it outside work's servers, I get this:
;; ANSWER SECTION: chicken-willing.com. 7200 IN MX 1 aspmx.l.google.com. chicken-willing.com. 7200 IN MX 5 alt2.aspmx.l.google.com. chicken-willing.com. 7200 IN MX 300 chicken-willing.com.s10a3.psmtp.com. chicken-willing.com. 7200 IN MX 10 aspmx3.googlemail.com. chicken-willing.com. 7200 IN MX 5 alt1.aspmx.l.google.com. chicken-willing.com. 7200 IN MX 10 aspmx2.googlemail.com. chicken-willing.com. 7200 IN MX 200 chicken-willing.com.s10a2.psmtp.com. chicken-willing.com. 7200 IN MX 100 chicken-willing.com.s10a1.psmtp.com. chicken-willing.com. 7200 IN MX 400 chicken-willing.com.s10a4.psmtp.com.
I passed it to our network guy, who did his research and he found what was happening. What we have for chicken-willing.com is more MX records than I have ever seen before, 9 of them. and they are long ones. So when I look it up at work, the results are too big for UDP, and that's why I get the "Truncated, trying again in TCP mode". UDP DNS lookup has a size limit of 512 bytes and this result exceeds that. So it retries the query using TCP. Unfortunately, our network guy has the network locked down so port 53 is only allowed for UDP and not TCP. DNS queries that get retried in TCP are going to be blocked by the firewall. Which is why my query timed out and could not contact any servers.
We have a few possible solutions. The most obvious one is to open up DNS on port 53 to both UDP and TCP. That's what we are going to try tomorrow.