Slow DNS resolving with enabled natdnsproxy

[rynop]

Outgoing network connections are very slow with VMs using 3.13+ kernel.

Below you can see the curl connect phase is taking 5+ seconds. Problem exists reguardless of using virtio or intel adapters.

vagrant@vagrant-ubuntu-trusty-64:~$ curl -o /dev/null -s -w %{time_connect}:%{time_starttransfer}:%{time_total}\\n https://www.google.com
5.559:5.703:5.706
vagrant@vagrant-ubuntu-trusty-64:~$ uname -a
Linux vagrant-ubuntu-trusty-64 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
vagrant@vagrant-ubuntu-trusty-64:~$ lsmod
Module                  Size  Used by
vboxsf                 43786  1
nfsd                  280297  2
auth_rpcgss            59338  1 nfsd
nfs_acl                12837  1 nfsd
nfs                   236636  0
lockd                  93977  2 nfs,nfsd
sunrpc                284404  6 nfs,nfsd,auth_rpcgss,lockd,nfs_acl
fscache                63988  1 nfs
dm_crypt               23177  0
ppdev                  17671  0
psmouse               102222  0
serio_raw              13462  0
parport_pc             32701  0
parport                42348  2 ppdev,parport_pc
vboxguest             248441  2 vboxsf
ahci                   25819  1
libahci                32168  1 ahci
e1000                 145174  0

Here is my Vagrantfile that allows for simple re-creation of this issue:

 -*- mode: ruby -*-
# vi: set ft=ruby :

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
#  config.vm.box = "ubuntu/trusty64"
  config.vm.box = "ubuntu-trusty64-20140429"
  config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/trusty/20140429/trusty-server-cloudimg-amd64-vagrant-disk1.box"

  config.vm.network :private_network, ip: "192.168.33.10"
  config.vm.synced_folder "..", "/vagrant"

  config.vm.provider :virtualbox do |vb|
     vb.gui = false
     vb.name = "slowconnection"

     # Use VBoxManage to customize the VM. For example to change memory:
     vb.customize ["modifyvm", :id, "--memory", "1024"]
  end
end

nslookup on www.google.com is fast, netperf is fast, download of files are fast once connection is made, mtr is fast (low response times).

[rynop]

So I've narrowed it down to DNS. My guess is something in the vbox dns server (10.0.2.3).

I changed from this:

nameserver 10.0.2.3
...

to google DNS:

nameserver 8.8.8.8
nameserver 8.8.4.4
...

and things fly.

I should note that I have a ubuntu 12.04 VM on this same box, also using nat, with the exact same /etc/resolv.conf and I do not have any dns issues. So pretty sure its not my router, or the DNS that my host machine is using.

I also have 14.04 installed on bare metal on another box in my office and it does not have any DNS issues (gets dns VIA DHCP - which is my gateway ip). So I don't think its a problem with my router or my ISP's DNS server.

[Frank Mehnert]

So far we were not able to reproduce this problem.

[rhuddleston]

I just wanted to comment I have the exact same issue. I found this ticket when researching the problem. I also am using both 12.04 and 14.04 with vagrant on virtual box and only have the problem with dns lookup on 14.04. Also of note that nslookup has no issues, only things like curl and /usr/lib/nagios/plugins/check_http etc and you can see it hang for 5 seconds at requests for DNS at 10.0.2.3 when using strace. I also can fix the problem by changing resolve.conf to different DNS.

Virtualbox 4.3.20

[Frank Mehnert]

I would like to see a VBox.log file to see your exact VM configuration.

[Frank Mehnert]

I just again executed the above curl command in a 14.10 VM:

vbox@vboxguest:~$ curl -o /dev/null -s -w %{time_connect}:%{time_starttransfer}:%{time_total}\\n https://www.google.com
0,053:0,178:0,178

Looks reasonable fast to me. I didn't use vagrant, I set up the VM manually (VBox 4.3.20).

[Frank Mehnert]

Actually the problem can be solved by VBoxManage modifyvm VM_NAME --natdnsproxy off. So activating this option slows down the above command for unknown reason. That is probably a bug but that has nothing to do with Linux 3.13.

[rhuddleston]

added the following to our Vagrantfile to fix the issue:

...

   mon.vm.box = "ubuntu/trusty64"   
   mon.vm.provider "virtualbox" do |v|
        v.memory = 2048
        v.customize ["modifyvm", :id, "--natdnsproxy1", "off"]
    end

...

[Valery Ushakov]

Please provide a packet capture of DNS traffic both from the guest and from the host.

[cerberos]

I have this problem with a vagrant machine that uses phusionpassenger-ubuntu-14.04-amd64-vbox, adding v.customize ["modifyvm", :id, "--natdnsproxy1", "off"] to my Vagrantfile fixed the problem for me. Thanks