vnic VID property is misteriously zeroed on Solaris 11.1 after VM start

[sscdvp]

I use VB 4.2.8

On Solaris 11.1 SRU1.4 VID property for vnic interface is zeroed after VM start. ORACLE VB uses that vnic interface. vnic created on the base of dot1Q trunk. The same problem was observed for vlan interfaces too.

How to reproduce:

1. dladm create-vnic -l net0 -v 998 v4vid998link0
2. dladm show-vnic

   LINK OVER SPEED MACADDRESS MACADDRTYPE VID
   upv4vid201link0 net0 100 2:8:20:5b:23:4c random 201
   upv4vid9link0 net0 100 2:8:20:a1:96:5f random 9
   upv4vid944link0 net0 100 2:8:20:85:4b:6 random 944
   v4vid998link0 net0 100 8:0:27:d1:fe:fe random 998
   

3. Create VM in ORACLE Virtualbox.
4. Add network "bridged adapter" v4vid998link0
5. Start VM
6. dladm show-vnic

   LINK OVER SPEED MACADDRESS MACADDRTYPE VID
   upv4vid201link0 net0 100 2:8:20:5b:23:4c random 201
   upv4vid9link0 net0 100 2:8:20:a1:96:5f random 9
   upv4vid944link0 net0 100 2:8:20:85:4b:6 random 944
   v4vid998link0 net0 100 8:0:27:d1:fe:fe fixed 0
   

7. No IP conectivity in VM.

Workaround:

After VM start restore VID setting: dladm modify-vnic -v 998 v4vid998link0 That helps to restore IP connectivity for that VM.

NOTE 1: the issue appears for the first time after VM started. VM reboot process doesn't affect IP connectivity. Only VM poweroff makes the issue to appear again. NOTE 2: vlan interfaces are affected by the issue too. NOTE 3: Seen the problem in 4.1.x and 4.2.x builds.

[Ramshankar Venkataraman]

I tried reproducing this issue a few weeks back and I wasn't able to. The VID property of the VNIC was intact after starting the VM. Could you please provide the output of:

pkg info kernel
cat /var/adm/messages | grep vbox
modinfo | grep vbox

[sscdvp]

Requested output:

# pkg info kernel
          Name: system/kernel
       Summary: Core Kernel
   Description: Core operating system kernel, device drivers and other modules.
      Category: System/Core
         State: Installed
     Publisher: solaris
       Version: 0.5.11
 Build Release: 5.11
        Branch: 0.175.1.5.0.3.2
Packaging Date: February 22, 2013 06:27:17 PM 
          Size: 32.67 MB
          FMRI: pkg://solaris/system/[email protected],5.11-0.175.1.5.0.3.2:20130222T182717Z


# cat /var/adm/messages | grep vbox

# modinfo | grep vbox
196 fffffffff7f145b8    ce0 304   1  vboxnet (VirtualBox NetAdp 4.2.6r82870)
198 fffffffff7f74000  34f00  40   1  vboxdrv (VirtualBox HostDrv 4.2.6r82870)
264 fffffffff8bd2000   3828 305   1  vboxbow (VirtualBox NetBow 4.2.6r82870)
265 fffffffff8bd6000   4580 306   1  vboxusbmon (VirtualBox USBMon 4.2.6r82870)
266 fffffffff8bdb000   7478 307   1  vboxusb (VirtualBox USB 4.2.6r82870)

[sscdvp]

In order to be able to reproduce your lower link for vnic must be dot1.Q trunk, vnic must be created over physical link. Hope it will help. We were able to reproduce the issue on different hardware and even on non-global zone with Virtualbox installed in. Please don't hesitate to ask more questions.

[sscdvp]

Please look link properties of affected vnic:

# dladm show-linkprop v4vid9link1
LINK     PROPERTY            PERM VALUE        DEFAULT      POSSIBLE
v4vid9link1 autopush         rw   --           --           -- 
v4vid9link1 zone             rw   --           --           -- 
v4vid9link1 state            r-   up           up           up,down 
v4vid9link1 mtu              rw   1500         1500         1500 
v4vid9link1 maxbw            rw   --           --           -- 
v4vid9link1 cpus             rw   --           --           -- 
v4vid9link1 cpus-effective   r-   0            --           -- 
v4vid9link1 rxfanout         rw   --           1            -- 
v4vid9link1 rxfanout-effective r- 1            --           -- 
v4vid9link1 pool             rw   --           --           -- 
v4vid9link1 pool-effective   r-   --           --           -- 
v4vid9link1 priority         rw   high         high         low,medium,high 
v4vid9link1 tagmode          rw   vlanonly     vlanonly     normal,vlanonly 
v4vid9link1 protection       rw   --           --           mac-nospoof,
                                                            restricted,
                                                            ip-nospoof,
                                                            dhcp-nospoof 
v4vid9link1 mac-address      rw   8:0:27:db:5a:1c 8:0:27:db:5a:1c -- 
v4vid9link1 allowed-ips      rw   --           --           -- 
v4vid9link1 allowed-dhcp-cids rw  --           --           -- 
v4vid9link1 rxrings          rw   --           --           -- 
v4vid9link1 rxrings-effective r-  --           --           -- 
v4vid9link1 txrings          rw   --           --           -- 
v4vid9link1 txrings-effective r-  --           --           -- 
v4vid9link1 txrings-available r-  0            --           -- 
v4vid9link1 rxrings-available r-  0            --           -- 
LINK     PROPERTY            PERM VALUE        DEFAULT      POSSIBLE
v4vid9link1 rxhwclnt-available r- 0            --           -- 
v4vid9link1 txhwclnt-available r- 0            --           -- 
v4vid9link1 vsi-typeid       rw   --           --           -- 
v4vid9link1 vsi-typeid-effective r- --         --           -- 
v4vid9link1 vsi-vers         rw   --           --           -- 
v4vid9link1 vsi-vers-effective r- --           --           -- 
v4vid9link1 vsi-mgrid        rw   --           --           -- 
v4vid9link1 vsi-mgrid-effective r- --          --           -- 
v4vid9link1 vsi-mgrid-enc-effective r- --      --           -- 
v4vid9link1 lro              rw   off          auto         on,off,auto 
v4vid9link1 etsbw-lcl        rw   --           0            -- 
v4vid9link1 etsbw-lcl-effective r- --          --           -- 
v4vid9link1 etsbw-rmt-effective r- --          --           -- 
v4vid9link1 etsbw-lcl-advice r-   --           --           -- 
v4vid9link1 cos              rw   --           0            -- 

[sscdvp]

# dladm show-vnic | grep v4vid9link1
LINK                OVER         SPEED  MACADDRESS        MACADDRTYPE       VID
v4vid9link1         net0         100    8:0:27:db:5a:1c   fixed             9

[Ramshankar Venkataraman]

Hm, your last comment shows the vlan-Id is '9' when the VNIC is being used by VBox (type of MAC is 'fixed' and has a VBox MAC address). Does the output reflect the state after you manually re-adjusted the vlan-Id using modify-vnic?

[Ramshankar Venkataraman]

Did some digging, this look like a bug in Solaris which passes a VLAN-ID of zero when it shouldn't be. This is of course only in the code-path specifically used by VirtualBox, userland VNIC operations (like dladm) does not go through this path.

I'll get in touch with the Solaris developers and see if we can get things moving here. Thanks for the report.

[sscdvp]

Replying to ramshankar:

Does the output reflect the state after you manually >re-adjusted the vlan-Id using modify-vnic?

Indeed. After Vm start I am always doing dladm modify-vnic -v 9 v4vid9link1 in order the thing goes right

[sscdvp]

Replying to ramshankar:

I'll get in touch with the Solaris developers and see if we can get things moving here. Thanks for the >report.

I had an opened SR on the problem so I can provide the details and the contact of engineer who had worked on the issue. Unfortunately they insisted on SR closure since they believed the core problem is in Virtualbox code , not Solaris.

[Ramshankar Venkataraman]

Sorry, I'm a a little late on the update.

As indicated in my previous comment it's a Solaris bug. I had filed a bug (well over a month ago) internally with my suggested fix. It's been accepted and fixed in the Solaris code base. For now, the fix appears to be in some build of the next major Solaris version.

[sscdvp]

Great news!

Could you share CR number of that bug for us?

Thanks.

[Ramshankar Venkataraman]

BugDB 16767320.

[sscdvp]

Replying to ramshankar:

BugDB 16767320.

Thanks for providing information. CR 16865027 (the same bug but filled for S11.1) is awaiting to be fixed in S11.1 SRU11.

[sscdvp]

Replying to ramshankar:

BugDB 16767320.

I've tested VB on S11.1 SRU11.4, the fix for bug works.