ESXi 5.1 Seperating SAN traffic w/Vlans

[gssieg]

So I've learned this week that vSphere 5.1 VmKernels no longer support multiple gateways.  This is causing me some confusion with how to properly setup my networking for my SAN.  I thought it was best practices to seperate the traffic so I created seperate vlans for management and for data (SAN) traffic.  Since they are in different subnets they both have their own gateway.  When trying to configure this I started having quite a bit of problems before I realized that the gateway must stay the same.  I contacted VMware and originally they told me the gateways can be changed before finally stating they were incorrect and that there can only be 1 gateway.  The response I received from them at that point is that I do in fact want to split the vlans and to just leave the gateway on my vmkernel for san traffic alone they stated that as long as my vlans were properly setup that vmware would just know where to send the data and it would not matter.  When this "magically" was not working they told me I must have a vlan issue and that they could not help me.

Could anyone give me some insight as to what the best method is for this?  I found an article that states you can manually add a second gateway via the CLI but when I tried I received an error stating the route already existed.

Vlan 18 (172.27.18.x/24 w/gw 172.27.18.1) - management

Vlan 40 (172.27.40.x/24 w/gw 172.27.40.1) - data/SAN

Any help would be greatly appreciated.

[a_p_]

Basically you only need a gateway if traffic needs to be routed between subnets. As long as the storage system is on the same subnet as the VMkernel ports used for storage access on your ESXi hosts you don't need to specify the gateway.

André

Message was edited by: a.p.

[gssieg]

Andre,

Forgive me but doesn't that defeat the purpose of seperating my storage traffic?  If they are on the same subnet then they are not seperate so now my management and storage get to fight each other?  Am I not understanding this?

[jrmunday]

Management will be on a separate subnet, but the IP address assigned to your VM kernel port group used for IP storage will be on the same subnet as your storage. We use this configuration successfully for all our NFS storage, an have even disabled the gateway on this VLAN to ensure that NFS traffic is not routable to other parts of the network. The IP storage VLAN is basically a layer 2 subnet.

[a_p_]

I think you misunderstood what I said. As also mentioned by jrmunday it's the VMkernel port for the storage traffic which has to be on the same subnet as the storage system. The VMkernel port for Management will be on another - usually routed - subnet.

André

[gssieg]

Yea, I guess I am not following.  I'm sorry I'm sure this is something stupid but it's just not sinking in.

So VMkernel #1 Management - Vlan 18 - 172.27.18.104 / GW: 172.27.18.1

VMkernel #2 Storage - Vlan 40 - 172.27.40.40

Or are you saying to use another Vlan 18 address on the second VM kernel also?  so VMkernel #2 Storage would be say 172.27.18.x?

VM told me to just configure the second VMKernel and ignore the Gateway but when I do this I cannot ping the address.  I tried setting the Vlan tag to 40 and everything with no luck.  If someone could give me an example it would be greatly appreciated I'm sure it will be one of those "duh" moments when I finally realize whats going on.

Thank you for your patience on this one guys,

Greg

[Josh26]

gssieg wrote:

Vlan 40 (172.27.40.x/24 w/gw 172.27.40.1) - data/SAN

Any help would be greatly appreciated.

If your SAN is on this range/VLAN, and you have a vmkernel on this range/VLAN, then those two items will communicate without ever using a gateway.

The only time a gateway will likely be involved, is when you want to access your management interface from a different VLAN.

This is precisely the reason why removing multiple gateway support was a very good thing - it doesn't do what people seemed to think it did.

[a_p_]

Please post a screen shot of the ESXi host's current virtual network setup. I assume the physical switch ports are configured as trunk/tagged ports (dot1q)!?

André

[gssieg]

Andre,

I included 2 screen shots.  One that shows my Access Port config, Core vlan Config for vlan 40 and then the vmkernel for my storage.  The other shows my virtual network view for the host.  Thank you for your time.

[depping]

How are the ports configured to which the "storage vmkernel nic" is connected? Are you sure they are also configured as VLAN trunks? If they are not then there is no need to provide a VLAN ID. If you stay in the same network then Support is right there should be no need to specify a VLAN.

[jrmunday · Accepted solution]

Hi Greg,

Here is an example configuration using VSS;

- vSwitch0 = Management is on VLAN130 (access port, no VLAN ID configured)

- vSwitch1 = vMotion is on VLAN131 (access port, no VLAN ID configured)

- vSwitch2 = IPStorage is on VLAN132 (access port, no VLAN ID configured) - L2 subnet with gateway disabled

- vSwitch3 = Guest Networking (trunk ports, VLAN tagging)

In this example, Management and IPStorage are segregated on different VLAN's and for additional security the IPStorage VLAN has the gateway disabled (so traffic can't be routed elsewhere).

So from the storage perspective, simply presented your storage (NFS exports as an example) on the same VLAN as your IPStorage VM Kernel port (or VLAN132 in this example).

Cheers,

Jon

[gssieg]

OMG, Guys I had my "Ahh Ha" moment this morning and boy oh boy do I feel like an idiot.  So this whole time I've been sitting here going I can't ping 172.27.40.110 (Storage Vmkernel) well Im on the 172.27.17.x Vlan and without the gateway I can't ping it... I don't even know what I was thinking... I've been on Dayquil for the last few days but still this was a big "Duh" moment.  So I added a device onto the .40 vlan and Im good to go kinda!

A couple things that are confusing me still but I think these fall back onto my Netapp SAN vs VMware.  My netapp SAN is setup on my vlan 40 which can interroute so maybe that is why but before I setup a vmkernel for my 40 storage vlan I was able to connect to my NFS share with my default vmkernel which is on Vlan 18.  So how do I make sure that my SAN uses this new VMkernel I created and doesn't try to just use the first one?

[jrmunday]

Hurrah - It's awesome when the penny finally drops and things make sense (well for me anyway)!

I suspect you NFS exports are not locked down so can be accessesd from any subnet (in your case both the 18 and 40 VLAN's)? Check this and restrict them to the IPStorage VM Kernel port IP addresses on VLAN 40 (or the entire 172.27.40.x subnet if you like, but it's more secure using the specific IP's).

[gssieg]

Jon,

Excellent point yes in testing I allowed the 18 vlan to connect.  I just removed that only allow my vmkernel port to connect and tested it successfully.  I have one more question for you Jon just really logisitics here on how you setup your network there. You have your Mgmt and your storage setup as Active/Passive but you have your vmotion and standard networks setup as active/active.  Management I can understand you are just adding the redundantcy there isn't a huge need for bandwidth but wouldn't you want your storage to load balance active/active?

Jon/Andre and everyone else on this thank you for being patient with me on this matter.  You guys have been a tremendous help!  Hopefully one day I will be able to be on the other side of these forums too!

[jrmunday]

For IPstorage, we currently only have 1GB uplinks and can't use link aggregation to increase the aggregate bandwith for our NFS traffic we chose this configuration so that we could favour paths (ISL) to NetApp Filers (we use FAS3240 in a MetroCluster configuration).

On the vSwitch used for vMotion, we have configured an additional VM Kernel port (requires another IP address) and have set the two VM Kernel Port groups to override the vSwitch failover order and use an Active/Passive (but with inverse vmnic selections). This ensures that we use all the uplinks and still have redundancy. It also allows us to double the vMotion speed and half the time for tasks such as putting a host into maintenance mode.

Configured as follows;

- vSwitch1 with 2x uplinks (vmnic4 and vmnic8) - Active/Active

- First VM Kernel port group, overrides switch failover order and uses vmnic4 and vmnic8 in a Active / Standby configuration

- Second VM Kernel port group, overrides switch failover order and uses vmnic8 and vmnic4 in a Active / Standby configuration

This use of both uplinks applies even when doing a vMotion for a single VM. See sample graphs below for a single vMotion operation.

Source host (this host use vmnic5 and vmnic7 - but the principal is the same);

Destination Host;

[jrmunday]

This doesn't seem to be showing im my previous message, so have added it again here ... +1 point

... This use of both uplinks applies even when doing a vMotion for a  single VM. See sample graphs below for a single vMotion operation.

Source host (this host use vmnic5 and vmnic7 - but the principal is the same);

Destination Host;

[KFM]

gssieg wrote:

So I've learned this week that vSphere 5.1 VmKernels no longer support multiple gateways. 

Hi, sorry to hijack this thread but when you said the above (emphasis mine) it got my curiousity piqued.  Can you tell me where you read this?  It's not that I disbelieve you (the opposite really) but I need it to verify some of my findings at a client.

I have found the following peculiarity between versions 5.0 and 5.1u1.  Notably:

• In version 5.0 I was able to ping any non-management VMkernel interface from any external network.

For example:

• VMkernel interface IP address 1.1.1.1 (management traffic enabled, default gateway is configured with 1.1.1.254)
• VMkernel interface IP address 2.2.2.2 (For iSCSI.  Note that there is a default gateway that exists on an upstream switch so we can route out if required [for management purposes only however].)
• PC IP address 3.3.3.3
• Ping would succeed from 3.3.3.3 to 1.1.1.1
• Ping would succeed from 3.3.3.3 to 2.2.2.2

• In version 5.1u1 I am now unable to ping from 3.3.3.3 to 2.2.2.2.  Ping would still work from 3.3.3.3 to 1.1.1.1.

It looks like your statement that in 5.1 multiple gateways, or as I'd rather put it static routes, are no longer supported verifies the behaviour I am seeing.  Put another way, it looks like only the the vmkernel interface that is on the same subnet as the configured default gateway is able to be routed.  Not that it would matter normally as you wouldn't want to be routing IP based (NFS/iSCSI) storage traffic anyway.

I just need to know if this is expected behaviour now in VMware ESXi 5.1 Update 1.

Cheers,

KFM