Deploying Oracle RAC on VMware vSphere using NVMe-oF/FC with Pure X50 FlashArray , Broadcom LPe36000 Fibre Channel Adapter & VMware vNVMe Controllers (2023)

Business Critical Oracle Workloads have stringent IO requirements and enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s, all the way from on-premises to VMware Hybrid Clouds

Oracle RAC provides high availability and scalability by having multiple instances access a single database which prevents the server from being a single point of failure. Oracle RAC enables you to combine smaller commodity servers into a cluster to create scalable environments that support mission critical business applications.

With increasing need for data to be “fast and furious”, customers are turning to NVMe transfer protocol for accessing data quickly from flash memory storage devices such as solid-state drives (SSDs).

NVMe over Fibre Channel (NVMe/FC) is a technology specification designed to enable NVMe-based message commands to transfer data and status information between a host computer and a target storage subsystem over a Fibre Channel network fabric, providing sustained, predictable performance and catering to low latency requirements of mission critical workloads.

KB 1034165 currently has the guidance “When using the multi-writer mode, the virtual disks must NOT be attached to the virtual NVMe controller.”.

This blog is a pre-validation of the deployment and configuration of Oracle RAC on VMware vSphere using NVMe/FC protocol with Pure X50 FlashArray and Broadcom LPe36000 Fibre Channel Adapter, pending official VMware Engineering validation efforts , currently in progress , into using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol.

The official guidance is that Customers deploy their Oracle RAC using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol , ONLY after KB 1034165 is updated.

Oracle Real Application Clusters (RAC) on VMware vSphere

Oracle Clusterware is portable cluster software that provides comprehensive multi-tiered high availability and resource management for consolidated environments. It supports clustering of independent servers so that they cooperate as a single system.

Oracle Clusterware is the integrated foundation for Oracle Real Application Clusters (RAC), and the high-availability and resource-management framework for all applications on any major platform.

There are two key requirements for Oracle RAC:
• Shared storage
• Multicast Layer 2 networking

These requirements are fully addressed when running Oracle RAC on VMware vSphere, as both shared storage and Layer 2 networking are natively supported.

VMware vSphere HA clusters enable a collection of VMware ESXi hosts to work together so that, as a group, they provide higher infrastructure-level availability for VMs than each ESXi host can provide individually. VMware vSphere HA provides high availability for VMs by pooling the VMs and the hosts on which they reside into a cluster. Hosts in the cluster are monitored and in the event of a failure, the VMs on a failed host are restarted on alternate hosts

Oracle RAC and VMware HA solutions are complementary to each other. Running Oracle RAC on a VMware platform provides the application-level HA enabled by Oracle RAC, in addition to the infrastructure-level HA enabled by VMware vSphere.

Extended Oracle RAC provides greater availability than local Oracle RAC. It provides extremely fast recovery from a site failure and enables all servers, in all sites, to actively process transactions as part of a single database cluster. Running Extended Oracle RAC on a stretched cluster architecture provides the same advantages as traditional Oracle RAC, across data centers and sites, in addition to the site-level protection enabled by stretched cluster architecture.

More information on Oracle RAC on VMware vSphere can be found at Oracle VMware Hybrid Cloud High Availability Guide – REFERENCE ARCHITECTURE

VMware multi-writer attribute for shared vmdk’s

VMFS is a clustered file system that disables (by default) multiple VMs from opening and writing to the same virtual disk (.vmdk file). This prevents more than one VM from inadvertently accessing the same .vmdk file. The multi-writer option allows VMFS-backed disks to be shared and written to by multiple VMs. An Oracle RAC cluster using shared storage is a typical use case.

In the case of VMware vSphere (non vSAN Storage) on VMFS, vVol (beginning with ESXi 6.5) and NFS datastores, using the multi-writer attribute to share the VMDKs for Oracle RAC requires

• SCSI bus sharing needs to be set to none
• VMDKs must be EZT; thick provision lazy zeroed or thin-provisioned formats are not allowed

KB 1034165 currently has the guidance “When using the multi-writer mode, the virtual disks must NOT be attached to the virtual NVMe controller.”.

This blog is a pre-validation of the deployment and configuration of Oracle RAC on VMware vSphere using NVMe/FC protocol with Pure X50 FlashArray and Broadcom LPe36000 Fibre Channel Adapter, pending official VMware Engineering validation efforts , currently in progress , into using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol.

The official guidance is that Customers deploy their Oracle RAC using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol , ONLY after KB 1034165 is updated.

Pure FlashArray X50 with NVMe/FC capability

Pure FlashArray X50 provides native NVMe storage solution for Tier 0 and Tier 1 block storage applications and fully supports NVMe-oF. NVMe-oF delivers faster connectivity between storage and applications on servers, as well as more efficient CPU usage.

More information on Pure FlashArray X50 with NVMe/FC capability can be found here.

Broadcom LPe36000 Fibre Channel Adapter

Broadcom’s NVMe over Fibre Channel (NVMe/FC) solution enables organizations to leverage NVMe-based storage within FC SANs. Broadcom Emulex NVMe over Fibre Channel (NVMe/FC) capable Gen 7 (32G/64G) Host Bus Adapters (HBAs) enable organizations to support both SCSI/FC and NVMe/FC protocols concurrently in the same FC SAN without having to rip and replace any storage infrastructure.

More information on Broadcom Emulex NVMe over Fibre Channel (NVMe/FC) capable Gen 7 Host Bus Adapters (HBAs) can be found here.

Test Use case

This test case showcases the validation of Oracle RAC for this setup below

• Pure X50 FlashArray NVMe-FC
• VMware ESXi, 7.0.3, 19482537 with Broadcom LPe36000 Fibre Channel Adapter
• Oracle RAC 19.15 with Grid Infrastructure, ASM Storage and ASMLIB

Test Bed

The Test bed is a 4 Node vSphere Cluster with 4 ESXi servers, all version ESXi, 7.0.3, 19482537. 2 of the 4 Servers, ‘sc2esx31.vslab.local ‘ & ‘sc2esx32.vslab.local ‘ were configured with Broadcom LPe36000 Fibre Channel Adapter.

All ESXi servers were VMware ESXi, 7.0.3, 19482537, each server with 2 sockets, 48 cores per socket with 768GB RAM. The vCenter version was 7.0.3 build 19480866.

Server ‘sc2esx31.vslab. local’ Broadcom LPe36000 HBA Controller details are shown as below.

Server ‘sc2esx31.vslab. local’ Broadcom LPe36000 HBA device details are shown as below.

Server ‘sc2esx32.vslab. local’ Broadcom LPe36000 HBA Controller details are shown as below.

Server ‘sc2esx32.vslab. local’ Broadcom LPe36000 HBA device details are shown as below.

Pure X50 FlashArray NVMe-FC details are shown as below.

2 hosts , ‘sc2esx31-NVMe-FC’ & ‘sc2esx32-NVMe-FC‘ with ESXI personality are created as shown below with appropriate NQN of ESXi Servers ‘sc2esx31.vslab.local’ and ‘sc2esx32.vslab.local’.

Both hosts ‘sc2esx31-NVMe-FC’ & ‘sc2esx32-NVMe-FC are connected to a 5TB NVMe LUN ‘Pure-Oracle-NVME’ as shown below

More information on Pure Storage NVMe/FC for VMware can be found at How To: Setup NVMe-FC with VMware

Oracle RAC VM’s ‘nrac19c1’ and ‘nrac19c2’ details are shown as below. Each VM has 8 vCPU’s and 64GB vRAM. RAC database ‘nrac19c’ was created with multi-tenant option & provisioned with Oracle Grid Infrastructure (ASM) and Database version 19.15 on O/S OEL 8.6 UEK. Oracle ASM was the storage platform with Oracle ASMLIB for device persistence. Oracle SGA & PGA set to 32G and 6G, respectively. All Oracle on VMware best practices were followed.

The vmdk’s for the VM ‘nrac19c1’ ^& ‘nrac19c2’ are shown as below –

• Hard Disk 1 (SCSI 0:0) – 80G for OS (/)
• Hard Disk 2 (SCSI 0:1) – 80G for Oracle Grid Infrastructure and RDBMS binaries
• Hard Disk 3 (NVME 0:0) – 500G for Oracle RAC (GIMR, CRS, VOTE, DATA) –

Note – We used 1 VMDK for the entire RAC Cluster , as the intention here was to quickly highlight that there is no difference in the steps when we add shared vmdk’s with multi-writer flag to either a PVSCSI or vNVMe controller on a VM.

Recommendation is to follow the RAC deployment Guide on VMware for Best Practices with respect to the RAC layout – Oracle VMware Hybrid Cloud High Availability Guide – REFERENCE ARCHITECTURE

Test Steps

The above section shows the configuration details of the Oracle RAC 19.15 with Grid Infrastructure, ASM Storage and ASMLIB on VMware ESXi, 7.0.3, 19482537 with Broadcom LPe36000 Fibre Channel Adapter using Pure X50 FlashArray NVMe/FC.

The O/S details are shown as below

[root@nrac19c1 ~]# cat /etc/oracle-release
Oracle Linux Server release 8.6
[root@nrac19c1 ~]#

[root@nrac19c1 ~]# uname -a
Linux nrac19c1.vslab.local 5.4.17-2136.308.9.el8uek.x86_64 #2 SMP Mon Jun 13 20:36:40 PDT 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@nrac19c1 ~]#

The Cluster services are up as shown below.

Details of the Oracle Instance, ASM, & MGMTDB are shown as below

Details of Oracle Listeners are shown as below

Details of the Oracle RAC database is shown as below.

Summary

KB 1034165 currently has the guidance “When using the multi-writer mode, the virtual disks must NOT be attached to the virtual NVMe controller.”.

This blog is a pre-validation of the deployment and configuration of Oracle RAC on VMware vSphere using NVMe/FC protocol with Pure X50 FlashArray and Broadcom LPe36000 Fibre Channel Adapter, pending official VMware Engineering validation efforts , currently in progress , into using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol.

The official guidance is that Customers deploy their Oracle RAC using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol , ONLY after KB 1034165 is updated.

This test case showcases the validation of Oracle RAC for this setup below

• Pure X50 FlashArray NVMe-FC
• VMware ESXi, 7.0.3, 19482537 with Broadcom LPe36000 Fibre Channel Adapter
• Oracle RAC 19.15 with Grid Infrastructure, ASM Storage and ASMLIB

Acknowledgements

This blog was authored by Sudhir Balasubramanian, Senior Staff Solution Architect & Global Oracle Lead – VMware.

Many thanks to the following for proving their invaluable support and inputs in this effort

• Jayamohan Kallickal, Distinguished Engineer, Broadcom
• Naveen Krishnamurthy , Senior Product Line Manager (Storage), VMware

We would like to thank the following vendors for lending their infrastructure and support in creating this collateral

• Pure Storage (X50 Flash Array)
• Broadcom (LPe36000 Fibre Channel Adapter)

Conclusion

Business Critical Oracle Workloads have stringent IO requirements and enabling, sustaining, and ensuring the highest possible performance along with continued application availability is a major goal for all mission critical Oracle applications to meet the demanding business SLA’s, all the way from on-premises to VMware Hybrid Clouds

Oracle RAC provides high availability and scalability by having multiple instances access a single database which prevents the server from being a single point of failure. Oracle RAC enables you to combine smaller commodity servers into a cluster to create scalable environments that support mission critical business applications.

With increasing need for data to be “fast and furious”, customers are turning to NVMe transfer protocol for accessing data quickly from flash memory storage devices such as solid-state drives (SSDs).

NVMe over Fibre Channel (NVMe/FC) is a technology specification designed to enable NVMe-based message commands to transfer data and status information between a host computer and a target storage subsystem over a Fibre Channel network fabric, providing sustained, predictable performance and low latency requirements of mission critical workloads.

KB 1034165 currently has the guidance “When using the multi-writer mode, the virtual disks must NOT be attached to the virtual NVMe controller.”.

This blog is a pre-validation of the deployment and configuration of Oracle RAC on VMware vSphere using NVMe/FC protocol with Pure X50 FlashArray and Broadcom LPe36000 Fibre Channel Adapter, pending official VMware Engineering validation efforts , currently in progress , into using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol.

The official guidance is that Customers deploy their Oracle RAC using vNVMe controllers for shared vmdk’s with multi-writer attribute using NVMe-oF/FC protocol , ONLY after KB 1034165 is updated.

All Oracle on VMware vSphere collaterals can be found in the url below

Oracle on VMware Collateral – One Stop Shop
https://blogs.vmware.com/apps/2017/01/oracle-vmware-collateral-one-stop-shop.html