Written by Bipul Kumar

is NVMe storage in Oracle Database Cloud Service ephemeral?

Oracle Cloud Database Service offers an excellent option for databases requiring extremely high disk IO performance using locally attached NVMe storage in HighIO and DenseIO compute shapes. However, there has been some discussion / doubts around using NVMe storage for databases. The cause of concern was persistence of data on NVMe devices across reboot of underlying hosts.
In this blog, I will explore:
  • ASM diskgroups configuration in DBCS on OCI using DenseIO.
  • is this Storage ephemeral  i.e. does the storage come back online following reboot of database host ?
  • and recommended additional protection if using NVMe storage for database?
ASM diskgroups configuration in DBCS on OCI using DenseIO
On the Database Service creation page, I chose the following attributes which will influence how the ASM Diskgroups will be setup for my database system.

Shape – BM.DenseIO1.36
Data Storage Percentage – 80% (default)
Disk Redundancy – NORMAL

After the database system has been created, each NMVe disks are partitioned in 80/20 as per my Data Storage percentage requirement as shown below in the lsblk output.

[oracle@db01 sql]$ lsblk
>AME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 372.6G 0 disk
|-sda1 8:1 0 486M 0 part /boot/efi
|-sda2 8:2 0 1.4G 0 part /boot
|-sda3 8:3 0 326G 0 part
|-VolGroupSys-LogVolRoot (dm-0) 251:0 0 150G 0 lvm /
|-VolGroupSys-LogVolSwap (dm-1) 251:1 0 20G 0 lvm [SWAP]
|-VolGroupSys-LogVolU01 (dm-2) 251:2 0 130G 0 lvm /u01
asm!commonstore-68 249:34817 0 5G 0 disk
nvme0n1 259:3 0 2.9T 0 disk
|-nvme0n1p1 259:21 0 2.3T 0 part
|-nvme0n1p2 259:25 0 596.2G 0 part
|-nvme0n1p1 (dm-3) 251:3 0 2.3T 0 part <— Used in DATA Diskgroup
|-nvme0n1p2 (dm-4) 251:4 0 596.2G 0 part <— Used in RECO Diskgroup
nvme1n1 259:0 0 2.9T 0 disk
|-nvme1n1p1 259:9 0 2.3T 0 part
|-nvme1n1p2 259:12 0 596.2G 0 part
|-nvme1n1p1 (dm-5) 251:5 0 2.3T 0 part
|-nvme1n1p2 (dm-6) 251:6 0 596.2G 0 part
… output truncated …

And two ASM diskgroups are created as shown below:

ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 512 4096 4194304 21976992 21954928 2441888 9756520 0 Y DATA/
MOUNTED NORMAL N 512 512 4096 4194304 5494212 5482124 610468 2435828 0 N RECO/
ASMCMD>

i.e. NORMAL disk redundancy for both disk groups. Data Diskgroup has 9.7 TB of usable storage and similarly RECO has 2.4 TB of usable storage.
I created a PDB, a schema, few objects and populated with some sample data.

SQL> sho con_name
CON_NAME
——————————
PDB01
SQL> select object_name from user_objects;
OBJECT_NAME
——————————————————————————–
MERCHANTS
SYS_C007432
ISEQ$$_73308
BUYER
SYS_C007434
ISEQ$$_73311
TRANSACTION_NO_COMP
LOAD_NO_COMP
8 rows selected.
SQL> select bytes/1024/1024 from user_segments where segment_name = ‘TRANSACTION_NO_COMP’;
BYTES/1024/1024
—————
1024
SQL>

Is this Storage ephemeral  i.e. does the storage come back online following reboot of database host ?

[oracle@db01 ~]$ date
Thu Nov 23 15:00:26 UTC 2017
[oracle@db01 ~]$ uptime
15:00:30 up 22:40, 3 users, load average: 0.29, 0.37, 0.58
[oracle@db01 ~]$

Now I will reboot the database host. I am expecting the ASM diskgroups to come back online after the reboot and all my data intact.
After reboot —

[opc@db01 ~]$ date
Thu Nov 23 15:10:12 UTC 2017
[opc@db01 ~]$ uptime
15:10:14 up 1 min, 1 user, load average: 1.73, 0.60, 0.21
[opc@db01 ~]$

ASM command lsdg output confirms that I still have the ASM diskgroups.

ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 512 4096 4194304 21976992 21954928 2441888 9756520 0 Y DATA/
MOUNTED NORMAL N 512 512 4096 4194304 5494212 5482124 610468 2435828 0 N RECO/
ASMCMD>

And the following SQL statements confirm that my data is intact.

SQL> select bytes/1024/1024 from user_segments where segment_name = ‘TRANSACTION_NO_COMP’;
BYTES/1024/1024
—————
1024
SQL> select count(*) from TRANSACTION_NO_COMP;
COUNT(*)
———-
4010000
SQL>

In short,  NVMe storage in Oracle Cloud database service are NOT ephemeral and will NOT disappear across host reboot.
However, it is is LOCALLY attached. So if the host (for whatever reason) terminates and cannot be brought online, then the data on NVMe storage will not be accessible. Backup of database or data guard instance will provide protection in this situation.