Oracle's ASMLib: Can We Just Be Friends?

Anyone who has used Oracle ASM on Linux in the last 5 – 10 years is most likely familiar with ASMLib. Oracle provides this software and associated kernel module in order to easily manage disks that are intended to be used as ASM disks for database storage. When it was first introduced there was no reliable way to persistently name disks and assign permissions easily on Linux. An external disk might be /dev/sdc one time, and /dev/sdg the next reboot. What’s an admin to do? ASMLib allowed us to label the disks so that no matter what device name they might have, we can still refer to them within ASM and know which disk we’re talking to, as well as handling the permissions needed. However, as the years have moved on the udev system has appeared and it seems like it does pretty much everything ASMLib can do.


Is it time to consider a change?




Why even consider changing? Well I first found myself in a bind when I was setting up a three node Oracle 11gR2 RAC within a VMWare vSphere hypervisor. Since this is just a POC setup I chose to go with the common strategy of using CentOS Linux to stand in for Red Hat Enterprise Linux 6.4. Setup was going swimmingly until I went to get the asmlib kernel module for this setup. Turns out that Oracle stopped providing this module after RHEL/Centos 5. It makes perfect sense, but it leaves me without a kernel module.

I remembered the udev system is capable of all sorts of manipulations to devices and quickly found this great documentation on fundamental rule writing. With a little help from random sources on the Internet (linked below) I quickly had a set of disks in place that I could go ahead and install Oracle with.

Hard drives can be sliced and diced and presented to the host several different ways. Let’s look at how udev can help us in different situations.

The Easy Way

The simplest setup is you have a single hard drive attached to your host that you want to use for ASM disks. Say you have an internal SCSI drive for the OS/software install, and an external drive that always appears as /dev/sdb. Using fdisk or gparted you can create several partitions on this drive. In my example sdb1, sdb2 and sdb3 were created as 2 GB partitions meant for the OCR and Vote diskgroup required when Grid Infrastructure is installed. sdb5 and sdb6 are both 50GB and destined for a diskgroup named +DATA.

In order for Oracle to make use of these disks they need to be owned by the oracle user and group with read/write permissions for owner and group. To make this happen change directories to /etc/udev/rules.d on your host. If you list the directory you will probably have  a few files named things like 70-persistent-net.rules and the like. Any filename that ends in .rules will be evaluated, lower numbered rule files will be evaluated before higher numbered ones. Let’s create one called 99-oracle.rules to put our rules in so that they run near the end of processing.

Let’s look at some rules we can put into that file to do some disk magic:

KERNEL=="sdb1", NAME+="oracleasm/ocrvote1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdb2", NAME+="oracleasm/ocrvote2", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdb3", NAME+="oracleasm/ocrvote3", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdb5", NAME+="oracleasm/data1", OWNER="oracle", GROUP="dba", MODE="0660"
KERNEL=="sdb6", NAME+="oracleasm/data2", OWNER="oracle", GROUP="dba", MODE="0660"

Each rule is a comma separated list of conditions that have to match, followed by properties that need to be changed/added to the device in question. So the first rule above says:

IF
THEN

The following rules are exactly the same, except matching different device names and creating new devices in their place. When the host is rebooted we now see a new set of devices under /dev/oracleasm:

# ll /dev/oracleasm
total 0brw-rw----. 1 oracle dba 8, 21 Dec 18 13:57 data1brw-rw----. 1 oracle dba 8, 22 Dec 18 13:57 data2brw-rw----. 1 oracle dba 8, 17 Dec 18 12:29 ocrvote1brw-rw----. 1 oracle dba 8, 18 Dec 18 13:57 ocrvote2brw-rw----. 1 oracle dba 8, 19 Dec 18 13:57 ocrvote3

When running the Oracle Universal Installer you can simply change your disk string to /dev/oracleasm/* and these disks are made available to use with ASM. 

A Slightly Less Trivial Case

Ok, so in the real world we tend to have disk arrays that have hundreds of spindles and LUNs are carved out of them and presented to the host. Instead of one big disk, we may get 20 smaller disks instead. To simulate this example I have a VM with three 5GB luns and two 50GB luns. The small LUNs are for OCR/VOTE again, and the larger ones are for the +DATA diskgroup. When initially presented to the host I just have a bunch of scsi devices:

# ll /dev/sd*
brw-rw---- 1 root disk 8, 0 Dec 18 14:03 /dev/sda
brw-rw---- 1 root disk 8, 1 Dec 18 14:03 /dev/sda1
brw-rw---- 1 root disk 8, 2 Dec 18 14:03 /dev/sda2
brw-rw---- 1 root disk 8, 16 Dec 18 14:03 /dev/sdb
brw-rw---- 1 root disk 8, 32 Dec 18 14:03 /dev/sdc
brw-rw---- 1 root disk 8, 48 Dec 18 14:03 /dev/sdd
brw-rw---- 1 root disk 8, 64 Dec 18 14:03 /dev/sde
brw-rw---- 1 root disk 8, 80 Dec 18 14:03 /dev/sdf

/dev/sda is the Linux O/S, the rest are LUNs meant to be used for ASM. How can udev help
us here? Let’s look at the following rules, again placed in /etc/udev/rules.d/99-oracle.rules.

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36000c290e2feb0a79b4ebe855688ae7c", NAME+="oracleasm/ocrvote1", OWNER="oracle", GROUP="dba", MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36000c297cae9913d395a112e3205cdc8", NAME+="oracleasm/ocrvote2", OWNER="oracle", GROUP="dba", MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36000c296772099ed8a7387461a1e99d3", NAME+="oracleasm/ocrvote3", OWNER="oracle", GROUP="dba", MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36000c294f6839516720e007185a3f617", NAME+="oracleasm/data1", OWNER="oracle", GROUP="dba", MODE="0660"

KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk", ENV{ID_SERIAL}=="36000c29a75f655d777930b9abd29ead0", NAME+="oracleasm/data2", OWNER="oracle", GROUP="dba", MODE="0660"

What is all that gibberish? Let’s look at the rules again. Remember, commas separate matches followed by actions. So for the first line:

IF
THEN
After a reboot and looking at /dev/oracleasm we see:

# ll /dev/oracleasm
total 0
brw-rw----. 1 oracle dba 8, 64 Dec 18 14:09 data1
brw-rw----. 1 oracle dba 8, 80 Dec 18 14:09 data2
brw-rw----. 1 oracle dba 8, 16 Dec 18 14:09 ocrvote1
brw-rw----. 1 oracle dba 8, 32 Dec 18 14:09 ocrvote2
brw-rw----. 1 oracle dba 8, 48 Dec 18 14:09 ocrvote3


How did we get the serial number from the scsi devices? Our friend scsi_id comes to the rescue. As root run the following command:

# scsi_id -g -u -d /dev/sdb
36000c290e2feb0a79b4ebe855688ae7c

There's your serial number. If you're running in a VM that doesn't enable disk UUID's by default (I'm looking at you VMWare) then you need to add the following entry to the VM's .vmx configuration file:

disk.EnableUUID = true

and serial numbers/IDs will become visible.

Enabling direct I/O

Remember also to set the oracle parameter filesystemio_options to ‘setall’ or ‘directio’ in order to enable direct i/o operations. See this My Oracle Support Document for more details: File System's Buffer
Cache versus Direct I/O (Doc ID 462072.1). As that article points out if you have been allocating a huge amount of memory to the operating system and unknowingly counting on it you’re going to be surprised when certain operations slow down. You are moving the cache from the operating system directly to Oracle, so give the database that memory so it can cache blocks instead. 

But! But! But!

I know by now some of you are reading this and saying to yourselves "Well this is nice but ASMLib provides specific performance gains under Linux. I'm not going to use this just because you say so." I know, I thought the same thing too. However, I challenge you to find anything in the official documentation or elsewhere on the internet that says you must use ASMLib to get high performance read/write in Linux. It's true that in the old days the only guarantee of direct I/O with Linux was via RAW devices and other horrible things that most of us would like to forget. Current Versions of Linux and Oracle support direct I/O without using ASMLib as we saw above.

Conclusions

Should you immediately rush out and remove ASMLib from every system you have in place? Of course not. But the next time you're building out a new system, take the time to investigate udev. As many of the linked articles below reiterate - ASMLib is fine, but it's another layer or moving part that you have to keep track of. udev comes with Linux right out of the box and it's very flexible. You don't need to worry about kernel modules when you patch your kernel. Is ASMLib a bad thing? No. But it's getting much harder to implement and if the alternative is just as good and removes some additional package maintenance why not give it a try?

I know it's hard to hear ASMLib, but we need to move on. We'll always be friends, right?

Related Links:

Labels: , ,