Given at the moment you have little control over how the EC2 kernel is booted and I wasn’t able to get the rPath image for Xen to boot… I am stuck.

The Openfiler image I built from the tarball is fine, however a bunch of services aren’t working and modules are missing. Given the instance is running its own flavour of linux, it was missing yum or up2date and even rpm. So getting the missing modules in place was a pain.

The next thing which was broken the iscsi-target, without this module I will be unable to present a block device to OCFS2 to partition.

So I can get Openfiler to create logical volume and the NFS service is running, but this was not the whole point of this exercise.

I tried to get iscsi-target to compile and had a bunch of issues with running make.

The other issues is that the /dev/hda2 is mounted when the instance is started, not as a block device as suggested in some of the Openfiler Xen wiki and other articles.

At this point, I am going to move on, I think issue is more the EC2 environment rather than any problems with Openfiler itself. This is clearly better done where you have some control over how Xen boots an instance and the disks it presents.

So I am going to move on with the Oracle RAC on EC2 and attempt Oracle RAC over NFS.

There has been some interest on getting Openfiler working so this is disappointing. Various Amazon forum posts have mentioned several alternatives, however the inability to present a block device may well end up being the biggest issue with EC2 at least for any cluster filesystem requiring that.

Have Fun

Paul

As I mentioned in this post, I was having trouble getting OpenFiler to use the physical volumes or volume groups I had created from the unix command line.

I finally had time to give this another go and was stuck again until I decided to read the error.log produced by lighthttpd and also the /var/log/messages file.

/var/log/messages was saying this

Jul  9 12:00:02 domU-12-31-35-00-0A-41 crond(pam_unix)[1936]:
  session closed for user openfiler
Jul  9 12:00:34 domU-12-31-35-00-0A-41 modprobe:
  FATAL: Could not load /lib/modules/2.6.16-xenU/modules.dep:
  No such file or directory

I had tried to use the Xen ready image from Rpath however the image had failed to boot under EC2. So I used the tarball instead and made the image from extracted tarball.

I noticed the message log was complaining about a missing module in a directory which didn’t exist either. That module was dev-mapper. No dev-mapper and lvcreate fails!!

So the Openfiler web admin tool was going through the motions of creating new volumes but failing with any indications in the tool as to why.

I rechecked the Openfiler tarball again and it doesn’t have dev-mapper in a version of Xen which EC2 can use given it boots Xen 2.6.16. So the clunky solution was to copy the files from another running AMI instance running CentOS 4.
After running mod-prod dm-mod, lvcreate worked and so did the OpenFiler admin tool.

Woot! Next stop getting iscsi running so I can get OCFS2 formatting the storage provided by Openfiler as a cluster block device for ASM to use.

Stay tuned…

Paul

Here is a dump of my screen showing how pvcreate, vgcreate and lvcreate should work.

[root@domU-12-31-36-00-31-73 ~]# umount /mnt
[root@domU-12-31-36-00-31-73 ~]# pvcreate /dev/sda2
Physical volume "/dev/sda2" successfully created
[root@domU-12-31-36-00-31-73 ~]# vgcreate /dev/sda2 vg
/dev/sda2: already exists in filesystem
New volume group name "sda2" is invalid
[root@domU-12-31-36-00-31-73 ~]# vgcreate vg /dev/sda2
Volume group "vg" successfully created
[root@domU-12-31-36-00-31-73 ~]# lvcreate -L4096M -n myvmdisk1 vg
Logical volume "myvmdisk1" created
[root@domU-12-31-36-00-31-73 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  937M  8.5G  10% /
[root@domU-12-31-36-00-31-73 ~]# mount /dev/vg/myvmdisk1 /oradata
mount: you must specify the filesystem type
[root@domU-12-31-36-00-31-73 ~]# mkfs -t ext3 /dev/vg/myvmdisk1
mke2fs 1.35 (28-Feb-2004)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
524288 inodes, 1048576 blocks
52428 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
      32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@domU-12-31-36-00-31-73 ~]# mount /dev/vg/myvmdisk1 /oradata
mount: mount point /oradata does not exist
[root@domU-12-31-36-00-31-73 ~]# mkdir /oradata
[root@domU-12-31-36-00-31-73 ~]# mount /dev/vg/myvmdisk1 /oradata
[root@domU-12-31-36-00-31-73 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1             9.9G  937M  8.5G  10% /
/dev/mapper/vg-myvmdisk1
                    4.0G   41M  3.7G   2% /oradata

I was working on the Openfiler step over the past couple of days.
Whilst I was able to get the Openfiler image working, I was unable to create any logical volumes or get it to see any physical volumes. It was looking like the proverbial business buzzword “it’s a Show Stopper”.

Hint: whilst pvcreate and vgcreate worked, they were unusable by Openfiler.

I googled and came up with very little apart from needing to learn more about Xen and vxd block devices.
I gave it another go (just before starting this post) and bingo, I found a bunch of potential resolution articles.
The Openfiler on Xen wiki (RFTM whoops)
A rpath forum post on handling xvd block devices.
Plus it looks like someone was doing something similar with a HA cluster setup whilst not as crazy as Openfiler on Xen flavoured virtual instance sitting on top of Amazon hardware ala EC2.

I have a week of leave to play with this and get Oracle RAC on EC2 a reality… in amongst some kids parties and a Burt Bacharach concert.

If you have any ideas or thoughts on Openfiler on Xen, feel free to contribute.

Editor note: Slight change pvcreate and vgcreate were working, lvcreate never was.

Update: Success. See the sequel to this article.

Have Fun

Paul