raw device for voting disk RHEL5

Posting to the oracle group, b/c all the googling points to only
Oracle folks using raw devices

This is pretty much the problem, can't write to raw device.

BTW, this is RHEL5, and there is no more 'chkconfig restart
rawservice" or whatever it used to be. Need raw for Std Edition RAC
10gr2.

[root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
/dev/raw/raw10: bound to major 8, minor 129
[root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
dd: opening `/dev/raw/raw10': Device or resource busy
[root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
20+0 records in
20+0 records out
10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s

permissions (played with)
[root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10


........
We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
remove NSPAM to email

On Nov 14, 1:07 am, NetComrade
wrote:
> Posting to the oracle group, b/c all the googling points to only
> Oracle folks using raw devices
>
> This is pretty much the problem, can't write to raw device.
> BTW, this is RHEL5, and there is no more 'chkconfig restart
> rawservice" or whatever it used to be. Need raw for Std Edition RAC
> 10gr2.
Or not - see below
> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
> /dev/raw/raw10: bound to major 8, minor 129
> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
> dd: opening `/dev/raw/raw10': Device or resource busy
> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
> 20+0 records in
> 20+0 records out
> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
> permissions (played with)
> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
> .......
> We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)

Debateable. You DID read the docs about ownership of the slices in
question right? On Solaris it is documented that the raw slices should
be owned by
oracleinstall so I assume that's the same for Lie-GN-ux. : >
FYI I saw no improvement over UFS cooked on Solaris 10 during several
swingbench runs, but ASM definitely was better. Consider ASM usage
instead??

On 14 Nov., 10:07, NetComrade
wrote:
>
> BTW, this is RHEL5, and there is no more 'chkconfig restart
> rawservice" or whatever it used to be. Need raw for Std Edition RAC
> 10gr2.

you don't need raw (char) devices anymore.
the raw interface is deprecated in RHEL4/5. just use the block
devices.
Oracle 10gR2 will open the devices with O_DIRECT bypassing the OS
cache buffer.

> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
> /dev/raw/raw10: bound to major 8, minor 129
> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
> dd: opening `/dev/raw/raw10': Device or resource busy
> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
> 20+0 records in
> 20+0 records out
> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
>
> permissions (played with)
> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10

on my laptop running RHEL5 it works.

maybe you've problem with udev. check it with udevmonitor, you should
see events like this:

UEVENT[1195066033.126148] add@/class/raw/raw10
UDEV [1195066033.126148] add@/class/raw/raw10
UEVENT[1195066042.355995] remove@/class/raw/raw10
UDEV [1195066042.355995] remove@/class/raw/raw10


regards,
-ap

On Wed, 14 Nov 2007 09:02:41 -0800, gerryt
wrote:

>On Nov 14, 1:07 am, NetComrade
>wrote:
>> Posting to the oracle group, b/c all the googling points to only
>> Oracle folks using raw devices
>>
>> This is pretty much the problem, can't write to raw device.
>> BTW, this is RHEL5, and there is no more 'chkconfig restart
>> rawservice" or whatever it used to be. Need raw for Std Edition RAC
>> 10gr2.
>Or not - see below
>> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
>> /dev/raw/raw10: bound to major 8, minor 129
>> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
>> dd: opening `/dev/raw/raw10': Device or resource busy
>> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
>> 20+0 records in
>> 20+0 records out
>> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
>> permissions (played with)
>> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
>> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
>> .......
>> We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
>
>Debateable. You DID read the docs about ownership of the slices in
>question right? On Solaris it is documented that the raw slices should
>be owned by
>oracleinstall so I assume that's the same for Lie-GN-ux. : >

You DO see that that i've been running all commands as root?
I think we figured it out by binding raw to /dev/mapper/mpathX
devices.

>FYI I saw no improvement over UFS cooked on Solaris 10 during several
>swingbench runs, but ASM definitely was better. Consider ASM usage
>instead??

We have'nt gotten to ASM yet, need voting and OCR disk, which need to
be raw
http://download.oracle.com/docs/cd/B...s.htm#CJAHAGJE
........
We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
remove NSPAM to email

On Wed, 14 Nov 2007 10:49:42 -0800, Andreas Piesk
wrote:

>On 14 Nov., 10:07, NetComrade
>wrote:
>>
>> BTW, this is RHEL5, and there is no more 'chkconfig restart
>> rawservice" or whatever it used to be. Need raw for Std Edition RAC
>> 10gr2.
>
>you don't need raw (char) devices anymore.
>the raw interface is deprecated in RHEL4/5. just use the block
>devices.
>Oracle 10gR2 will open the devices with O_DIRECT bypassing the OS
>cache buffer.

Apparently not
http://download.oracle.com/docs/cd/B...s.htm#CJAHAGJE

>> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
>> /dev/raw/raw10: bound to major 8, minor 129
>> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
>> dd: opening `/dev/raw/raw10': Device or resource busy
>> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
>> 20+0 records in
>> 20+0 records out
>> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
>>
>> permissions (played with)
>> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
>> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
>
>on my laptop running RHEL5 it works.
>
>maybe you've problem with udev. check it with udevmonitor, you should
>see events like this:
>
>UEVENT[1195066033.126148] add@/class/raw/raw10
>UDEV [1195066033.126148] add@/class/raw/raw10
>UEVENT[1195066042.355995] remove@/class/raw/raw10
>UDEV [1195066042.355995] remove@/class/raw/raw10

Thanks for the udevmonitor tip.. I do remember dealing with it when I
tried to get Xen working.


........
We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
remove NSPAM to email

On Nov 14, 11:58 am, NetComrade
wrote:
> On Wed, 14 Nov 2007 09:02:41 -0800, gerryt
> wrote:
>
>
>
> >On Nov 14, 1:07 am, NetComrade
> >wrote:
> >> Posting to the oracle group, b/c all the googling points to only
> >> Oracle folks using raw devices
>
> >> This is pretty much the problem, can't write to raw device.
> >> BTW, this is RHEL5, and there is no more 'chkconfig restart
> >> rawservice" or whatever it used to be. Need raw for Std Edition RAC
> >> 10gr2.
> >Or not - see below
> >> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
> >> /dev/raw/raw10: bound to major 8, minor 129
> >> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
> >> dd: opening `/dev/raw/raw10': Device or resource busy
> >> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
> >> 20+0 records in
> >> 20+0 records out
> >> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
> >> permissions (played with)
> >> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
> >> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
> >> .......
> >> We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
>
> >Debateable. You DID read the docs about ownership of the slices in
> >question right? On Solaris it is documented that the raw slices should
> >be owned by
> >oracleinstall so I assume that's the same for Lie-GN-ux. : >
> You DO see that that i've been running all commands as root?
> I think we figured it out by binding raw to /dev/mapper/mpathX
> devices.

Wow is that what you have to do?
Well, what I saw was:
# ls -ltrh /dev/raw/raw10
crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
That indicated to me that "it" was not owned by oracleinstall.
Maybe its different for PenguinOS, but the "c" also indicates
a cooked device vs block O well.

> >FYI I saw no improvement over UFS cooked on Solaris 10 during several
> >swingbench runs, but ASM definitely was better. Consider ASM usage
> >instead??
> We have'nt gotten to ASM yet, need voting and OCR disk, which need to
> be rawhttp://download.oracle.com/docs/cd/B19306_01/license.102/b14199/optio...

Hmm truncated on Google
OK - but you dont have to mount everything under ASM either.. : >

On Nov 14, 11:58 am, NetComrade
wrote:
> On Wed, 14 Nov 2007 09:02:41 -0800, gerryt
> wrote:
>
>
>
> >On Nov 14, 1:07 am, NetComrade
> >wrote:
> >> Posting to the oracle group, b/c all the googling points to only
> >> Oracle folks using raw devices
>
> >> This is pretty much the problem, can't write to raw device.
> >> BTW, this is RHEL5, and there is no more 'chkconfig restart
> >> rawservice" or whatever it used to be. Need raw for Std Edition RAC
> >> 10gr2.
> >Or not - see below
> >> [root-at-lsbcmdbp01 ~]# /bin/raw /dev/raw/raw10 /dev/sdi1
> >> /dev/raw/raw10: bound to major 8, minor 129
> >> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/raw/raw10 bs=512 count=20
> >> dd: opening `/dev/raw/raw10': Device or resource busy
> >> [root-at-lsbcmdbp01 ~]# dd if=/dev/zero of=/dev/sdi1 bs=512 count=20
> >> 20+0 records in
> >> 20+0 records out
> >> 10240 bytes (10 kB) copied, 0.00076 seconds, 13.5 MB/s
> >> permissions (played with)
> >> [root-at-lsbcmdbp01 ~]# ls -ltrh /dev/raw/raw10
> >> crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
> >> .......
> >> We run Oracle 9iR2,10gR1/2 on RH4/RH3 and Solaris 10 (Sparc)
>
> >Debateable. You DID read the docs about ownership of the slices in
> >question right? On Solaris it is documented that the raw slices should
> >be owned by
> >oracleinstall so I assume that's the same for Lie-GN-ux. : >
> You DO see that that i've been running all commands as root?
> I think we figured it out by binding raw to /dev/mapper/mpathX
> devices.

Wow is that what you have to do?
Well, what I saw was:
# ls -ltrh /dev/raw/raw10
crw-rw-rw- 1 root root 162, 10 Nov 14 03:31 /dev/raw/raw10
That indicated to me that "it" was not owned by oracleinstall.
Maybe its different for PenguinOS, but the "c" also indicates
a cooked device vs block O well.

> >FYI I saw no improvement over UFS cooked on Solaris 10 during several
> >swingbench runs, but ASM definitely was better. Consider ASM usage
> >instead??
> We have'nt gotten to ASM yet, need voting and OCR disk, which need to
> be rawhttp://download.oracle.com/docs/cd/B19306_01/license.102/b14199/optio...

Hmm truncated on Google
OK - but you dont have to mount everything under ASM either.. : >

>
> >you don't need raw (char) devices anymore.
> >the raw interface is deprecated in RHEL4/5. just use the block
> >devices.
> >Oracle 10gR2 will open the devices with O_DIRECT bypassing the OS
> >cache buffer.
>
> Apparently nothttp://download.oracle.com/docs/cd/B19306_01/license.102/b14199/optio...

did you read metalink note 220970.1 (RAC FAQ)? there's a chapter
explaining the different storage options.
according to this note you can use block devices with 10gR2 and i can
confirm that O_DIRECT is used by CRS and RDBMS in 10..2.0.3.

regards,
-ap

On 14 Nov., 23:32, Andreas Piesk wrote:
> > >you don't need raw (char) devices anymore.
> > >the raw interface is deprecated in RHEL4/5. just use the block
> > >devices.
> > >Oracle 10gR2 will open the devices with O_DIRECT bypassing the OS
> > >cache buffer.
>
> > Apparently nothttp://download.oracle.com/docs/cd/B19306_01/license.102/b14199/optio...
>
> did you read metalink note 220970.1 (RAC FAQ)? there's a chapter
> explaining the different storage options.
> according to this note you can use block devices with 10gR2 and i can
> confirm that O_DIRECT is used by CRS and RDBMS in 10..2.0.3.
>

found another note in my bookmark list which could be helpful:

Note:401132.1 How to install Oracle Clusterware with shared storage on
block devices

regards,
-ap

On Wed, 14 Nov 2007 09:02:41 -0800, gerryt wrote:

> but ASM definitely was better.

Why do you think so? Usual file manipulation programs like tar,cp and mv
are not available from ASM. ASM is proprietary thing, rather complex,
which locks you in with backup solutions (RMAN only) and makes simple
things like writing crontab "cleanup" scripts rather difficult. If
possible, I would opt for HP's Polyserve or VCS, rather then ASM. ASM is
free, thus forced down my gullet, but I don't like it at all. GFS is
also a good alternative. Matt Zito's Gridapp has a great deal of
experience with Red Hat's GFS. The only thing that ASM has going for it
is its price. It's worth every penny you pay for it. So is Polyserve.

> Consider ASM usage instead??

Unfortunately so. I would try avoiding it by any means, except by using
OCFS. OCFS is badly maintained, has a ton of bugs and is rather unsafe
for use. So, if you can't get a decent commercial clustering filesystem
from your management, ASM is a thing to try. The bucks thus saved will
probably be spent on configuration and work.


--
Mladen Gogala