Wednesday, January 29, 2014

Unix Shell Scripting - Line Command

- It is possible to run two or more Unix commands on the same line in a shell script, by separating the commands with ; (semicolon) character

[oracle@rac1 ~]$ echo Todays date is ;date
Todays date is
Thu Jan 30 08:45:08 IST 2014

- For aesthetic reasons if you wish to split the command into multiple lines, we can use single quotes, double quotes or back slash

Using Double Quotes:

[oracle@rac1 ~]$ echo "This is an example
> of command splitting
> across multiple
> lines"
This is an example
of command splitting
across multiple
lines

Using Single Quotes:

[oracle@rac1 ~]$ echo 'This is an example
> of command splitting
> across multiple
> lines'
This is an example
of command splitting
across multiple
lines

Using Backslash:

[oracle@rac1 ~]$ echo This is an example \
> of command splitting \
> across multiple \
> lines
This is an example of command splitting across multiple lines

Notice the difference for splitting the command using backslash, that’s the specialty of Backslash

Unix Shell Scripting – Grouping Commands ()

- Occasionally in Unix Shell scripting we may need to group multiple commands together and treat them as single command

Why that is needed ?  Lets validate the requirements examples for doing so

- We may need to run the command in background (via the & command)

- We may need to redirect the output of the command to a file or other program (or command)  (using > or |)

- If you wish to apply the same action (in the example below output to other file using >) to several consecutive programs(in the example below echo & date programs), it is possible to group them together and apply the action to the group as below

Without Grouping-

[oracle@rac1 ~]$ echo Todays date is > DateOutput
[oracle@rac1 ~]$ date >> DateOutput
[oracle@rac1 ~]$ more DateOutput
Todays date is
Thu Jan 30 08:24:23 IST 2014

With Grouping – Checking the parentheses used to group commands echo & date, > before the date in second line(highlighted in yellow) shows that the you have opened the parentheses for grouping and waiting for other commands.

Check the output is same from the above(in which programs as executed individually) and below (programs are grouped together using parentheses () )

[oracle@rac1 ~]$ (echo Todays date is
> date ) > DateOutput
[oracle@rac1 ~]$ cat DateOutput
Todays date is
Thu Jan 30 08:29:03 IST 2014

Instead you can use the ; (Line COntroller) character to separate the commands and mention the commands in the same line as below

[oracle@rac1 ~]$ (echo User Logged is $LOGNAME today at ;date) > UserLogged

[oracle@rac1 ~]$ cat UserLogged
User Logged is oracle today at
Thu Jan 30 08:48:04 IST 2014

Unix Shell Scripting - Trouble with Quotes



Trouble with Quotes in Unix shell Scripting:

- In Unix shell scripting, there are 3 types of quotes
    - single quote -  '
    - Double quote -  "
    - Back quote   - `

- Single Quotes - The text enclosed in single quote is considered to be normal text including the special characters like - $, #, *, [, space character(space is a special character used to identify the list of variables), newline character except for another single quote(')
[oracle@rac1 ~]$ echo 'The user logged in is $LOGNAME'
The user logged in is $LOGNAME

- Double Quotes - The double quote will maintain the special characters as is, like $, `
[oracle@rac1 ~]$ echo "The user logged in is $LOGNAME"
The user logged in is oracle

To remove the special meaning of characters used in double quotes those need to be escaped using backslash \
[oracle@rac1 ~]$ echo "The user logged in is \$LOGNAME"
The user logged in is $LOGNAME


- Back Quotes - 
- The commands mentioned in the back quotes are treated as Unix commands and executed in a new shell and output is displayed on to the shell from which the command is run

[oracle@rac1 ~]$ echo "Todays date is `date`"
Todays date is Thu Jan 30 08:03:16 IST 2014

- The output generated by the commands in back quotes will not have the newlines, multiple spaces or tabs are replaced with single space.

[oracle@rac1 ~]$ who | sort
oracle   pts/0        2014-01-27 07:47 (:0.0)
oracle   tty1         2014-01-27 07:45 (:0)

[oracle@rac1 ~]$ list=`who | sort`
[oracle@rac1 ~]$ echo $list
oracle pts/0 2014-01-27 07:47 (:0.0) oracle tty1 2014-01-27 07:45 (:0)

Tuesday, January 14, 2014

Blocker Sessions in Oracle

In this post basically I will try to explain all about session blockers(or locks), how to identify them and finally how to resolve them.(In Production after getting the confirmation from the application development teams.)

For identifying the blockers, all the details are present in the dynamic views

1. v$session
2. v$lock
3. v$process

In the above dynamic views the columns which are useful to identify the details are as below

1. In v$session

- SID – Session Identifier.(This is a sequence number generated from audses$ sequence)
- SERIAL# – Session serial number. Used to identify a session’s objects.( SID & Serial# uniquely identify the session to be killed after the blocker session is identified).
- PADDR – Address of OS level process that owns the session
- USERNAME – Oracle User Name
- COMMAND – Command in progress (last statement parsed). We can find the sql command using the query - SELECT command_name FROM v$sqlcommand WHERE command_type = n;
- LOCKWAIT – Address of the lock the session is waiting for; NULL if none
- STATUS – Status of the session;
        ACTIVE – Session currently working executing the SQL statements.
        INACTIVE - Session which is inactive and either has no configured limits or has not yet exceeded the configured limits
        KILLED - Session marked to be killed
        CACHED - Session temporarily cached for use by Oracle*XA
        SNIPED - An inactive session that has exceeded some configured limits.
- SCHEMANAME - Schema User name
- OSUSER - Operating System client user name
- PROCESS - Operating system client process ID
- MACHINE - Operating system machine name
- TERMINAL - Operating system Terminal name
- PROGRAM - Operating system program name
- TYPE - Session Type
- SQL_ID - SQL identifier of the SQL statement that is currently being executed
- MODULE - Name of the currently executing module as set by calling the DBMS_APPLICATION_INFO.SET_MODULE procedure
- ROW_WAIT_OBJ# - Object ID of the table containing the row specified in ROW_WAIT_ROW#
- ROW_WAIT_FILE# - Identifier of the data file containing the row specified in ROW_WAIT_ROW#
- ROW_WAIT_BLOCK# - Identifier for the block containing the row specified in ROW_WAIT_ROW#
- ROW_WAIT_ROW# - Current Row being locked
- LAST_CALL_ET - If the session status is ACTIVE, then the value represents the elapsed time(in seconds) since the session has become active.
                 If the session status is INACTIVE, then the value represents the elapsed time(in seconds) since the session has become inactive.
- BLOCKING_SESSION - Session identifier of the blocking session.
- EVENT - Resource or event for which the session is waiting
- STATE - Wait State
            - WAITING - Session is currently waiting
            - WAITED UNKNOWN TIME - Duration of the last wait is unknown
            - WAITED SHORT TIME - Last wait was less than hundredth of a second
            - WAITED KNOWN TIME - Duration of the last wait is specified in the WAIT_TIME column

Creating the Blocking Sessions (For Demonstration):

Let us now create the row blockers for demonstration purpose. Consider the HR sample schemas which come with Oracle Installation.

For creating the row level locks, let us have two sessions

Session I – Try to updating Region_ID value of India country to 1 (currently it is 3 as shown below) in HR schema table – COUNTRIES. And please note that after the update statement the COMMIT statement is not executed making the update statement grab the exclusive row lock on table – COUNTRIES

captured_Image.png 
SQL> update HR.COUNTRIES set REGION_ID = 1 where COUNTRY_NAME = 'India';

1 row updated.

SQL
> --Note that the changes are not COMMITTED--

Session II – Open a new session by connecting to the database using user account SH and try updating the same row(Country India, Region Id value to say 10) in HR.COUNTRIES table.


SQL> show user
USER is "SH"
SQL
>
SQL
> update HR.COUNTRIES set REGION_ID=10 where COUNTRY_NAME='India';

--Note that the SQL> prompt is not returned as this session is waiting above HR session update to get committed or rollback--

Now lets check the details on which session is blocking which one and gathered the necessary details by querying the v$session, v$lock & v$process dynamic views.


Lets connect to database as sys dba to check the details about locks executing the below query on v$session dynamic view.


SQL> set linesize 10000
column STATE format a20
column USERNAME format a10
column EVENT format a30
select SID, SERIAL#, USERNAME, STATUS, STATE, EVENT, BLOCKING_SESSION from v$session where USERNAME is not null;

SID SERIAL# USERNAME STATUS STATE EVENT BLOCKING_SESSION
---------- ---------- ---------- -------- -------------------- ------------------------------ ----------------
249 5487 HR INACTIVE WAITING SQL*Net message from client
262 1889 SH ACTIVE WAITING enq: TX - row lock contention 249
377 1135 SYS ACTIVE WAITED SHORT TIME SQL*Net message to client

Below are the inference we can make from the above query details::
1. It can be noticed from the above query output that the session with SID – 249 (Update query executed by HR user) is basically blocking the session with SID – 262 (the update query executed by SH user).
2. Note the Event column of SID – 262 (SH User), “enq: TX – row lock contention” mean that the lock currently existing row level lock.
3. SID – 249 is in INACTIVE status and it is basically WAITING for other messages to be received from the client(in our case sql*plus client), perhaps the commit/rollback or other queries.
4. SID – 262 is in ACTIVE status and it is basically WAITING for the SID – 249 (HR User) to commit/rollback the transaction in which he is trying to update the row(Country India, REGION_ID = 1)
5. SID – 377 is the session from SYS login, where we have executed the query to find the details about the locks.
 
Now lets find the query details which is basically leading to row contention(lock). For this we need to look for sql_id column value in v$session and then sql_text value in v$sql dynamic view.
Modified query with the sql_id column included is below (please note that my DB session has disconnected and when tried the row lock creation after connecting to DB, system created different SID’s but the concept wise it is all same)
 
set linesize 10000
column STATE format a20
column USERNAME format a10
column EVENT format a30
select SID, SERIAL#, USERNAME, STATUS, STATE, EVENT, SQL_ID, BLOCKING_SESSION from v$session where USERNAME is not null;SQL> SQL> SQL> SQL>

SID SERIAL# USERNAME STATUS STATE EVENT SQL_ID BLOCKING_SESSION
---------- ---------- ---------- -------- -------------------- ------------------------------ ------------- ----------------
26 5 HR INACTIVE WAITING SQL*Net message from client 50tbf7vd82htx
254 14391 SYS ACTIVE WAITED SHORT TIME SQL*Net message from client 3tyc5ht9g7ynw
259 7621 SH ACTIVE WAITING enq: TX - row lock contention 63r8m52wk2uut 26
Now querying the v$sql view for the sql_id will give the details of the query causing the row contention.
 
select DISTINCT sq.sql_id, s.PROCESS "OS_ProcessID", s.SID, s.serial#, s.USERNAME "Executed_By_User", sq.sql_text from v$sql sq
JOIN v$session s ON (s.sql_id = sq.sql_id) where s.USERNAME is not null;

SQL_ID OS_ProcessID SID SERIAL# Executed_By_User SQL_TEXT
------------- ------------------------ ---------- ---------- ------------------------------ ------------------------------
50tbf7vd82htx 3799 26 5 HR update HR.COUNTRIES set REGION
_ID
= 1 where COUNTRY_NAME = '
India
'

9xbzgbk9znfkd
6288 254 14391 SYS select DISTINCT sq.sql_id, s.P
ROCESS "OS_ProcessID", s.SID,
s.serial#, s.USERNAME "
Execute
d_By_User", sq.sql_text
from v
$sql sq
JOIN v$session s ON (s
.sql_id
= sq.sql_id) where s.U
SERNAME
is not null

63r8m52wk2uut
27508 259 7621 SH update HR.COUNTRIES set REGION
_ID
=10 where COUNTRY_NAME='Ind
ia
'
 
 
Now that you have identified the queries causing the contention, we need to decide on killing them or committing or rollback based on the application need. Once the development team confirms that we can go ahead and kill the culprit session.  There are two ways we can kill the offensive session 
1. Using the database – ALTER SYSTEM KILL SESSION ‘SID, SERIAL#’;
2. Using the OS level Kill command – KILL –9 <OS_ProcessID>

SQL> !ps -ef | grep '3799\|6288\|27508' | grep -v grep
oracle
3799 3027 0 Jan14 pts/0 00:00:00 sqlplus as sysdba
oracle
6288 6221 0 Jan14 pts/2 00:00:00 sqlplus
oracle
27508 7698 0 07:51 pts/3 00:00:00 sqlplus

Monday, November 18, 2013

RAC - Validating ASM Instance


- For ASM storage, we need two components 1. ASM Drivers 2. Grid Infrastructure

- ASM Drivers is a optional nice to have - configures ASM disks, assigns ASM drivers, configuration at the node bootup

- To install oracleasm driver we need to install - 1. oracleasmlib(chipset specific) 2. oracleasm-support(chipset specific) 3. oracleasm(specific to OS)

- oracleasm driver will be installed in /etc/init.d location

[root@rac1 init.d]# pwd
/etc/init.d
[root@rac1 init.d]# ls -lart oracl*
-rwxr-xr-x. 1 root root 7124 Feb  9  2013 oracleasm
-rwx------. 1 root root 1450 Apr 24  2013 oracle-rdbms-server-12cR1-preinstall-firstboot
[root@rac1 init.d]# ./oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

- Grid Infrastructure installation installs 1. ASM software 2. Clusterware software(CRS, CSS, SSH) 3. Installs & configures ASM instance

- After installing the GI, check for ASM process running

[root@rac1 init.d]# ps -ef | grep smon
oracle    1959     1  0 00:49 ?        00:00:00 ora_smon_MyDB1
oracle    2008     1  0 00:49 ?        00:00:00 mdb_smon_-MGMTDB
oracle    2690     1  0 Nov18 ?        00:00:01 asm_smon_+ASM1  <-- This confirms that the GI installation has installed and configured ASM instance.
root      2726     1  2 Nov18 ?        00:14:27 /u01/app/12.1.0/grid/bin/osysmond.bin
root      4461  3453  0 01:38 pts/0    00:00:00 grep smon

As part of GI installation ASM entry is made in /etc/oratab file

[root@rac1 init.d]# more /etc/oratab
#Backup file is  /u01/app/oracle/product/12.1.0/dbhome_1/srvm/admin/oratab.bak.rac1 line added by Agent
#

# This file is used by ORACLE utilities.  It is created by root.sh
# and updated by either Database Configuration Assistant while creating
# a database or ASM Configuration Assistant while creating ASM instance.

# A colon, ':', is used as the field terminator.  A new line terminates
# the entry.  Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
#   $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third field indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/u01/app/12.1.0/grid:N: # line added by Agent
-MGMTDB:/u01/app/12.1.0/grid:N: # line added by Agent
MyDB:/u01/app/oracle/product/12.1.0/dbhome_1:N: # line added by Agent

Apart from above process, oratab entry, GI installation will create the two network called - vip & scan network

[root@rac1 init.d]# ping rac1-vip
PING rac1-vip.localdomain (192.168.56.81) 56(84) bytes of data.
64 bytes from rac1-vip.localdomain (192.168.56.81): icmp_seq=1 ttl=64 time=0.026 ms
64 bytes from rac1-vip.localdomain (192.168.56.81): icmp_seq=2 ttl=64 time=0.029 ms
64 bytes from rac1-vip.localdomain (192.168.56.81): icmp_seq=3 ttl=64 time=0.066 ms
^C
--- rac1-vip.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2824ms
rtt min/avg/max/mdev = 0.026/0.040/0.066/0.018 ms

[root@rac1 init.d]# ping rac-scan
PING rac-scan.localdomain (192.168.56.91) 56(84) bytes of data.
64 bytes from rac-scan.localdomain (192.168.56.91): icmp_seq=1 ttl=64 time=1.48 ms
64 bytes from rac-scan.localdomain (192.168.56.91): icmp_seq=2 ttl=64 time=0.405 ms
64 bytes from rac-scan.localdomain (192.168.56.91): icmp_seq=3 ttl=64 time=0.477 ms
^C
--- rac-scan.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2846ms
rtt min/avg/max/mdev = 0.405/0.788/1.484/0.493 ms

- Manually asm disk groups configuration can be performed using - asmca

- To check the details of spfile(binary) of ASM instance, create a pfile from spfile by logging into sqlplus as sysasm user. 

[oracle@rac1 disks]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1 disks]$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.1.0 Production on Tue Nov 19 01:53:53 2013

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> create pfile='/home/oracle/init+ASM1.ora' from spfile;

File created.

SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
[oracle@rac1 disks]$ cd ~
[oracle@rac1 ~]$ ls -lart init*
-rw-r--r-- 1 oracle oinstall 272 Nov 19 01:55 init+ASM1.ora
[oracle@rac1 ~]$ cat init+ASM1.ora
+ASM1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from in memory value
+ASM2.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from in memory value
*.asm_diskstring='/dev/oracleasm/disks'
*.asm_power_limit=1
*.large_pool_size=12M
*.remote_login_passwordfile='EXCLUSIVE'

Sunday, November 17, 2013

RAC - Oracle Cluster Registry (OCR), OLR & VOTING Disk Locations


In RAC cluster, 2 disks (to be presise files) are important
- 1. Voting Disk - Heartbeat of the RAC components
- 2. OCR - Oracle Cluster Registry - It has all list of RAC components (it is just like windows registry)
           It has all the components of RAC as DB Instances, ASM Instances, Listeners, SCAN listeners, RAC nodes

Voting disk checks for the heart beat of all the RAC nodes and if it can't receive the heart beat from a particular node, then voting disk evicts that particular node from RAC cluster.
Voting disk gets the details of all the nodes it is supposed to check the alive heartbeat from OCR disk.

Utilities to determine the location of OCR 
- OLR (Oracle Local Registry) - registry specific to a node
          OCR - Registry for overall RAC
          OLR - Registry for specific node
- OCRCHECK
- ASMCMD

- To work in OCR, the environment need to be set to ASM instance (+ASM1/2) or GI

--------------------------------------------------------------------------------------------
- To know which environment we need to mention in the rac cluster on executing . oraenv, we need to check the details in the /etc/oratab

[oracle@rac1 ~]$ cat /etc/oratab

#Backup file is  /u01/app/oracle/product/12.1.0/dbhome_1/srvm/admin/oratab.bak.rac1 line added by Agent
#
# This file is used by ORACLE utilities.  It is created by root.sh
# and updated by either Database Configuration Assistant while creating
# a database or ASM Configuration Assistant while creating ASM instance.

# A colon, ':', is used as the field terminator.  A new line terminates
# the entry.  Lines beginning with a pound sign, '#', are comments.
#
# Entries are of the form:
#   $ORACLE_SID:$ORACLE_HOME:<N|Y>:
#
# The first and second fields are the system identifier and home
# directory of the database respectively.  The third field indicates
# to the dbstart utility that the database should , "Y", or should not,
# "N", be brought up at system boot time.
#
# Multiple entries with the same $ORACLE_SID are not allowed.
#
#
+ASM1:/u01/app/12.1.0/grid:N: # line added by Agent
-MGMTDB:/u01/app/12.1.0/grid:N: # line added by Agent
MyDB:/u01/app/oracle/product/12.1.0/dbhome_1:N: # line added by Agent


- On entering the ORACLE_SID value while setting the environment variable, oracle RAC shouldn't prompt us with any other parameter to enter, that confirms we have set the environment correctly.

[oracle@rac2 ~]$ . oraenv
ORACLE_SID = [oracle] ? +ASM1 <Entering wrong ASM instance value +ASM1 instead of +ASM2, environment asked for ORACLE_HOME value to enter as below>
ORACLE_HOME = [/home/oracle] ? 

[oracle@rac2 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM2 <On entering the correct ASM instance +ASM2 on node 2, system didn't prompt 
for any variable value to be entered. It successfully set the environment as below>
The Oracle base has been changed from exit to /u01/app/oracle
[oracle@rac2 ~]$ 
----------------------------------------------------------------------------------------------------------------------------------
Find location of OCR Local disk file:
To find the location of the olr disk(or file), use ocrcheck with local clause as below.
Per the below details, the olr binary file is located at - /u01/app/12.1.0/grid/cdatarac1.olr

- On node 1 (rac1)

[root@rac1 ~]# . oraenv
ORACLE_SID = [root] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[root@rac1 ~]# ocrcheck -local
Status of Oracle Local Registry is as follows :
 Version                  :          4
 Total space (kbytes)     :     409568
 Used space (kbytes)      :        968
 Available space (kbytes) :     408600
 ID                       :  882531601
 Device/File Name         : /u01/app/12.1.0/grid/cdata/rac1.olr
                                    Device/File integrity check succeeded

 Local registry integrity check succeeded

 Logical corruption check succeeded


- On node 2 (rac2)

[root@rac2 ~]# . oraenv
ORACLE_SID = [+asm2] ? +ASM2
The Oracle base has been changed from /home/oracle to /u01/app/oracle
[root@rac2 ~]# ocrcheck -local
Status of Oracle Local Registry is as follows :
Version                  :          4
Total space (kbytes)     :     409568
Used space (kbytes)      :        976
Available space (kbytes) :     408592
ID                       :  873311553
Device/File Name         : /u01/app/12.1.0/grid/cdata/rac2.olr
                                    Device/File integrity check succeeded

Local registry integrity check succeeded

Logical corruption check succeeded



Alternatively one can find the location of the olr file from olr.loc file present in /etc/oracle
- On node 1 (rac1)

[root@rac1 oracle]# pwd
/etc/oracle
[root@rac1 oracle]# ls -lart
total 2876
drwxr-xr-x    3 root oinstall    4096 Nov  8 06:12 scls_scr
drwxrwxr-x    5 root oinstall    4096 Nov  8 06:12 oprocd
-rws--x---    1 root oinstall 2903931 Nov  8 06:12 setasmgid
-rw-r--r--    1 root root           0 Nov  8 06:12 olr.loc.orig
-rw-r--r--    1 root oinstall      80 Nov  8 06:12 olr.loc
-rw-r--r--    1 root root           0 Nov  8 06:12 ocr.loc.orig
-rw-r--r--    1 root oinstall      37 Nov  8 06:12 ocr.loc
drwxr-xr-x    6 root oinstall    4096 Nov  8 06:12 .
drwxrwx---    2 root oinstall    4096 Nov  8 06:29 lastgasp
drwxrwxrwt    2 root oinstall    4096 Nov 16 23:30 maps
drwxr-xr-x. 123 root root       12288 Nov 17 18:56 ..
[root@rac1 oracle]# more olr.loc
olrconfig_loc=/u01/app/12.1.0/grid/cdata/rac1.olr
crs_home=/u01/app/12.1.0/grid

- On node 2 (rac2)

[root@rac2 ~]# cd /etc/oracle
[root@rac2 oracle]# ls -lart
total 2876
drwxr-xr-x    3 root oinstall    4096 Nov  8 06:36 scls_scr
drwxrwxr-x    5 root oinstall    4096 Nov  8 06:36 oprocd
-rws--x---    1 root oinstall 2903931 Nov  8 06:36 setasmgid
-rw-r--r--    1 root root           0 Nov  8 06:36 ocr.loc.orig
-rw-r--r--    1 root oinstall      37 Nov  8 06:36 ocr.loc
-rw-r--r--    1 root root           0 Nov  8 06:36 olr.loc.orig
-rw-r--r--    1 root oinstall      80 Nov  8 06:36 olr.loc
drwxr-xr-x    6 root oinstall    4096 Nov  8 06:36 .
drwxrwx---    2 root oinstall    4096 Nov  8 06:42 lastgasp
drwxrwxrwt    2 root oinstall    4096 Nov 16 23:30 maps
drwxr-xr-x. 123 root root       12288 Nov 17 18:56 ..
[root@rac2 oracle]# more olr.loc
olrconfig_loc=/u01/app/12.1.0/grid/cdata/rac2.olr
crs_home=/u01/app/12.1.0/grid


Find location of OCR disk file:
- OCR & Voting disk must reside in the ASM disk group
- OCR & Voting disks works hand-in-hand to determine what are the nodes currently active and responding to heartbeat
- In RAC cluster all nodes should have the same server timestamp, else if they are off by even few secs, voting disks considers those nodes as not active and evicts them from the cluster.  To maintain the same timestamp across the cluster nodes, NTP(Network Time Protocol) is used.

Using ocrcheck one can find the location of the ASM disk group in which OCR file is located

- [root@rac1 bin]# ocrcheck
Status of Oracle Cluster Registry is as follows :
 Version                              :          4
 Total space (kbytes)        :     409568
 Used space (kbytes)        :       1528
 Available space (kbytes) :     408040
 ID                       :  121423581
 Device/File Name         :      +DATA
                                    Device/File integrity check succeeded

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

                                    Device/File not configured

 Cluster registry integrity check succeeded

 Logical corruption check succeeded

- Location of the OCR disk file in ASM disk group

[oracle@rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1 ~]$ asmcmd
ASMCMD> ls
DATA/
ASMCMD> cd data
ASMCMD> ls
ASM/
MYDB/
_MGMTDB/
orapwasm
rac-cluster/   <-- This is the cluster name given during the GI installation
ASMCMD> cd rac-cluster
ASMCMD> ls
ASMPARAMETERFILE/
OCRFILE/
ASMCMD> cd ocrfile
ASMCMD> ls
REGISTRY.255.830932005 <--OCR file is located in ASM disk grip
                     +data/rac-cluster/ocrfile/registry.255.830932005 (Binary file)




Find location of Voting Disk:
- Typically CRSCTL utility commands should be issued from the Grid Infrastructure environment(+ASM1 or +ASM2 ...)
  SRVCTL utility commands should be issued from Database environment (+<DBInstance>1, +<DBInstance>2, ..)

- To know the location of the voting disk file, execute the below command in GI environment as below

[oracle@rac1 ~]$ . oraenv
ORACLE_SID = [+ASM1] ? +ASM1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@rac1 ~]$ crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   7427f69521004f49bf9221af584d4e6e (/dev/oracleasm/disks/DISK1) [DATA]
Located 1 voting disk(s).

Friday, November 8, 2013

Oracle 12c RAC configuration in Linux virtual machine using Virtual Box – Guest OS Windows 8

For installing RAC database you will need the below files which can be downloaded from oracle site.
Oracle Linus 6.4 64 bit OS - V37084-01.iso     3.6G
https://edelivery.oracle.com/EPD/Download/get_form?egroup_aru_number=16064752
Oracle Database (includes Oracle Database and Oracle RAC)
  
  linuxamd64_12c_database_1of2.zip        1.3G  (1,361,028,723 bytes) (cksum - 3389130601)
  linuxamd64_12c_database_2of2.zip        1.1G  (1,116,527,103 bytes) (cksum - 273248753)

Oracle Grid Infrastructure (includes Oracle ASM, Oracle Clusterware):
  linuxamd64_12c_grid_1of2.zip                 1.7G  (1,750,478,910 bytes) (cksum - 3177055641)
  linuxamd64_12c_grid_2of2.zip                 196M  (201,673,595 bytes) (cksum - 2753782116)

Virtual Box Version – 4.2.18 (or the latest version available currently – 4.3.2)
Below is the system configuraton which I used for this virtual RAC configuration
RAM – 16GB
OS – Windows 8
HDD – 1 TB
Processor – 3rd generation i7 Quad Core Processor

Virtual Machine Setup:
Navigate to Virtual Box File à Preferences à Network and Edit the Host-Only Network with IP subnet as below and click OK
IP – 192.168.56.0
Subnet mask – 255.255.255.0

Lets start creating the Oracle Linux virtual server machine, and then eventually clone the same to create the other node of RAC set-up.

Launch Virtual box and click on New and enter the details as below


Have 5G of memory is allotted to virtual machine

Create a new virtual drive as below

Select VDI as below

Select Dynamically allocated

Browse the location in windows explorer to save the VDI. Have the drive created for 50G

Below is how the newly created virtual machine looks like with the details in the right pane.


Now let’s define the networking adapters by clicking the Network option for rac1 virtual machine created as below

For Adapter1 select the options as below, make a note of the MAC address


For Adapter 2 select/mention the details as below

For Adapter 3, select/mention the details as below. Select Wireless or Ethernet drive name which you are using to access the internet in your laptop while you trying to configure this RAC set up now.
In my case I am have my laptop connected to Wireless signal, that’s why I selected Wireless driver name.

Once all the 3 adapters are configured, the details under Network option look like below



Guest Linux 6.4 OS Installation:

In the new virtual machine created, select Storage option in the right pane as below

Select DVD icon and browse for the V37ob4-01.iso (Oracle Linux 6.3 OS) under Storage & Controller: IDE option as below

In Systems tab in the left pane, move CD/DVD-ROM option to top as below and click OK


Under System à Processor increase processor allotted to 4, to have good performance when two nodes of RAC are started.

Now start the rac1 virtual machine to install the Oracle Linux OS by clicking Start button as below

When the selection is on ‘Install and Upgrade an existing system‘ press Enter

Skip Disc checking




Click Next


Select English and click Next

Select US English Keyboard and click Next

Select Basic Storage Devices and click Next


Select ‘Yes, Discard any data’ option

Mention Hostname as – rac1.localdomain, and click on configure network

Select System eth0 network and mention the IP details as  192.168.56.71, 255.255.255.0




Now select System eth1 network and mention the IP details as – 192.168.10.1, 255.255.255.0



Finally select System eth2 network and select the options as below, this network is used for accessing the internet in the guest linux virtual machine



Now click Next

Select your Time Zone and click Next

Set the root password and click Next

Select Use All Space and Review & Modify partition layout and then click Next

Check that the swap memory is around 5G and rest of the space is mounted to /


Format /dev/sda and write changes to the disk



Have defaults and click next in the below screen

Select database server and customize each component as mentioned in below screenshots which will be useful to configure the RAC setup once the Linux OS is installed








Installation begins as below




After Installation is complete. Reboot the Virtual machine and follow the screens mentioned post reboot










Check Internet Access by pinging any .com site like – yahoo.com as below. If the response comes back we are good. Else disconnect from eth0 & eth1 networks keeping eth2 (Bridged Adapter) is connected and try to ping yahoo.com to test the internet connectivity



Oracle 12c Pre-Install package installation:

Run  yum install oracle-rdbms-server-12cR1-preinstall –y to perform Oracle database 12c kernel parameter change and creation of Linux oracle account.

If any app is holding the yum lock, kill that app using the PID displayed.



Once the pre-install package is installed, do a complete yum update using - $ yum update command (screenshots for this are not captured in here)


Installation of Guest Additions:

Reboot the Virtual machine and click Devices à Install Guest Additions. In the Guest Additions install pop-up displayed, select Open Autorun prompt and then install it by mentioning the root password.






If for some reason building of guest addition module failed, install the missing package and then try again the installation of Guess Addition as shown above.





Reboot the virtual machine and then login as root using su and Install ASMLib from yum as below
# yum install oracleasm
# yum install oracleasm-support
It is noticed that as part of yum update oracleasm package has already been installed

Install oracleasm-support in root login itself


Configure ASM in an interactive (-i) way as below.
Answers to the interactive questions posted are in the order - oracle, oinstall, y & default y (just press enter key)

Create the directory in which the Oracle software will be installed. And map oracle user to group oinstall and give ownership to /u01 and 775 permissions.
mkdir -p  /u01
chown -R oracle:oinstall /u01
chmod -R 775 /u01/

[rvalusa@rac1~]$ su - root
Password:
[root@rac1~]# mkdir -p  /u01
[root@rac1~]# chown -R oracle:oinstall /u01
[root@rac1~]# chmod -R 775 /u01/
[root@rac1~]# 

Add oracle account to dba and vboxsf groups. The vboxsf group was created by VirtualBox Guest Additions and will allow oracle user access folders in the Host OS:

# usermod -G dba,vboxsf oracle

[root@rac1~]# usermod -G dba,vboxsf oracle

Reset oracle user password:

[root@rac1~]# passwd oracle
Changing password for user oracle.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.
[root@rac1~]#

Disable secure linux by editing the "/etc/selinux/config" file, making sure the SELINUX flag is set as follow

SELINUX=disabled
Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. In this case we will deconfigure NTP.

# service ntpd stop
Shutting down ntpd:                                        [FAILED]
# chkconfig ntpd off
# mv /etc/ntp.conf /etc/ntp.conf.orig
# rm /var/run/ntpd.pid

Cleanup YUM repositories:
# yum clean all

[root@rac1 ~]# yum clean all
Loaded plugins: refresh-packagekit, security
Cleaning repos: public_ol6_UEK_latest public_ol6_latest
Cleaning up Everything

Edit "/etc/hosts" file by appending the following information:

# Private
192.168.10.1    rac1-priv.localdomain   rac1-priv
192.168.10.2    rac2-priv.localdomain   rac2-priv

# Public
192.168.56.71    rac1.localdomain        rac1
192.168.56.72    rac2.localdomain        rac2

# Virtual
192.168.56.81    rac1-vip.localdomain    rac1-vip
192.168.56.82    rac2-vip.localdomain    rac2-vip

# SCAN
192.168.56.91    rac-scan.localdomain    rac-scan
192.168.56.92    rac-scan.localdomain    rac-scan
192.168.56.93    rac-scan.localdomain    rac-scan

Note. The SCAN address should not really be defined in the hosts file. Instead it should be defined on the DNS to round-robin between 3 addresses on the same subnet as the public IPs. For this installation, we will compromise and use the hosts file. If you are using DNS, then comment out lines with SCAN addresses. 

Now we need to disable the firewall: Linux Main menu | System | Administration | Firewall. Click on "Disable" icon, then on "Apply".




Creating Sharing Folders in VirtualBox:
To install the grid infrastructure and database, we need to access the required installables - grid & database which are present in the folder in Host OS windows 8.

Unzip both grid and database files to a folder named - OracleInstallables and keep the files related to grid 12c in grid folder & database 12c files in database folder

Content of the below zip will be kept in folder named grid (Copy files in install & stage folder in linuxamd64_12c_grid_2of2.zip to respective folder found on unzipping linuxamd64_12c_grid_1of2.zip )
  linuxamd64_12c_grid_1of2.zip
  linuxamd64_12c_grid_2of2.zip

Content of the below zip will be kept in folder named database (Copy files in install & stage folder in linuxamd64_12c_database_2of2.zip to respective folder found on unzipping linuxamd64_12c_database_1of2.zip )
  linuxamd64_12c_database_1of2.zip
  linuxamd64_12c_database_2of2.zip

Browse for the folder - OracleInstallables and then Auto-Mount and make it permanent.
The shared folder will be listed in Guest OS -Linux at  /media/sf_OracleInstallables.


Now reboot the VM and install the grid pre-install rpm as root login as below

Clone the Virtual Machine:
Shutdown the running rac1 VM to clone it to create a similar VM which can be used as other node of RAC configuration.  Below are the screenshots which depict the details on cloning the VM

select rac1 VM --> Machine --> Clone.. option, mention the name of new VM as rac2 (don't select Reinitialize the MAC address of all network cards)

Select Full clone and let the clone get created.. It may take few mins to get the clone creation complete.


Start cloned VM rac2 and login as root user. Then change hostname by editing file "/etc/sysconfig/network", HOSTNAME parameter:

HOSTNAME=rac2.localdomain

Start "Network Connections" tool (Main menu | System | Preferences | Network Connections). Edit eth0 and eth1 interfaces and set in IPv4 addresses 192.168.56.72 and 192.168.10.2 correspondingly and then Reboot





Now we need to change MAC address for all three interfaces. At the moment we have two VMs with the same set of MAC addresses. We can run one machine or another, but not both of them at the same time because MAC address must be unique. No changes will be made to rac1, we will pick up three new unused addresses and set them for eth0, eth1, and eth2 in rac2. 

The easiest way to do that is to change just last two characters of the address. We are going to change them to '00'. If the last two characters are already '00', then change to something else, '01', for example. Just make sure that these addresses don't collide with the MAC addresses of rac1. In running rac2 node, open "Network Connections" and edit MAC address in the "Wired" tab. The screenshot below shows where to set MAC address. 

Don't forget to change MAC addresses for all three interfaces. Please note that your setup will have a different set of MAC addresses because they are random-generated by VirtualBox.





Write down the new MAC addresses for all three interfaces. Save new settings pressing "Apply" button, then shutdown the machine. After shutdown, return to the VirtualBox Manager, select rac2 VM and edit "Network" settings. Make same changes to the MAC addresses. Don't forget to change MAC addresses for all three adapters.

Rac2
eth0 - 192.168.56.72 - MAC - 08:00:27:BF:D4:00 – Adapter 1
eth1 - 192.168.10.2  - MAC - 08:00:27:13:FE:00 – Adapter 2
eth2 - Automatic        - MAC - 08:00:27:28:2B:00 – Adapter 3





Start both rac1 & rac2 VMs and check the connectivity to both VMs from each other by pinging each.




Creating Shared Disk: Shut down both virtual machines. We need to create a new virtual disk, change its attribute to Shareable and add to both VMs. In the current version of VirtualBox, the only way to create a new disk in the GUI is through the "Storage" page in the virtual machine's settings. Select either rac1 or rac2 VM, then click on "Storage" link. Select "SATA Controller" and click on "Add Hard Disk" icon. If not sure, which icon to use, same action is available through the popup menu, right-click on the "SATA Controller" and select "Add Hard Disk"


save the .vdi file to local drive in your PC as below and mention the size of the shared drive to 12G (or > 12G)


Note that the .vdi shared disk added will be in normal type (not shared type). We need to change that to shared type so that it is accessible to both RAC nodes


Remove the existing shared disk using remove virtual disk button and then click on the Add Virtual Disk button to add the same virtual disk in Shared mode or type.





Start either of the machines and log in as root. The current disks can be seen by issuing the following commands.

# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb
#
Use the "fdisk" command to partition the new disk "sdb".

# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd724aa83.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-305, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-305, default 305): 
Using default value 305

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
#
The sequence of answers is "n", "p", "1", "Return", "Return" and "w".

Once the new disk is partitioned, the result can be seen by repeating the previous "ls" command.

# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sdb  /dev/sdb1
#
Mark the new shared disk in the ASMLib as follows.

# oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done

Run the "scandisks" command to refresh the ASMLib disk configuration.

# oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
#
We can see the disk is now visible to ASM using the "listdisks" command.

# oracleasm listdisks
DISK1
#
Start another VM and log in as root. Check that the shared disk is visible to ASM using the "listdisks" command.

# oracleasm listdisks
DISK1
#
The virtual machines and shared disks are now configured for the grid infrastructure!


Installing Grid Infrastructure:
Make sure the "rac1" and "rac2" virtual machines are started, then login to "rac1" or switch the user to oracle and start the Oracle installer.

The below screenshots are self explanatory on what options need to be selected while installing the Grid Infrastructure (GI). Please note that the it is required to install the GI in one of the RAC node and then automatically that same will be installed in the other RAC node.





Work In-Progress.. Thanks for visiting the page.  Will update this for closure shortly