How to approach RMAN error  "UNTIL CHANGE is before RESETLOGS change"

A problem simulation and solution approach :-
First take your full database backup including archivelogs of your database,

RMAN> backup database plus archivelog;
Starting backup at 05-FEB-15
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=8 RECID=39 STAMP=870706643
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\11PUH46H_1_1 tag=TAG20150205T145721 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=1 RECID=40 STAMP=870879410
input archived log thread=1 sequence=2 RECID=41 STAMP=870879441
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\12PUH46I_1_1 tag=TAG20150205T145721 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 05-FEB-15
Starting backup at 05-FEB-15
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=C:\ORADATA\DB11\SYSTEM01.DBF
input datafile file number=00002 name=C:\ORADATA\DB11\SYSAUX01.DBF
input datafile file number=00003 name=C:\ORADATA\DB11\UNDOTBS01.DBF
input datafile file number=00004 name=C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:25
Finished backup at 05-FEB-15
Starting backup at 05-FEB-15
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=3 RECID=42 STAMP=870879532
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\14PUH49C_1_1 tag=TAG20150205T145852 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-FEB-15
Starting Control File and SPFILE Autobackup at 05-FEB-15
piece handle=C:\RMAN\C-1400701463-20150205-00 comment=NONE
Finished Control File and SPFILE Autobackup at 05-FEB-15
RMAN> exit
Recovery Manager complete.

Create some objects and do some action(DML and DDL) and note down the respective SCN after every statement.
C:\Users\mahekarthya>sqlplus sys as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 5 15:04:36 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
SQL> create
2 table test
3 as select * from
4 all_objects;
Table created.
SQL> select
2 current_scn
3 from
4 v$database;
CURRENT_SCN
-----------
1045503 -------------------- SCN Before delete
SQL> delete from test;
68604 rows deleted.
SQL> commit;
Commit complete.
SQL>
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
1048517 --------------------- SCN after delete
SQL> drop table test;
Table dropped.
SQL>
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
1048584
SQL>
SQL> alter system switch logfile;
System altered.
SQL>
SQL> select incarnation#, resetlogs_change#,status from v$database_incarnation;
INCARNATION# RESETLOGS_CHANGE# STATUS
------------ ----------------- -------
1 1 PARENT
2 834328 PARENT
3 838675 PARENT
4 1026219 PARENT
5 1028060 CURRENT --------- 1028060 is the CURRENT RESETLOGS_CHANGE#
SQL>
shutdown immediate
SQL>
C:\Users\mahekarthya>rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Feb 5 15:26:01 2015
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> run
2> {
3> startup mount;
4> set until scn=1048517; -----recovering to a point after delete statement is fired.
5> restore database;
6> recover database;
7> }
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
executing command: SET until clause
Starting restore at 05-FEB-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to C:\ORADATA\DB11\SYSTEM01.DBF
channel ORA_DISK_1: restoring datafile 00002 to C:\ORADATA\DB11\SYSAUX01.DBF
channel ORA_DISK_1: restoring datafile 00003 to C:\ORADATA\DB11\UNDOTBS01.DBF
channel ORA_DISK_1: restoring datafile 00004 to C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: reading from backup piece C:\RMAN\13PUH46M_1_1
channel ORA_DISK_1: piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:45
Finished restore at 05-FEB-15
Starting recover at 05-FEB-15
using channel ORA_DISK_1
starting media recovery
media recovery complete, elapsed time: 00:00:03
Finished recover at 05-FEB-15
RMAN> alter database open resetlogs;
database opened
RMAN>
As we opened the database using the resetlogs option, a new incarnation has been created. Below, is the new incarnation and the recovered table:
SQL> select incarnation#, resetlogs_change#,status from v$database_incarnation;
INCARNATION# RESETLOGS_CHANGE# STATUS
------------ ----------------- -------
1 1 PARENT
2 834328 PARENT
3 838675 PARENT
4 1026219 PARENT
5 1028060 PARENT --------- 1028060 became PARENT
6 1048518 CURRENT --------- & 1048518 became CURRENT(one SCN higher than what we provided in until SCN ,ie 1048517+1
6 rows selected.
SQL> select * from test;
no rows selected
SQL>
After this operation, I tried to recover the database up to the delete command(ie state before delete command has been fired against the table) issued as follows:
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
C:\Users\mahekarthya>rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Feb 5 15:39:23 2015
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> run
2> {
3> startup mount;
4> set until scn=1045503;
5> restore database;
6> recover database;
7> }
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
executing command: SET until clause
Starting restore at 05-FEB-15
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/05/2015 15:40:47
RMAN-20208: UNTIL CHANGE is before RESETLOGS change
RMAN>

I got this error , because I am trying to recover my database to a SCN(1045503) ie lower than the CURRENT resetlog SCN(1048518 ).Now I checked the list of all the database incarnations from RMAN, and I resets the database to the previous incarnation and performs an incomplete recovery as follows:
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 DB11 1400701463 PARENT 1 13-JAN-15
2 2 DB11 1400701463 PARENT 834328 14-JAN-15
3 3 DB11 1400701463 PARENT 838675 14-JAN-15
4 4 DB11 1400701463 PARENT 1026219 03-FEB-15
5 5 DB11 1400701463 PARENT 1028060 04-FEB-15
6 6 DB11 1400701463 CURRENT 1048518 05-FEB-15
RMAN> shutdown immediate;
database dismounted
Oracle instance shut down
RMAN> startup mount
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
RMAN> reset database to incarnation 5;
database reset to incarnation 5
RMAN> run
2> {
3> set until scn=1045503;
4> restore database;
5> recover database;
6> }
executing command: SET until clause
Starting restore at 05-FEB-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to C:\ORADATA\DB11\SYSTEM01.DBF
channel ORA_DISK_1: restoring datafile 00002 to C:\ORADATA\DB11\SYSAUX01.DBF
channel ORA_DISK_1: restoring datafile 00003 to C:\ORADATA\DB11\UNDOTBS01.DBF
channel ORA_DISK_1: restoring datafile 00004 to C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: reading from backup piece C:\RMAN\13PUH46M_1_1
channel ORA_DISK_1: piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:05
Finished restore at 05-FEB-15
Starting recover at 05-FEB-15
using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 3 is already on disk as file C:\ORADATA\DB11\ARCHIVE\LOG
archived log for thread 1 with sequence 4 is already on disk as file C:\ORADATA\DB11\ARCHIVE\LOG
archived log file name=C:\ORADATA\DB11\ARCHIVE\LOG1_3_870795195.ARC thread=1 sequence=3
archived log file name=C:\ORADATA\DB11\ARCHIVE\LOG1_4_870795195.ARC thread=1 sequence=4
media recovery complete, elapsed time: 00:00:02
Finished recover at 05-FEB-15
RMAN> alter database open resetlogs;
database opened
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 DB11 1400701463 PARENT 1 13-JAN-15
2 2 DB11 1400701463 PARENT 834328 14-JAN-15
3 3 DB11 1400701463 PARENT 838675 14-JAN-15
4 4 DB11 1400701463 PARENT 1026219 03-FEB-15
5 5 DB11 1400701463 PARENT 1028060 04-FEB-15
7 7 DB11 1400701463 CURRENT 1045504 05-FEB-15
6 6 DB11 1400701463 ORPHAN 1048518 05-FEB-15
RMAN> exit
Recovery Manager complete.
C:\Users\mahekarthya>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 5 15:48:08 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select count(1) from test;
COUNT(1)
----------
68604
SQL>

The table recovered to the previous incarnation successfully and a new incarnation has been created.

Convert a single instance database to 2 node RAC cluster database.

Here I am going to explain how to convert a single instance database to a two node RAC database .Keep in mind that, even if your source database is in RAC , there is no direct method to convert it into a  clone RAC database . First we need to convert it into a single instance database , later we need to add the newly configured single instance to cluster.

1.First take the full database backup of the single instance database,

BACKUP AS COMPRESSED BACKUPSET FULL TAG 'DFWEBFULL' DATABASE FORMAT '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL/Backup_%d_%T_%s'
INCLUDE CURRENT CONTROLFILE PLUS ARCHIVELOG format '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL/ARCHIVE_%d_%T_%s';

Note:- change the permission of the backup directory so that oracle software owner can read those files , otherwise you wil receive error.
$cd /nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL
$chmod 777 *
Here I took my rman backups into a nfs mount point , and its accessible  from the auxiliary database so that no need to copy the entire backup set .

2. Take one init parameter from the Cluster ( if your cluster have already some instance configured) where you want to configure your RAC instances, otherwise make a new one .keep the locat_listener and remote_listener as it is . Change the other parameter accordingly .Here I am going to configure RAC database TTWEB having two instances TTWEB1 and TTWEB2. Here is my init parameter and is named as initTTWEB1.ora
Note:- Comment out *.cluster_database=TRUE
$vi initTTWEB1.ora
TTWEB2.__db_cache_size=956301312
TTWEB1.__db_cache_size=1610612736
TTWEB2.__java_pool_size=67108864
TTWEB1.__java_pool_size=67108864
TTWEB2.__large_pool_size=83886080
TTWEB1.__large_pool_size=83886080
TTWEB2.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
TTWEB1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
TTWEB2.__pga_aggregate_target=1493172224
TTWEB1.__pga_aggregate_target=1493172224
TTWEB2.__sga_target=2801795072
TTWEB1.__sga_target=2801795072
TTWEB2.__shared_io_pool_size=0
TTWEB1.__shared_io_pool_size=0
TTWEB2.__shared_pool_size=1660944384
TTWEB1.__shared_pool_size=1006632960
TTWEB2.__streams_pool_size=0
TTWEB1.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/TTWEB/adump'
*.audit_trail='db'
#*.cluster_database=TRUE
*.compatible='11.2.0.4'
*.control_files='+ORADATA/TTWEB/control1.ctl','+ORAFRA/TTWEB/control2.ctl'
*.db_block_size=8192
*.db_create_file_dest='+ORADATA'
*.db_create_online_log_dest_1='+ORADATA'
*.db_create_online_log_dest_2='+ORAFRA'
*.db_domain=''
*.db_name='TTWEB'
*.db_recovery_file_dest='+ORAFRA'
*.db_recovery_file_dest_size=107374182400
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=TTWEBXDB)'
TTWEB1.instance_number=1
TTWEB2.instance_number=2
TTWEB1.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.81.155.146)(PORT=1521))'
TTWEB2.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.81.155.157)(PORT=1521))'
*.memory_target=4G
*.nls_language='AMERICAN'
*.open_cursors=300
*.pga_aggregate_target=536870912
*.processes=500
*.remote_listener='ptgracscan:1521'
*.remote_login_passwordfile='exclusive'
*.sga_target=1610612736
TTWEB1.thread=1
TTWEB2.thread=2
TTWEB1.undo_tablespace='UNDOTBS1'
TTWEB2.undo_tablespace='UNDOTBS2'
Copy initTTWEB1.ora into $ORACLE_HOME/dbs/ ( in node1)

3. Make adump directories on both node,
mkdir -p /u01/app/oracle/admin/TTWEB/adump (in both the nodes)

4. Edit /et/oratab (in both the node) ,
in node1
TTWEB1:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
in node2
TTWEB2:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
if you don't add 'TTWEB' in /etc/oratab file it will be automatically added by oraagent once
your instance are configured through srvctl command.

5.Set oraenv at node1 and start rman duplication,
[oracle@dbnode1]:[] $ . oraenv
ORACLE_SID = [oracle] ? TTWEB1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@dbnode1]:[TTWEB1] $sqlplus / as sysdba
SQL>startup nomount;
[oracle@dbnode1]:[TTWEB1] $rman auxiliary /
DUPLICATE DATABASE TO TTWEB BACKUP LOCATION '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL';
Once the duplication finish , do the rest of the work as given below

6.Create spfile in one of the disk group,
SQL> create spfile='+ORADATA' from pfile;
File created.
SQL>
Check the location of the spfile created by
[oracle@dbnode1]:[TTWEB1] $ su - grid
Password:
[grid@dbnode1 ~]$. oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[grid@dbnode1 ~]$ asmcmd
ASMCMD> cd ORADATA
ASMCMD> ls
ASMCMD> cd TTWEB/
ASMCMD> ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
control1.ctl
spfileTTWEB.ora
ASMCMD> cd PARAMETERFILE/
ASMCMD> ls
spfile.1465.871230913
ASMCMD> ls -lt
Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE FEB 09 16:00:00 Y spfile.1465.871230913
ASMCMD> pwd

7.Make an entry in initTTWEB1.ora as,
spfile='+ORADATA/TTWEB/PARAMETERFILE/spfile.1465.871230913'
--- note down any of the spfile name from the above output
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 2315256712 bytes
Database Buffers 1946157056 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +ORADATA/ttweb/parameterfile/s
pfile.1465.871230913
SQL>

8. Add undo tablespace for second instance,
SQL> create undo tablespace UNDOTBS2 datafile '+ORADATA' size 100M;
Tablespace created.
SQL>

9. Add redo log groups for the second instance , for that we have to verify how redo logs are configured for first instance.
SQL> set lines 300
col member for a80
SELECT a.group#,b.thread#, a.member, b.bytes/1024/1024 FROM v$logfile a, v$log b WHERE a.group# = b.group#;
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
3 1 +ORADATA/ttweb/onlinelog/group_3.1460.871230663 50
3 1 +ORAFRA/ttweb/onlinelog/group_3.1775.871230663 50
2 1 +ORADATA/ttweb/onlinelog/group_2.1459.871230661 50
2 1 +ORAFRA/ttweb/onlinelog/group_2.9378.871230661 50
1 1 +ORADATA/ttweb/onlinelog/group_1.1458.871230661 50
1 1 +ORAFRA/ttweb/onlinelog/group_1.464.871230661 50
6 rows selected.
SQL>
Here we have 3 redo log groups having two members each , so we will configure redo groups 4,5 and 6 for the second instance.

10. Add redo log groups for the second instance,
alter database add logfile thread 2 group 4 ('+ORADATA','+ORAFRA') size 50m reuse;
alter database add logfile thread 2 group 5 ('+ORADATA','+ORAFRA') size 50m reuse;
alter database add logfile thread 2 group 6 ('+ORADATA','+ORAFRA') size 50m reuse;
SQL> SELECT a.group#,b.thread#, a.member, b.bytes/1024/1024 FROM v$logfile a, v$log b WHERE a.group# = b.group#;
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
3 1 +ORADATA/ttweb/onlinelog/group_3.1460.871230663 50
3 1 +ORAFRA/ttweb/onlinelog/group_3.1775.871230663 50
2 1 +ORADATA/ttweb/onlinelog/group_2.1459.871230661 50
2 1 +ORAFRA/ttweb/onlinelog/group_2.9378.871230661 50
1 1 +ORADATA/ttweb/onlinelog/group_1.1458.871230661 50
1 1 +ORAFRA/ttweb/onlinelog/group_1.464.871230661 50
4 2 +ORADATA/ttweb/onlinelog/group_4.1468.871234269 50
4 2 +ORAFRA/ttweb/onlinelog/group_4.1825.871234269 50
5 2 +ORADATA/ttweb/onlinelog/group_5.1469.871234297 50
5 2 +ORAFRA/ttweb/onlinelog/group_5.7489.871234297 50
6 2 +ORADATA/ttweb/onlinelog/group_6.1470.871234303 50
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
6 2 +ORAFRA/ttweb/onlinelog/group_6.8822.871234303 50
12 rows selected.
Now we have second thread of online redo logs in order to start instance 2 and we can enable thread 2.
SQL> alter database enable public thread 2;
Database altered.
SQL>

11. Set cluster_database to true,before starting the SRVCTL utility
cluster_database is a static parameter and it require to bounce the database
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean FALSE
cluster_database_instances integer 1
cluster_interconnects string
SQL>
SQL> alter system set cluster_database=true scope=spfile sid='*';
System altered.
SQL> shut immediate;
sDatabase closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 2399142792 bytes
Database Buffers 1862270976 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean TRUE
cluster_database_instances integer 2
cluster_interconnects string
SQL>

12. Copy initTTWEB1.ora into seconds nodes's $ORACLE_HOME/dbs directory
[oracle@dbnode1]:[TTWEB1] $ scp initTTWEB1.ora oracle@dbnode2:/u01/app/oracle/product/11.2.0.4/db_1/dbs
in node2 ,go to dbnode2:/u01/app/oracle/product/11.2.0.4/db_1/dbs and rename the init file.
$mv initTTWEB1.ora initTTWEB2.ora

13. Register the database instances with CRS framework using srvctl command:
[oracle@dbnode1]:[TTWEB1] $ cat /etc/oratab | grep TTWEB
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB1:/u01/app/oracle/product/11.2.0.4/db_1:N
[oracle@dbnode1]:[TTWEB1] $ srvctl add database -d TTWEB -o /u01/app/oracle/product/11.2.0.4/db_1
[oracle@dbnode1]:[TTWEB1] $ srvctl add instance -d TTWEB -i TTWEB1 -n dbnode1
[oracle@dbnode1]:[TTWEB1] $ srvctl add instance -d TTWEB -i TTWEB2 -n dbnode2
Now the instances are registered with CRS, use SRVCTL to stop and start the database.
[oracle@dbnode1]:[TTWEB1] $ srvctl status database -d TTWEB
Instance TTWEB1 is not running on node dbnode1
Instance TTWEB2 is not running on node dbnode2
At first time, srvctl won't detect the status of the instances even when our instances are up and running . so stop it and start it again throuhg srvctl command .
[oracle@dbnode1]:[TTWEB1] $ srvctl stop database -d TTWEB
[oracle@dbnode1]:[TTWEB1] $ srvctl start database -d TTWEB
[oracle@dbnode1]:[TTWEB1] $ srvctl status database -d TTWEB
Instance TTWEB1 is running on node dbnode1
Instance TTWEB2 is running on node dbnode2
[oracle@dbnode1]:[TTWEB1] $
Now check the pmon process at the node2 , you can see new instance TTWEB2 is running there
[oracle@dbnode2]:[] $ ps -ef | grep pmon | grep TT
oracle 15312 1 0 17:55 ? 00:00:00 ora_pmon_TTWEB2
[oracle@dbnode2]:[] $
You can also see TTWEB got automatically added in /etc/oratab in node2
[oracle@dbnode2]:[] $ cat /etc/oratab | grep TT
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N # line added by Agent
[oracle@dbnode2]:[] $
You have to add an entry for TTWEB2 if it not already added , once enviormnet are set check the status of the instances through node2.
[oracle@dbnode2]:[TTWEB2] $ srvctl status database -d TTWEB2
PRCD-1120 : The resource for database TTWEB2 could not be found.
PRCR-1001 : Resource ora.ttweb2.db does not exist
[oracle@dbnode2]:[TTWEB2] $
[oracle@dbnode2]:[TTWEB2] $ srvctl status database -d TTWEB
Instance TTWEB1 is running on node dbnode1
Instance TTWEB2 is running on node dbnode2
[oracle@dbnode2]:[TTWEB2] $

14. Finally crete one password file for each instance
[oracle@dbnode1]:[TTWEB1] $ orapwd file= $ORACLE_HOME/dbs/orapwTTWEB1 password=xxxxxxx
[oracle@dbnode2]:[TTWEB2] $ orapwd file= $ORACLE_HOME/dbs/orapwTTWEB2 password=xxxxxxx

Check the logs:-
It is strongly advised to check all the logs related to Cluster, Database and instances when you perform installation, system changes or patch updates…etc. Make sure to check the log files to see the unexpected issues, if any.
CRS_HOME/log/hostname/crsd/ – The log files for the CRS daemon
CRS_HOME/log/hostname/cssd/ – The log files for the CSS daemon
CRS_HOME/log/hostname/evmd/ – The log files for the EVM daemon
CRS_HOME/log/hostname/client/ – The log files for the Oracle Cluster Registry (OCR)
CRS_HOME/log/hostname/racg/ – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostname/racg – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostanme/alert.log – The alert.log for Clusterware issues.
Please note that the CRS_HOME is the directory in which the Oracle Clusterware software was installed and hostname is the name of the node.

---------------------------------------------------------------------------------------------------------------------------------------------------------

Solution for ORA-16191: Primary log shipping client not logged on standby in DataGuard 

If you change SYS password with ALTER USER SYS IDENTIFIED BY NEWPASSWORD on the primary database of a dataguard environment, Logs should applied on standby side and sychronization between primary and standby database is fine .But once you restart your standby database, primary side stops to transfer archivelogs to standby and you will see an error on primary database alertlog file like:
------------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191
------------------------------------------------------------------
check whether you are able to log into the standby from the primary using sqlplus or not 

sqlplus sys@STANDBY as sysdba  --- you can't 

This is because "If you issue the ALTER USER statement to change the password for SYS, both the password stored in the data dictionary and the password stored in the password file are updated." So your password file is updated in primary side but not in standby side.

In this situation set your password file in standby server with:

$orapwd file=orapw$ORACLE_SID password=***** entries=5 ignorecase=Y (don't forget to move/delete old one)
or you can copy the primay database password file and rename it at standby side 

Conclusion: If you're going to change your sys password in a dataguard environment you must set the password files with new password in both primary and standby servers.

Its advisable to follow the steps below when you are planning to change the SYS password in PRIMARY 

Step 1: Defer the remote archival destination (log_archive_dest_state_n) parameter in primary. In case of RAC defer LOG_ARCHIVE_DEST_STATE_n in all nodes.
Step 2: change the password in primary 
Step 3: Recreate or copy the password file in standby. In case of RAC shutdown all instance of standby before recreating the password file.
Step 4: Enable the remote archival destination (log_archive_dest_state_n) parameter in primary

pwd file enhancement in oracle 12c
Until Oracle Database 11g password file of Oracle Database and ASM instance had to be stored in regular filesystem of windows or unix. However, starting from Oracle Database 12c R1 Oracle Database and ASM instance password file can be stored in ASM storage. This gives a great advantage in terms of ease of management of password file in Real Application Clusters (RAC) environment. When using ASM, in earlier versions of RAC, password files had to be stored in individual servers where RAC instances are running. For RAC environments password file can be stored in shared ASM disk group. For this, the compatible.asm disk group attribute must be set to 12.1 or higher for the disk group where the password is to be stored. The SYSASM or SYSDBA privilege is required to manage the Oracle ASM and database password files.

Shell Script:- How to colorful your shell script in linux

Shell scripts commonly used ANSI escape codes for color output. Following table shows Numbers representing colors in Escape Sequences.

Attribute codes:
00=none 01=set to bold intensity 02=set to faint intensity 03=use italic font 04=underscore 05=slow blink 06=fast blink 07=reverse foreground/background colors 08=Set foreground color to background color
Text color codes:
30=black 31=red 32=green 33=yellow 34=blue 35=magenta 36=cyan 37=white
Background color codes:
40=black 41=red 42=green 43=yellow 44=blue 45=magenta 46=cyan 47=white
Numbers Representing Colors in Escape Sequences

Color	Foreground	Background
Black	30	40
Red	31	41
Green	32	42
Yellow	33	43
Blue	34	44
Magenta	35	45
Cyan	36	46
White	37	47

The numbers in the above table work for xterm terminal.Result may vary for other terminal emulators.
Use the following template for writing colored text.
$ echo -e "\033[COLORm Sample text"
The "\033[" begins the escape sequence.You can also use "\e[" instead of "\033[". COLOR specifies a foreground color, according to the table above.The "m" terminates escape sequence, and text begins immediately after that.
eg:
$ echo -e "\033[1mThis is bold text.\033[0m"
The “\033[“ represents an escape( also you can use “\E[“ or “\e[“ ), the "1" turns on the bold attribute, 
while the "0" switches it off. The "m" terminates each term of the escape sequence.

Note:
Always use \033[0m at the end of the line to turn off the colour attribute at the end of the line. With an echo, the -e option enables the escape sequences.

Use the following template for writing colored text on a colored background.
$ echo -e "\E[COLOR1;COLOR2mSome text goes here."
The "\E[" begins the escape sequence. The semicolon-separated numbers "COLOR1" and "COLOR2" specify a foreground and a background color, according to the table below. (The order of the numbers does not matter, since the foreground and background numbers fall in non-overlapping ranges.) The "m" terminates the escape sequence, and the text begins immediately after that. Note also that single quotes enclose the remainder of the command sequence following the echo -e.

Note:
With an echo, the -e option enables the escape sequences.You can also use printf instead of echo.
printf "\e[COLORm sample text\n"

To print Green text
echo -e "\033[32m Hello World"

or
printf "\e[32m Hello World"

The problem with above statement is that the blue color that starts with the 32 color code is never switched back to the regular color, so any text you type after the prompt and even prompt also is still in the Green color.To return to the plain, normal mode, we have yet another sequence. The "\033[0m" ( so you have to add these entry whenever you required)

Note: Blink attribute doesn't work in any terminal emulator, but it will work on the console.
Combining all these Escape Sequences, you can get more fancy effect.

Use the following template for writing colored text on a colored background.
echo -e "\033[COLOR1;COLOR2m sample text\033[0m";

The semicolon separated numbers "COLOR1" and "COLOR2" specify a foreground and a background color. The order of the numbers does not matter, since the foreground and background numbers fall in non
- overlapping ranges."m" terminates the escape sequence, and the text begins immediately after that.
Although setting the colors separately also work (i.e. \033[44m\033[32m).

There are some differences between colors when combining colors with bold text attribute.
The following table summarises these differences.

Bold OFF	Color	Bold On	Color
0;30	Balck	1;30	Dark Gray
0;31	Red	1;31	Dark Red
0;32	Green	1;32	Dark Green
0;33	Brown	1;33	Yellow
0;34	Blue	1;34	Dark Blue
0;35	Magenta	1;35	Dark Magenta
0;36	Cyan	1;30	Dark Cyan
0;37	Light Gray	1;30	White

The following shell script prints all the colors and codes on the screen.
#!/bin/bash
# This script echoes colors and codes
echo -e "\n\033[4;31mLight Colors\033[0m \t\t\t \033[1;4;31mDark Colors\033[0m"
echo -e " \e[0;30;47m Black \e[0m 0;30m \t\t \e[1;30;40m Dark Gray \e[0m 1;30m"
echo -e " \e[0;31;47m Red \e[0m 0;31m \t\t \e[1;31;40m Dark Red \e[0m 1;31m"
echo -e " \e[0;32;47m Green \e[0m 0;32m \t\t \e[1;32;40m Dark Green \e[0m 1;32m"
echo -e " \e[0;33;47m Brown \e[0m 0;33m \t\t \e[1;33;40m Yellow \e[0m 1;33m"
echo -e " \e[0;34;47m Blue \e[0m 0;34m \t\t \e[1;34;40m Dark Blue \e[0m 1;34m"
echo -e " \e[0;35;47m Magenta \e[0m 0;35m \t\t \e[1;35;40m Dark Magenta\e[0m 1;35m"
echo -e " \e[0;36;47m Cyan \e[0m 0;36m \t\t \e[1;36;40m Dark Cyan \e[0m 1;36m"
echo -e " \e[0;37;47m Light Gray\e[0m 0;37m \t\t \e[1;37;40m White \e[0m 1;37m"

----------------------------------------end of script---------------------------------------------

Run the following shellscript on a console and also in a terminal and see the difference.
#!/bin/bash
clear
echo -e " \033[30m* 30 black forground *\033[0m"
echo -e " \033[31m* 31 red forground *\033[0m"
echo -e " \033[32m* 32 green forground *\033[0m"
echo -e " \033[33m* 33 yellow forground *\033[0m"
echo -e " \033[34m* 34 blue forground *\033[0m"
echo -e " \033[35m* 35 magenta forground *\033[0m"
echo -e " \033[36m* 36 cyan forground *\033[0m"
echo -e " \033[37m* 37 white forground *\033[0m"

echo -e "\033[33;40m 33;40 yellow text on black background\033[0m"
echo -e "\033[33;41m 33;41 yellow text on red background\033[0m"
echo -e "\033[33;42m 33;42 yellow text on green background\033[0m"
echo -e "\033[33;44m 33;44 yellow text on blue background\033[0m"
echo “ Note that 33 will display as brown in console and as yello in terminal”
echo -e "\033[33;45m 33;45 yellow text on magenta background\033[0m"
echo -e "\033[33;46m 33;46 yellow text on cyan background\033[0m"
echo -e "\033[33;47m 33;47 yellow text on white background\033[0m"

echo -e " \033[35;5m * 35 magenta text with slow blink*\033[0m"
echo -e "\033[1;4;33;44m 1;4;33;44 Bold yellow underlined text on blue background\033[0m"

Thus we can conclude
Colors:
\033[30m set foreground color to black
\033[31m set foreground color to red
\033[32m set foreground color to green
\033[33m set foreground color to yellow
\033[34m set foreground color to blue
\033[35m set foreground color to magenta (purple)
\033[36m set foreground color to cyan
\033[37m set foreground color to white
\033[40m set background color to black

\033[41m set background color to red
\033[42m set background color to green
\033[43m set background color to yellow
\033[44m set background color to blue
\033[45m set background color to magenta (purple)
\033[46m set background color to cyan
\033[47m set background color to white

\033[1;30m set foreground color to dark gray
\033[1;31m set foreground color to light red
\033[1;32m set foreground color to light green
\033[1;33m set foreground color to yellow
\033[1;34m set foreground color to light blue
\033[1;35m set foreground color to light magenta (purple)
\033[1;36m set foreground color to light cyan
\033[1;37m set foreground color to white

\033[1;40m set background color to dark gray
\033[1;41m set background color to light red
\033[1;42m set background color to light green
\033[1;43m set background color to yellow
\033[1;44m set background color to light blue
\033[1;45m set background color to light magenta (purple)
\033[1;46m set background color to light cyan
\033[1;47m set background color to white

For other features:

\033[0m reset; clears all colors and styles (to white on black)
\033[1m bold on
\033[3m italics on
\033[4m underline on
\033[5m blink on
\033[7m reverse video on
\033[8m nondisplayed (invisible)
\033[x;yH moves cursor to line x, column y
\033[xA moves cursor up x lines
\033[xB moves cursor down x lines
\033[xC moves cursor right x spaces
\033[xD moves cursor left x spaces
\033[2J clear screen and home cursor

Concept of mirroring Diskgroup in ASM

Mirroring or Redundancy protects data integrity by storing copies of data on multiple disks. To achieve mirroring ,Oracle introduce concept of Diskgroup in Automatic Storage Management.
Oracle mirrors at the extent level, so you have a primary extent and a mirrored extent.When a disk fails, ASM rebuilds the failed disk using mirrored extents from the other disks within the group, this may have a slight impact on performance as the rebuild takes place.
A diskgorup is a group of fail groups together, a fail group is a group of disks together. So, disk is the minimum identity in this architecture.The disk group type determines the mirroring levels with which Oracle creates files in a disk group.In general, ASM supports three types of redundancy (mirroring*) options.When you create a disk group, you specify an Automatic Storage Management (ASM) disk group type based on one of the following three redundancy levels.The default mirroring levels indicate the mirroring level with which each file is created unless a different mirroring level is designated.

Mirroring Options for ASM Disk Group Types

Disk Group Type	Supported Mirroring Levels	Default Mirroring Level
External redundancy	Unprotected (none)	Unprotected
Normal redundancy	Two-way Three-way Unprotected (None)	Two-way
High redundancy	Three-way	Three-way

External redundancy - doesn't have failure groups and thus is effectively a no-mirroring strategy, which is used when mirroring is provided by the disk subsystem such as RAID.Any write error cause a forced dismount of the disk group. 

Normal redundancy - provides two-way mirroring of all extents in a disk group, which result in two failure groups .A loss of one ASM disk is tolerated.

High redundancy - provides three-way mirroring of all extents in a disk group, which result in three failure groups.A loss of two ASM disks in different failure groups is tolerated.
 
The redundancy level controls how many disk failures are tolerated without dismounting the diskgroup or losing data.There are always failure groups even if they are not explicitly created. If you do not specify a failure group for a disk, then Oracle automatically creates a new failure group containing just that disk. A normal redundancy disk group must contain at least two failure groups. A high redundancy disk group must contain at least three failure groups. However, Oracle recommends using several failure groups.

Below is an example of disk group,
Let's assume, we have two disk controllers at Hardware level .Controller 1 (have four disk connected from diska), Controller 2 (have four disk connected from diskb). If, controller will fail, all disks under it will be unavailable. So, we need to plan our Disk group to take case this as well.

Controller 1:
/devices/diska1
/devices/diska2
/devices/diska3
/devices/diska4

Controller 2:
/devices/diskb1
/devices/diskb2
/devices/diskb3
/devices/diskb4

creates a disk group named dgroup1 with normal redundancy consisting of two failure groups controller1 or controller2 with four disks in each failure group.

$SQLPLUS /NOLOG
SQL> CONNECT / AS SYSASM
Connected to an idle instance.
SQL> STARTUP NOMOUNT

SQL>CREATE DISKGROUP dgroup1 NORMAL REDUNDANCY
FAILGROUP controller1 DISK
'/devices/diska1' NAME diska1,
'/devices/diska2' NAME diska2,
'/devices/diska3' NAME diska3,
'/devices/diska4' NAME diska4

FAILGROUP controller2 DISK
'/devices/diskb1' NAME diskb1,
'/devices/diskb2' NAME diskb2,
'/devices/diskb3' NAME diskb3,
'/devices/diskb4' NAME diskb4

ATTRIBUTE 'au_size'='4M',
 'compatible.asm' = '11.1', 
 'compatible.rdbms' = '11.1';

In the above example,  We made a disk group dgroup1 with normal redundancy (2-way mirroring),  So, we need two fail groups controller1 and controller2. 

When Automatic Storage Management allocates an extent for a normal redundancy file, ASM allocates a primary copy and a secondary copy. Automatic Storage Management chooses the disk on which to store the secondary copy in a different failure group other than the primary copy. Failure groups are used to place mirrored copies of data so that each copy is on a disk in a different failure group. The simultaneous failure of all disks in a failure group does not result in data loss. 

Failure Scenarios:
1. Single Disk failure: Suppose a disk failed in controller2 say(controller2.diskb4) data will remain intact, This disk either will have primary copy or secondary copy of extents. Since, we have two copies, so any how we will have another copy of failed disk into another disk group. So, I would say in this scenario, your data is 100% safe.

2. Multiple Disk Failure: This situation can further divide into two parts Multiple disk failure from Same disk group:  In this case, you will not loos data, because another disk group will have all data. Multiple disk failure from different disk group: Here is the risk, Suppose you loose controller1.diska1 and controller2.diskb4 and unfortunately, extents in diska1 have their mirror copy into diskb4 and both disks are unavailable. So, you will end up loosing data.

3. Disk Controller Failure: Suppose you loose one of your disk controller Controller 1 that means all disks in a fail group. Even, in this case data will not lost because, each extant in fail groupfailgroup1 will have a copy in failgroup2.

You define the failure groups for a disk group when you create an ASM disk group. After a disk group is created, you cannot alter the redundancy level of the disk group. To change the redundancy level of a disk group, create another disk group with the appropriate redundancy and then move the files to the new disk group. 

 Note:- Disk groups with external redundancy do not use failure groups.