Sunday, May 24, 2015

Data Guard Configuration for RAC through RMAN active database duplication.
In this topic I am going to cover how we can  configure dataguard in a RAC enviorment .
The main advantage of setting up dataguard in 11g/12c is that the database can be opened in Read-Only mode allowing the Users to access the physical standby database for fetching reports and on the same time the physical standby database can be in recovery mode. In other words, the physical standby database would be in recovery mode and in hand the standby database can be used for reporting purposes.

Assumptions
I have two servers (physical or VMs) with an operating system and Oracle installed on them. In this case I've used Oracle Linux 5.6 and Oracle Database 12.1.0.2.
The primary server has a running instance (TDDWH1 & TDDWH2)
The standby server has a software only installation.

Here is my enviornment setup .I am going to make a standby database for my cluster database TDDWH .
pdnode1 - node 1 primary 
pdnode2 - node 2 primary 
stnode1 - node 1 standby -10.81.155.146
stnode2 - node 2 standby -10.81.155.147

Primary Server Setup

Enable Archivelog Mode
Check that the primary database is in archivelog mode.

SQL>SELECT log_mode FROM v$database;

LOG_MODE
------------
NOARCHIVELOG

SQL>

If it is noarchivelog mode, switch is to archivelog mode.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup mount
ORACLE instance started.

Total System Global Area 1241513984 bytes
Fixed Size                  1273420 bytes
Variable Size             318767540 bytes
Database Buffers          905969664 bytes
Redo Buffers               15503360 bytes
Database mounted.

SQL> alter database archivelog;

Database altered.

SQL> ALTER SYSTEM SET LOG_ARCHIVE_FORMAT='%t_%s_%r.arc' SCOPE=SPFILE;

Where %tthread number , %s log sequence number and %r resetlogs ID that ensures unique names are constructed for the archived log files across multiple incarnations of the database

SQL> alter database open;

Database altered.

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     34
Next log sequence to archive   36
Current log sequence           36

Note:- 

Enabled forced logging by issuing the following command.
Any nologging operations performed on the primary database do not get fully logged within the redo stream. As Oracle Data Guard relies on the redo stream to maintain the standby database, this can result in data inconsistencies between the primary and standby along with a massive headache for the DBA to resolve. To prevent this from occurring, one solution is to place the primary database into force logging mode. In this mode, all nologging operations are permitted to run without error, however, the changes will be placed in the redo stream anyway.

SQL> ALTER DATABASE FORCE LOGGING;

To verify force logging is enabled for the database:


SQL> select force_logging from v$database;

FORCE_LOGGING
--------------
YES
Configure a Standby Redo Log
A Standby Redo log is added to enable Data Guard Maximum Availability and Maximum Protection modes. It is important to configure the Standby Redo Logs (SRL) with the same size/more than the size  of the online redo logs. Oracle recommented to create one more standby redo log file group than the number of online redo log file groups on the primary database. 


A best practice generally followed is to create the standby redo logs on both the primary and the standby database so as to make role transitions smoother. By creating the standby redo logs at this stage, it is assured that they will exist on both the primary and the newly created standby database. From the primary database, connect as SYS and run the following to create four standby redo log file groups:

SQL>select thread#, group# ,bytes/1024/1024 from v$log order by 1,2;
  THREAD#     GROUP# BYTES/1024/1024
---------- ---------- ---------------
         1          1              50
         1          2              50
         2          3             100
         2          4             100

Here thread 1 have two groups for each thread . So we need to create 3 standby redo log groups for thread 1 and 2.

Here thread 2 is configured with size 100MB , so we have to create standby redo logs of size 100 M for thread 2.

SQL> alter database add standby logfile thread 1 group 5 size 50M,group 6 size 50M,group 7  size 50M;

SQL> alter database add standby logfile thread 2 group 8 size 100M,group 9 size 100M,group 10 size 100M;

Check the standby redos by 

SQL> select GROUP#,TYPE from v$logfile ;
SQL> SELECT GROUP#,THREAD#,SEQUENCE#,ARCHIVED,STATUS FROM V$STANDBY_LOG;

Create a Password File
As part of the new redo transport security and authentication features, it is now mandatory that each database in an Oracle Data Guard configuration utilize a password file. In addition, the SYS password must be identical on every database in order for redo transport to function. If a password file does not exist for the primary database, create one using the following steps: create a new password file or copy the existing password file from node1 of primary and copy to the rest of the box.

[oracle@pdnode1]:[TDDWH1] $ scp orapwTDDWH1 pdnode2:/u01/app/oracle/product/12.1.0.2/db_1/dbs/orapwTDDWH2  -- node 2 of primary
[oracle@pdnode1]:[TDDWH1] $ scp orapwTDDWH1 10.81.155.146:/u01/app/oracle/product/12.1.0.2/db_1/dbs/orapwTGDWH1 -- node 1 of standby 
[oracle@pdnode1]:[TDDWH1] $ scp orapwTDDWH1 10.81.155.147:/u01/app/oracle/product/12.1.0.2/db_1/dbs/orapwTGDWH2 -- node 2 of standby 

After creating the password file, set the remote_login_passwordfile initialization parameter to EXCLUSIVE in the spfile on the primary database. Since this parameter cannot be dynamically modified for the current running instance, the change will have to be made to the spfile and bounced in order to take effect:

SQL> alter system set remote_login_passwordfile=exclusive scope=spfile;

System altered.

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> startup;
ORACLE instance started.
Total System Global Area 1241513984 bytes
Fixed Size                  1273420 bytes
Variable Size             318767540 bytes
Database Buffers          905969664 bytes
Redo Buffers               15503360 bytes
Database mounted.

Database opened.

Note:- 
If you have implemented any kind of  encryption  in your primary database you need to copy  those wallet directory to the standby boxes .If you forgot to copy your wallet directory the  MRP process won't come up in standby side.

[oracle@pdnode1]:[TDOSB1] $ scp -r TDDWH1  10.81.155.146:/etc/ORACLE/WALLETS/
Go to 10.81.155.146 and rename /etc/ORACLE/WALLETS/TDDWH1 to TGDWH1
oracle@pdnode1]:[TDOSB1] $ scp -r TDDWH1  10.81.155.147:/etc/ORACLE/WALLETS/
Go to 10.81.155.146 and rename /etc/ORACLE/WALLETS/TDDWH1 to TGDWH2

Configure the Primary Database Initialization Parameters
The configuration examples use the names shown in the following table:
DatabaseDB_UNIQUE_NAMEOracle Net Service Name
PrimaryTDDWHTDDWH
Physical standbyTGDWHTGDWH
Check the setting for the DB_NAME and DB_UNIQUE_NAME parameters. In this case they are both set to "TDDWH" on the primary database.

SQL> show parameter db_name

NAME      TYPE  VALUE
------------------------------------ ----------- ------------------------------
db_name       string  TDDWH

SQL> show parameter db_unique_name

NAME      TYPE  VALUE
------------------------------------ ----------- ------------------------------
db_unique_name      string  TDDWH

The DB_NAME of the standby database will be the same as that of the primary, but it must have a different DB_UNIQUE_NAME value. The DB_UNIQUE_NAME values of the primary and standby database should be used in the DG_CONFIG setting of the LOG_ARCHIVE_CONFIG parameter. For this example, the standby database will have the value "TGDWH".

Set Standby related parameters as follows ,
SQL> alter system set log_archive_dest_1='location=+ORADATA/TDDWH/ARCHIVELOG valid_for=(all_logfiles,all_roles) db_unique_name=TDDWH' sid='*' scope=both;

SQL> alter system set log_archive_dest_2='service=TGDWH NOAFFIRM LGWR ASYNC COMPRESSION=ENABLE VALID_FOR=(ALL_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=TGDWH' sid='*' scope=both;

 SQL>alter system set LOG_ARCHIVE_DEST_state_1=ENABLE sid='*' scope=both;
SQL>alter system set LOG_ARCHIVE_DEST_state_2=ENABLE sid='*' scope=both;

SQL>alter system set log_archive_config='DG_CONFIG=(TDDWH,TGDWH)' sid='*' scope=both;

Here TDDWH & TGDWH are db_unique_name for primary and standby (not oracle-net entry )

Set FAL_CLIENT and FAL_SERVER Parameters
Under certain circumstances, either because of network failures or a busy source environment, a gap builds up between the target sending the information and the destination receiving the changes and applying them. Since the MRP/LSP process has no direct communication link with the primary database to obtain a copy of the missing archive files, such gaps or archive gaps are resolved by the fetch archive log (FAL) client and server,identified by the initialization parameters FAL_CLIENT and FAL_SERVER.

FAL_SERVER specifies the FAL (fetch archive log) server for a standby database.The value is an Oracle Net service name, which is assumed to be configured properly on the standby database system to point to the desired FAL server.

FAL_CLIENT specifies the FAL (fetch archive log) client name that is used by the FAL service, configured through the FAL_SERVER parameter, to refer to the FAL client. The value is an Oracle Net service name, which is assumed to be configured properly on the FAL server system to point to the FAL client (standby database). Given the dependency of FAL_CLIENT on FAL_SERVER, the two parameters should be configured or changed at the same time.

FAL_CLIENT and FAL_SERVER are initialization parameters used to configure log gap detection and resolution at the standby database side of a physical database configuration. This functionality is provided by log apply services and is used by the physical standby database to manage the detection and resolution of archived redo logs.


FAL_CLIENT and FAL_SERVER only need to be defined in the initialization parameter file for the standby database(s). It is possible; however, to define these two parameters in the initialization parameter for the primary database server to ease the amount of work that would need to be performed if the primary database were required to transition its role.

Create oracle-net entry for TGDWH and also for TDDWH  on both nodes

SQL> alter system set fal_client='TDDWH' sid='*' scope=both;

SQL> alter system set fal_server='TGDWH' sid='*' scope=both;

Make sure tns entry for both TDDWH and TGDWH are present in all 4 boxes ie, you should able to tnsping TDDWH and TGDWH from all 4 boxes

The initialization parameter, STANDBY_FILE_MANAGEMENT, enables you to control whether or not adding a datafile to the primary database is automatically propagated to the standby database, as follows:

SQL>ALTER SYSTEM SET STANDBY_FILE_MANAGEMENT=AUTO;

Duplicating stanby databsae from active database 

Make an entry UR=A for the auxiliary instance both in standby and primary ,otherwise make a static listener entry for the standby database 

From node 1 of standby

[oracle@stnode1]:[TGDWH1] $ tnsping TGDWH

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 28-APR-2015 17:40:53

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.1.0.2/db_1/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ptgracscan)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TGDWH)(UR=A)))
OK (0 msec)
[oracle@stnode1]:[TGDWH1] $

From node 1 of primary

[oracle@pdnode1]:[TDDWH1] $ tnsping TGDWH

TNS Ping Utility for Linux: Version 12.1.0.2.0 - Production on 28-APR-2015 17:41:40

Copyright (c) 1997, 2014, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/product/12.1.0.2/db_1/network/admin/sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.81.155.146)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = TGDWH)(UR=A)))
OK (0 msec)
[oracle@pdnode1]:[TDDWH1] $

Make a proper initTGDWH1.ora and put it in the dbs directory of 10.81.155.146
the important perameter to consider 
*.db_unique_name='TGDWH'
*.db_name='TDDWH'
*.compatible='same as that of primary'
*.cluster_database=FALSE
*.db_create_file_dest='+ORADATA'
*.db_recovery_file_dest='+ORAFRA'
*.db_recovery_file_dest_size=4000M
*.control_files='+ORADATA/tgdwh/controlfile/stanby_control1.ctl','+ORAFRA/tgdwh/controlfile/standby_control2.ctl'
*.fal_server='TDDWH'
*.fal_client='TGDWH'
*.log_archive_config='DG_CONFIG=(TDDWH, TGDWH)'
*.log_archive_dest_1='location=use_db_recovery_file_dest valid_for=(all_logfiles,all_roles) db_unique_name=TGDWH'
*.log_archive_dest_2='service=TDDWH NOAFFIRM LGWR ASYNC COMPRESSION=ENABLE valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=TDDWH'
*.log_archive_dest_state_2='defer'
*.STANDBY_FILE_MANAGEMENT=AUTO

Note:- if you are using use_db_recovery_file_dest , you have to define db_recovery_file_dest . You don't have to worry about other parameter like adump , as oracle will automatically create while doing duplicaste

SQL>startup noumount pfile='initTGDWH1.ora';
 



====> duplicate the standby database by 
[oracle@stnode1]:[TGDWH1] $rman TARGET sys/Orcl1234@TDDWH AUXILIARY sys/Orcl1234@TGDWH
run {
allocate channel prmy1 type disk;
allocate channel prmy2 type disk;
allocate auxiliary channel stby type disk;
duplicate target database for standby from active database dorecover;
}

Once duplicate database finished its woks , start redo apply by

SQL> alter database recover managed standby database using current logfile disconnect from session;
SQL> select  thread#, sequence# ,name,applied from v$archived_log order by sequence#

SQL> select process,status,sequence#,block#,blocks from v$managed_standby;

---- if rfs is not coming check LOG_ARCHIVE_DEST_state_2 is set to enable or not in primary.

Once mrp process start applying the redo records  check error from primary 

set linesize 500
col DESTINATION for a30
SELECT inst_id,DEST_ID , STATUS ,DESTINATION , ERROR FROM gV$ARCHIVE_DEST WHERE DEST_ID <4 order by 1,2;

Adding instances to cluster 
Once mrp start applying archives , we can configure the new database to cluster,

[oracle@stnode1]:[TGHTB1] $ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 29 18:22:36 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Advanced Analytics, Oracle Database Vault and Real Application Testing options

SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> 
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Advanced Analytics, Oracle Database Vault and Real Application Testing options
[oracle@stnode1]:[TGHTB1] $
[oracle@stnode1]:[TGHTB1] $ vi initTGHTB1.ora
[oracle@stnode1]:[TGHTB1] $ here I made changes cluster_database='TRUE'
[oracle@stnode1]:[TGHTB1] $ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Apr 29 18:26:33 2015

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup  nomount;
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1610612736 bytes
Fixed Size                  2924928 bytes
Variable Size             989859456 bytes
Database Buffers          603979776 bytes
Redo Buffers               13848576 bytes
SQL>
SQL> create spfile='+ORAFRA' from pfile;

File created.

------------------------> note down the location of spfile created .
SQL> shut immediate;
ORA-01507: database not mounted
ORACLE instance shut down.
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Advanced Analytics
and Real Application Testing options
[oracle@stnode1]:[TGHTB1] $ cp initTGHTB1.ora initTGHTB1.ora_old
[oracle@stnode1]:[TGHTB1] $ vi initTGHTB1.ora
[oracle@stnode1]:[TGHTB1] $ here I added spfile='+ORAFRA/TGHTB/PARAMETERFILE/spfile.11344.878322445'
[oracle@stnode1]:[TGHTB1] $ pwd
/u01/app/oracle/product/12.1.0.2/db_1/dbs
[oracle@stnode1]:[TGHTB1] $ scp initTGHTB1.ora stnode2:/u01/app/oracle/product/12.1.0.2/db_1/dbs/initTGHTB2.ora
initTGHTB1.ora                                                                                                             100%   60     0.1KB/s   00:00
[oracle@stnode1]:[TGHTB1] $
[oracle@stnode1]:[TGHTB1] $ cat /etc/oratab | grep TGHTB1
TGHTB1:/u01/app/oracle/product/12.1.0.2/db_1:N          # line added by Agent
[oracle@stnode1]:[TGHTB1] $ srvctl add database -d TGHTB -o /u01/app/oracle/product/12.1.0.2/db_1
[oracle@stnode1]:[TGHTB1] $ srvctl add instance -d TGHTB -i TGHTB1 -n stnode1
[oracle@stnode1]:[TGHTB1] $ srvctl add instance -d TGHTB -i TGHTB2 -n stnode2
[oracle@stnode1]:[TGHTB1] $ srvctl status database -d TGHTB
Instance TGHTB1 is not running on node stnode1
Instance TGHTB2 is not running on node stnode2
[oracle@stnode1]:[TGHTB1] $ srvctl start database -d TGHTB
[oracle@stnode1]:[TGHTB1] $ srvctl status database -d TGHTB
Instance TGHTB1 is running on node stnode1
Instance TGHTB2 is running on node stnode2
[oracle@stnode1]:[TGHTB1] $

Sunday, May 17, 2015

How to approach RMAN error  "UNTIL CHANGE is before RESETLOGS change"

A problem simulation and solution approach :-
First take your full database backup including archivelogs of your database,

RMAN> backup database plus archivelog;
Starting backup at 05-FEB-15
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=8 RECID=39 STAMP=870706643
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\11PUH46H_1_1 tag=TAG20150205T145721 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=1 RECID=40 STAMP=870879410
input archived log thread=1 sequence=2 RECID=41 STAMP=870879441
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\12PUH46I_1_1 tag=TAG20150205T145721 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:03
Finished backup at 05-FEB-15
Starting backup at 05-FEB-15
using channel ORA_DISK_1
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=C:\ORADATA\DB11\SYSTEM01.DBF
input datafile file number=00002 name=C:\ORADATA\DB11\SYSAUX01.DBF
input datafile file number=00003 name=C:\ORADATA\DB11\UNDOTBS01.DBF
input datafile file number=00004 name=C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:01:25
Finished backup at 05-FEB-15
Starting backup at 05-FEB-15
current log archived
using channel ORA_DISK_1
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=3 RECID=42 STAMP=870879532
channel ORA_DISK_1: starting piece 1 at 05-FEB-15
channel ORA_DISK_1: finished piece 1 at 05-FEB-15
piece handle=C:\RMAN\14PUH49C_1_1 tag=TAG20150205T145852 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 05-FEB-15
Starting Control File and SPFILE Autobackup at 05-FEB-15
piece handle=C:\RMAN\C-1400701463-20150205-00 comment=NONE
Finished Control File and SPFILE Autobackup at 05-FEB-15
RMAN> exit
Recovery Manager complete.

Create some objects and do some action(DML and DDL) and note down the respective SCN after every statement.
C:\Users\mahekarthya>sqlplus sys as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 5 15:04:36 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Enter password:
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL>
SQL> create
2 table test
3 as select * from
4 all_objects;
Table created.
SQL> select
2 current_scn
3 from
4 v$database;
CURRENT_SCN
-----------
1045503 -------------------- SCN Before delete
SQL> delete from test;
68604 rows deleted.
SQL> commit;
Commit complete.
SQL>
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
1048517 --------------------- SCN after delete
SQL> drop table test;
Table dropped.
SQL>
SQL> select current_scn from v$database;
CURRENT_SCN
-----------
1048584
SQL>
SQL> alter system switch logfile;
System altered.
SQL>
SQL> select incarnation#, resetlogs_change#,status from v$database_incarnation;
INCARNATION# RESETLOGS_CHANGE# STATUS
------------ ----------------- -------
1 1 PARENT
2 834328 PARENT
3 838675 PARENT
4 1026219 PARENT
5 1028060 CURRENT --------- 1028060 is the CURRENT RESETLOGS_CHANGE#
SQL>
shutdown immediate
SQL>
C:\Users\mahekarthya>rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Feb 5 15:26:01 2015
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> run
2> {
3> startup mount;
4> set until scn=1048517; -----recovering to a point after delete statement is fired.
5> restore database;
6> recover database;
7> }
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
executing command: SET until clause
Starting restore at 05-FEB-15
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to C:\ORADATA\DB11\SYSTEM01.DBF
channel ORA_DISK_1: restoring datafile 00002 to C:\ORADATA\DB11\SYSAUX01.DBF
channel ORA_DISK_1: restoring datafile 00003 to C:\ORADATA\DB11\UNDOTBS01.DBF
channel ORA_DISK_1: restoring datafile 00004 to C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: reading from backup piece C:\RMAN\13PUH46M_1_1
channel ORA_DISK_1: piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:45
Finished restore at 05-FEB-15
Starting recover at 05-FEB-15
using channel ORA_DISK_1
starting media recovery
media recovery complete, elapsed time: 00:00:03
Finished recover at 05-FEB-15
RMAN> alter database open resetlogs;
database opened
RMAN>
As we opened the database using the resetlogs option, a new incarnation has been created. Below, is the new incarnation and the recovered table:
SQL> select incarnation#, resetlogs_change#,status from v$database_incarnation;
INCARNATION# RESETLOGS_CHANGE# STATUS
------------ ----------------- -------
1 1 PARENT
2 834328 PARENT
3 838675 PARENT
4 1026219 PARENT
5 1028060 PARENT --------- 1028060 became PARENT
6 1048518 CURRENT --------- & 1048518 became CURRENT(one SCN higher than what we provided in until SCN ,ie 1048517+1
6 rows selected.
SQL> select * from test;
no rows selected
SQL>
After this operation, I tried to recover the database up to the delete command(ie state before delete command has been fired against the table) issued as follows:
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>
C:\Users\mahekarthya>rman target /
Recovery Manager: Release 11.2.0.1.0 - Production on Thu Feb 5 15:39:23 2015
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database (not started)
RMAN> run
2> {
3> startup mount;
4> set until scn=1045503;
5> restore database;
6> recover database;
7> }
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
executing command: SET until clause
Starting restore at 05-FEB-15
using target database control file instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 02/05/2015 15:40:47
RMAN-20208: UNTIL CHANGE is before RESETLOGS change
RMAN>

I got this error , because I am trying to recover my database to a SCN(1045503) ie lower than the CURRENT resetlog SCN(1048518 ).Now I checked the list of all the database incarnations from RMAN, and I resets the database to the previous incarnation and performs an incomplete recovery as follows:
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 DB11 1400701463 PARENT 1 13-JAN-15
2 2 DB11 1400701463 PARENT 834328 14-JAN-15
3 3 DB11 1400701463 PARENT 838675 14-JAN-15
4 4 DB11 1400701463 PARENT 1026219 03-FEB-15
5 5 DB11 1400701463 PARENT 1028060 04-FEB-15
6 6 DB11 1400701463 CURRENT 1048518 05-FEB-15
RMAN> shutdown immediate;
database dismounted
Oracle instance shut down
RMAN> startup mount
connected to target database (not started)
Oracle instance started
database mounted
Total System Global Area 626327552 bytes
Fixed Size 2178336 bytes
Variable Size 478151392 bytes
Database Buffers 138412032 bytes
Redo Buffers 7585792 bytes
RMAN> reset database to incarnation 5;
database reset to incarnation 5
RMAN> run
2> {
3> set until scn=1045503;
4> restore database;
5> recover database;
6> }
executing command: SET until clause
Starting restore at 05-FEB-15
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=133 device type=DISK
channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to C:\ORADATA\DB11\SYSTEM01.DBF
channel ORA_DISK_1: restoring datafile 00002 to C:\ORADATA\DB11\SYSAUX01.DBF
channel ORA_DISK_1: restoring datafile 00003 to C:\ORADATA\DB11\UNDOTBS01.DBF
channel ORA_DISK_1: restoring datafile 00004 to C:\ORADATA\DB11\USERS01.DBF
channel ORA_DISK_1: reading from backup piece C:\RMAN\13PUH46M_1_1
channel ORA_DISK_1: piece handle=C:\RMAN\13PUH46M_1_1 tag=TAG20150205T145726
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:01:05
Finished restore at 05-FEB-15
Starting recover at 05-FEB-15
using channel ORA_DISK_1
starting media recovery
archived log for thread 1 with sequence 3 is already on disk as file C:\ORADATA\DB11\ARCHIVE\LOG
archived log for thread 1 with sequence 4 is already on disk as file C:\ORADATA\DB11\ARCHIVE\LOG
archived log file name=C:\ORADATA\DB11\ARCHIVE\LOG1_3_870795195.ARC thread=1 sequence=3
archived log file name=C:\ORADATA\DB11\ARCHIVE\LOG1_4_870795195.ARC thread=1 sequence=4
media recovery complete, elapsed time: 00:00:02
Finished recover at 05-FEB-15
RMAN> alter database open resetlogs;
database opened
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
------- ------- -------- ---------------- --- ---------- ----------
1 1 DB11 1400701463 PARENT 1 13-JAN-15
2 2 DB11 1400701463 PARENT 834328 14-JAN-15
3 3 DB11 1400701463 PARENT 838675 14-JAN-15
4 4 DB11 1400701463 PARENT 1026219 03-FEB-15
5 5 DB11 1400701463 PARENT 1028060 04-FEB-15
7 7 DB11 1400701463 CURRENT 1045504 05-FEB-15
6 6 DB11 1400701463 ORPHAN 1048518 05-FEB-15
RMAN> exit
Recovery Manager complete.
C:\Users\mahekarthya>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Feb 5 15:48:08 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> select count(1) from test;
COUNT(1)
----------
68604
SQL>

The table recovered to the previous incarnation successfully and a new incarnation has been created.

Convert a single instance database to 2 node RAC cluster database.

Here I am going to explain how to convert a single instance database to a two node RAC database .Keep in mind that, even if your source database is in RAC , there is no direct method to convert it into a  clone RAC database . First we need to convert it into a single instance database , later we need to add the newly configured single instance to cluster.

1.First take the full database backup of the single instance database,

BACKUP AS COMPRESSED BACKUPSET FULL TAG 'DFWEBFULL' DATABASE FORMAT '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL/Backup_%d_%T_%s'
INCLUDE CURRENT CONTROLFILE PLUS ARCHIVELOG format '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL/ARCHIVE_%d_%T_%s';

Note:- change the permission of the backup directory so that oracle software owner can read those files , otherwise you wil receive error.
$cd /nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL
$chmod 777 *
Here I took my rman backups into a nfs mount point , and its accessible  from the auxiliary database so that no need to copy the entire backup set .

2. Take one init parameter from the Cluster ( if your cluster have already some instance configured) where you want to configure your RAC instances, otherwise make a new one .keep the locat_listener and remote_listener as it is . Change the other parameter accordingly .Here I am going to configure RAC database TTWEB having two instances TTWEB1 and TTWEB2. Here is my init parameter and is named as initTTWEB1.ora
Note:- Comment out *.cluster_database=TRUE
$vi initTTWEB1.ora
TTWEB2.__db_cache_size=956301312
TTWEB1.__db_cache_size=1610612736
TTWEB2.__java_pool_size=67108864
TTWEB1.__java_pool_size=67108864
TTWEB2.__large_pool_size=83886080
TTWEB1.__large_pool_size=83886080
TTWEB2.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
TTWEB1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
TTWEB2.__pga_aggregate_target=1493172224
TTWEB1.__pga_aggregate_target=1493172224
TTWEB2.__sga_target=2801795072
TTWEB1.__sga_target=2801795072
TTWEB2.__shared_io_pool_size=0
TTWEB1.__shared_io_pool_size=0
TTWEB2.__shared_pool_size=1660944384
TTWEB1.__shared_pool_size=1006632960
TTWEB2.__streams_pool_size=0
TTWEB1.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/TTWEB/adump'
*.audit_trail='db'
#*.cluster_database=TRUE
*.compatible='11.2.0.4'
*.control_files='+ORADATA/TTWEB/control1.ctl','+ORAFRA/TTWEB/control2.ctl'
*.db_block_size=8192
*.db_create_file_dest='+ORADATA'
*.db_create_online_log_dest_1='+ORADATA'
*.db_create_online_log_dest_2='+ORAFRA'
*.db_domain=''
*.db_name='TTWEB'
*.db_recovery_file_dest='+ORAFRA'
*.db_recovery_file_dest_size=107374182400
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=TTWEBXDB)'
TTWEB1.instance_number=1
TTWEB2.instance_number=2
TTWEB1.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.81.155.146)(PORT=1521))'
TTWEB2.local_listener='(ADDRESS=(PROTOCOL=TCP)(HOST=10.81.155.157)(PORT=1521))'
*.memory_target=4G
*.nls_language='AMERICAN'
*.open_cursors=300
*.pga_aggregate_target=536870912
*.processes=500
*.remote_listener='ptgracscan:1521'
*.remote_login_passwordfile='exclusive'
*.sga_target=1610612736
TTWEB1.thread=1
TTWEB2.thread=2
TTWEB1.undo_tablespace='UNDOTBS1'
TTWEB2.undo_tablespace='UNDOTBS2'
Copy initTTWEB1.ora into $ORACLE_HOME/dbs/ ( in node1)

3. Make adump directories on both node,
mkdir -p /u01/app/oracle/admin/TTWEB/adump (in both the nodes)

4. Edit /et/oratab (in both the node) ,
in node1
TTWEB1:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
in node2
TTWEB2:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
if you don't add 'TTWEB' in /etc/oratab file it will be automatically added by oraagent once
your instance are configured through srvctl command.

5.Set oraenv at node1 and start rman duplication,
[oracle@dbnode1]:[] $ . oraenv
ORACLE_SID = [oracle] ? TTWEB1
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@dbnode1]:[TTWEB1] $sqlplus / as sysdba
SQL>startup nomount;
[oracle@dbnode1]:[TTWEB1] $rman auxiliary /
DUPLICATE DATABASE TO TTWEB BACKUP LOCATION '/nfs_1/DB_ADHOC_BACKUPS/DFWEBFULL';
Once the duplication finish , do the rest of the work as given below

6.Create spfile in one of the disk group,
SQL> create spfile='+ORADATA' from pfile;
File created.
SQL>
Check the location of the spfile created by
[oracle@dbnode1]:[TTWEB1] $ su - grid
Password:
[grid@dbnode1 ~]$. oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /u01/app/oracle
[grid@dbnode1 ~]$ asmcmd
ASMCMD> cd ORADATA
ASMCMD> ls
ASMCMD> cd TTWEB/
ASMCMD> ls
CONTROLFILE/
DATAFILE/
ONLINELOG/
PARAMETERFILE/
TEMPFILE/
control1.ctl
spfileTTWEB.ora
ASMCMD> cd PARAMETERFILE/
ASMCMD> ls
spfile.1465.871230913
ASMCMD> ls -lt
Type Redund Striped Time Sys Name
PARAMETERFILE UNPROT COARSE FEB 09 16:00:00 Y spfile.1465.871230913
ASMCMD> pwd

7.Make an entry in initTTWEB1.ora as,
spfile='+ORADATA/TTWEB/PARAMETERFILE/spfile.1465.871230913'
--- note down any of the spfile name from the above output
SQL> shut immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 2315256712 bytes
Database Buffers 1946157056 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +ORADATA/ttweb/parameterfile/s
pfile.1465.871230913
SQL>

8. Add undo tablespace for second instance,
SQL> create undo tablespace UNDOTBS2 datafile '+ORADATA' size 100M;
Tablespace created.
SQL>

9. Add redo log groups for the second instance , for that we have to verify how redo logs are configured for first instance.
SQL> set lines 300
col member for a80
SELECT a.group#,b.thread#, a.member, b.bytes/1024/1024 FROM v$logfile a, v$log b WHERE a.group# = b.group#;
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
3 1 +ORADATA/ttweb/onlinelog/group_3.1460.871230663 50
3 1 +ORAFRA/ttweb/onlinelog/group_3.1775.871230663 50
2 1 +ORADATA/ttweb/onlinelog/group_2.1459.871230661 50
2 1 +ORAFRA/ttweb/onlinelog/group_2.9378.871230661 50
1 1 +ORADATA/ttweb/onlinelog/group_1.1458.871230661 50
1 1 +ORAFRA/ttweb/onlinelog/group_1.464.871230661 50
6 rows selected.
SQL>
Here we have 3 redo log groups having two members each , so we will configure redo groups 4,5 and 6 for the second instance.

10. Add redo log groups for the second instance,
alter database add logfile thread 2 group 4 ('+ORADATA','+ORAFRA') size 50m reuse;
alter database add logfile thread 2 group 5 ('+ORADATA','+ORAFRA') size 50m reuse;
alter database add logfile thread 2 group 6 ('+ORADATA','+ORAFRA') size 50m reuse;
SQL> SELECT a.group#,b.thread#, a.member, b.bytes/1024/1024 FROM v$logfile a, v$log b WHERE a.group# = b.group#;
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
3 1 +ORADATA/ttweb/onlinelog/group_3.1460.871230663 50
3 1 +ORAFRA/ttweb/onlinelog/group_3.1775.871230663 50
2 1 +ORADATA/ttweb/onlinelog/group_2.1459.871230661 50
2 1 +ORAFRA/ttweb/onlinelog/group_2.9378.871230661 50
1 1 +ORADATA/ttweb/onlinelog/group_1.1458.871230661 50
1 1 +ORAFRA/ttweb/onlinelog/group_1.464.871230661 50
4 2 +ORADATA/ttweb/onlinelog/group_4.1468.871234269 50
4 2 +ORAFRA/ttweb/onlinelog/group_4.1825.871234269 50
5 2 +ORADATA/ttweb/onlinelog/group_5.1469.871234297 50
5 2 +ORAFRA/ttweb/onlinelog/group_5.7489.871234297 50
6 2 +ORADATA/ttweb/onlinelog/group_6.1470.871234303 50
GROUP# THREAD# MEMBER B.BYTES/1024/1024
---------- ---------- -------------------------------------------------------------------------------- -----------------
6 2 +ORAFRA/ttweb/onlinelog/group_6.8822.871234303 50
12 rows selected.
Now we have second thread of online redo logs in order to start instance 2 and we can enable thread 2.
SQL> alter database enable public thread 2;
Database altered.
SQL>

11. Set cluster_database to true,before starting the SRVCTL utility
cluster_database is a static parameter and it require to bounce the database
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean FALSE
cluster_database_instances integer 1
cluster_interconnects string
SQL>
SQL> alter system set cluster_database=true scope=spfile sid='*';
System altered.
SQL> shut immediate;
sDatabase closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup;
ORACLE instance started.
Total System Global Area 4275781632 bytes
Fixed Size 2260088 bytes
Variable Size 2399142792 bytes
Database Buffers 1862270976 bytes
Redo Buffers 12107776 bytes
Database mounted.
Database opened.
SQL> show parameter cluster
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database boolean TRUE
cluster_database_instances integer 2
cluster_interconnects string
SQL>

12. Copy initTTWEB1.ora into seconds nodes's $ORACLE_HOME/dbs directory
[oracle@dbnode1]:[TTWEB1] $ scp initTTWEB1.ora oracle@dbnode2:/u01/app/oracle/product/11.2.0.4/db_1/dbs
in node2 ,go to dbnode2:/u01/app/oracle/product/11.2.0.4/db_1/dbs and rename the init file.
$mv initTTWEB1.ora initTTWEB2.ora

13. Register the database instances with CRS framework using srvctl command:
[oracle@dbnode1]:[TTWEB1] $ cat /etc/oratab | grep TTWEB
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N
TTWEB1:/u01/app/oracle/product/11.2.0.4/db_1:N
[oracle@dbnode1]:[TTWEB1] $ srvctl add database -d TTWEB -o /u01/app/oracle/product/11.2.0.4/db_1
[oracle@dbnode1]:[TTWEB1] $ srvctl add instance -d TTWEB -i TTWEB1 -n dbnode1
[oracle@dbnode1]:[TTWEB1] $ srvctl add instance -d TTWEB -i TTWEB2 -n dbnode2
Now the instances are registered with CRS, use SRVCTL to stop and start the database.
[oracle@dbnode1]:[TTWEB1] $ srvctl status database -d TTWEB
Instance TTWEB1 is not running on node dbnode1
Instance TTWEB2 is not running on node dbnode2
At first time, srvctl won't detect the status of the instances even when our instances are up and running . so stop it and start it again throuhg srvctl command .
[oracle@dbnode1]:[TTWEB1] $ srvctl stop database -d TTWEB
[oracle@dbnode1]:[TTWEB1] $ srvctl start database -d TTWEB
[oracle@dbnode1]:[TTWEB1] $ srvctl status database -d TTWEB
Instance TTWEB1 is running on node dbnode1
Instance TTWEB2 is running on node dbnode2
[oracle@dbnode1]:[TTWEB1] $
Now check the pmon process at the node2 , you can see new instance TTWEB2 is running there
[oracle@dbnode2]:[] $ ps -ef | grep pmon | grep TT
oracle 15312 1 0 17:55 ? 00:00:00 ora_pmon_TTWEB2
[oracle@dbnode2]:[] $
You can also see TTWEB got automatically added in /etc/oratab in node2
[oracle@dbnode2]:[] $ cat /etc/oratab | grep TT
TTWEB:/u01/app/oracle/product/11.2.0.4/db_1:N # line added by Agent
[oracle@dbnode2]:[] $
You have to add an entry for TTWEB2 if it not already added , once enviormnet are set check the status of the instances through node2.
[oracle@dbnode2]:[TTWEB2] $ srvctl status database -d TTWEB2
PRCD-1120 : The resource for database TTWEB2 could not be found.
PRCR-1001 : Resource ora.ttweb2.db does not exist
[oracle@dbnode2]:[TTWEB2] $
[oracle@dbnode2]:[TTWEB2] $ srvctl status database -d TTWEB
Instance TTWEB1 is running on node dbnode1
Instance TTWEB2 is running on node dbnode2
[oracle@dbnode2]:[TTWEB2] $

14. Finally crete one password file for each instance
[oracle@dbnode1]:[TTWEB1] $ orapwd file= $ORACLE_HOME/dbs/orapwTTWEB1 password=xxxxxxx
[oracle@dbnode2]:[TTWEB2] $ orapwd file= $ORACLE_HOME/dbs/orapwTTWEB2 password=xxxxxxx

Check the logs:-
It is strongly advised to check all the logs related to Cluster, Database and instances when you perform installation, system changes or patch updates…etc. Make sure to check the log files to see the unexpected issues, if any.
CRS_HOME/log/hostname/crsd/ – The log files for the CRS daemon
CRS_HOME/log/hostname/cssd/ – The log files for the CSS daemon
CRS_HOME/log/hostname/evmd/ – The log files for the EVM daemon
CRS_HOME/log/hostname/client/ – The log files for the Oracle Cluster Registry (OCR)
CRS_HOME/log/hostname/racg/ – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostname/racg – The log files for the Oracle RAC high availability component
CRS_HOME/log/hostanme/alert.log – The alert.log for Clusterware issues.
Please note that the CRS_HOME is the directory in which the Oracle Clusterware software was installed and hostname is the name of the node.

---------------------------------------------------------------------------------------------------------------------------------------------------------