what is histogram in oracle and its importance

DBMS_STATS by default calculates the number of rows in a table,The number of distinct values in a column etc.Without additional information, optimizer assumes that values are evenly distributed .But if they are not evenly distributed,  inefficient execution plans can result. A histogram holds the data distribution of values within a column of a table.Number of occurrences for a specific value/range .

Histogram is collected by using DBMS_STATS.Histograms are feature in CBO and it helps to optimizer to determine how data are skewed(distributed) with in the column. Histogram is good to create for the column which are included in the WHERE clause where the column is highly skewed. Histogram helps to optimizer to decide whether to use an index or full-table scan or help the optimizer to determine the fastest table join order.

When a query executes in database, Optimizer records the predicate usage in sys.col_usage$ (like number of equi joins, non equi joins, between operators & like operators). This recorded values in sys.col_usage$ will be used by dbms_stats to decide the need for histograms on the columns or *NOT*.

Types of Histograms

Histograms are classified into frequency histograms and height-balanced histograms .
Type of histogram is stored in the HISTOGRAM  column of the *tab_col_statistics views (USER/DBA) and its Value will be (‘HEIGHT BALANCED’, ‘FREQUENCY’,  or ‘NONE’).
For example,

SQL>select column_name, num_distinct, histogram FROM user_tab_col_statistics
WHERE table_name = 'SALES';

Histogram and Buckets
A bucket is a range of value for a particular column, for example if we had a column named true_or_false , and it could only have two values  true or false ,then we could have upto 2 buckets and these two buckets would have a value describing the number of times that where true and false.When Histograms are created the number of  buckets can be specified otherwise oracle will automatically chose the right choice.It is this number that controls the type of histogram created. 
When Distinct Values <= Buckets 
– Then Frequency Histogram is created
– Else Height-Balanced Histogram is created

One limitation of Oracle's frequency histograms is that it has a hard limit of 254 buckets.
If you have a small number of distinct values, say less than 100 and queries are frequently executed against these columms for specific values and column is skewed, then Frequency Histograms are your best option.If your number of distinct values is more than 254, Frequency Histograms are out of the question. You will have to use Height Balanced Histograms.

Frequency Histograms
Each value of the column corresponds to a single bucket of the histogram. Each bucket contains the number of occurrences of that single value.Frequency histograms are automatically created instead of height-balanced histograms when the number of distinct values is less than or equal to the number of histogram buckets specified. 

Height-Balanced Histograms
In a height-balanced histogram, the column values are divided into bands so that each 
band contains approximately the same number of rows. The column values are divided into bands so that each band contains approximately the same number of rows. For instances, we have 10 distinct values in the column and only five buckets. It will create height based(Height balanced) histograms and it will evenly spread values through the buckets. A height-based histogram is when there are more distinct values than the number of buckets and the histogram statistics shows a range of rows across the buckets .The useful information that the histogram  provides is where in the range of values the 
endpoints fall.

** EPN - end point number & EPV - end point value 

How to Create a Histogram

You have the following options to create histograms using Oracle:
ANALYZE TABLE 
As part of the ANALYZE TABLE command for compiling statistics, you can specify
FOR ALL [INDEXED] COLUMNS [SIZE <buckets>]
to create histograms (<buckets> > 1). If you do not specify the SIZE, 75 buckets are used. For example, with the following command, histograms are created with 75 buckets for all indexed columns of the T100 table:
SQL> ANALYZE TABLE T100 COMPUTE STATISTICS FOR ALL INDEXED COLUMNS;

DBMS_STATS
You can also specify the METHOD_OPTS argument as part of the DBMS_STATS command GATHER_*_STATS, the contents of this argument are syntactically identical to the FOR expression of ANALYZE TABLE.
Therefore, the following is comparable with the above ANALYZE TABLE example:
SQL>DBMS_STATS.GATHER_TABLE_STATS('SAPR3', 'T100',
 METHOD_OPT=>'FOR ALL INDEXED COLUMNS');

Method_opt Parameter: This is the parameter which tells about creating histogram while collecting the statistics. The default is FOR ALL COLUMNS SIZE AUTO in Oracle10g. But in oracle9i, the default is FOR ALL COLUMN SIZE 1 which will turn off the histogram collection. 

METHOD_OPT dictates values how histograms are done ,

• for all columns - gather histograms on all columns -- hugely expensive and rarely should be used
• for all indexed columns - gather histograms on all columns that are currently indexed. This is typically to be the "most useful" although you can arguably skip nearly unique columns such as primary keys
• for all hidden columns - gather histograms on columns you cannot see, usefulif using the object relational features which incorporate tons of hidden columnsin your objects.
• for columns <list> - gather histograms on these specific columns, could be useful to skip primary key indexes or just to get the 2 or 3 columns of interest

Following options are available fro size clause,

• integer : Number of histogram buckets. Must be in the range [1,254] (if you not specify the integer oracle will take its default 75)
• REPEAT : says to do the histograms just like you did them last time . It reads the data dictionary to figure our what to do.
• AUTO : Oracle determines the columns to collect histograms based on data distribution and the workload of the columns. We have a table called sys.col_usage$ that stores information about column usage. dbms_stats use this information to determine whether histogram is required for the columns.
• SKEWONLY : Oracle determines the columns to collect histograms based on the data distribution of the columns.

For example,

SQL> execute dbms_stats.gather_table_stats  (ownname => 'oe', tabname => 'inventories', 

METHOD_OPT => 'FOR COLUMNS SIZE 10 quantity_on_hand');

Demonstration for how optimizer  work with and without histogram
SQL> create table t nologging as
  2  select case when rownum = 1 then 5 else mod(rownum,4) end as id, a.* from all_objects a;

Table created.

SQL> select count(*),id from t group by id;

  COUNT(*)         ID
---------- ----------
     12508          1
     12508          2
        1              5
     12508          3
     12508          0

SQL> create index t_ind on t(id) nologging;

Index created.

SQL> begin

  2  DBMS_STATS.GATHER_TABLE_STATS(OWNNAME => 'HR', TABNAME =>     'T',ESTIMATE_PERCENT =>

  3  10, METHOD_OPT => 'FOR ALL COLUMNS SIZE 1',CASCADE => TRUE);

  4  end;

  5  /

PL/SQL procedure successfully completed.

Note:- Histogram collection is disabled by "FOR ALL COLUMNS SIZE 1"

SQL> set autotrace traceonly explain;

SQL> select * from t where id = 1;

Execution Plan

----------------------------------------------------------

Plan hash value: 1601196873

--------------------------------------------------------------------------

| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |

--------------------------------------------------------------------------

|   0 | SELECT STATEMENT  |      | 12455 |  1167K|   162   (1)| 00:00:02 |

|*  1 |  TABLE ACCESS FULL| T    | 12455 |  1167K|   162   (1)| 00:00:02 |

--------------------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------

   1 - filter("ID"=1)

SQL> select * from t where id = 5;

Execution Plan

----------------------------------------------------------

Plan hash value: 1601196873

--------------------------------------------------------------------------

| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |

--------------------------------------------------------------------------

|   0 | SELECT STATEMENT  |      |  4152 |   389K|   162   (1)| 00:00:02 |

|*  1 |  TABLE ACCESS FULL| T    |  4152 |   389K|   162   (1)| 00:00:02 |

--------------------------------------------------------------------------

Predicate Information (identified by operation id):

---------------------------------------------------

   1 - filter("ID"=5)

SQL>

Conclusion: Optimizer is using full table scan for the query which returns 12508  records as well as it using full table scan for query which returns just only one record. This is obviously incorrect. This problem will be resolved by collecting histogram. Let us see in the next scenario.

Scenario 2 : Let us generate the statistics with histogram and see what kind of execution path optimizer is using. FOR COLUMN SIZE 5 ID will create 5 bucket for column ID. If we are not sure the distinct number of values in the column, then we can use AUTO option to collect histogram. With this histogram, oracle optimizer knows that, the column ID is highly skewed and it has 5 bucket and 4 bucket has around  12508  records with values 0 to 3 and another bucket has only one record with ID equal to 5 . Now depends upon the query, optimizer decides whether to use index or Full table scan.

SQL> set autotrace off;
SQL> begin
  2  DBMS_STATS.GATHER_TABLE_STATS(OWNNAME => 'HR', TABNAME => 'T',ESTIMATE_PERCENT =>
  3  10, METHOD_OPT => 'FOR COLUMNS SIZE 5 ID',CASCADE => TRUE);
  4  end;
  5  /

PL/SQL procedure successfully completed.

SQL> set autotrace traceonly explain;
SQL> select * from t where id = 1;

Execution Plan
----------------------------------------------------------
Plan hash value: 1601196873

--------------------------------------------------------------------------
| Id  | Operation         | Name | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |      | 12150 |  1139K|   162   (1)| 00:00:02 |
|*  1 |  TABLE ACCESS FULL| T    | 12150 |  1139K|   162   (1)| 00:00:02 |
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("ID"=1)

SQL> select * from t where id = 5;

Execution Plan
----------------------------------------------------------
Plan hash value: 1376202287

-------------------------------------------------------------------------------------
| Id  | Operation                   | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT            |       |  2025 |   189K|   119   (0)| 00:00:02 |
|   1 |  TABLE ACCESS BY INDEX ROWID| T     |  2025 |   189K|   119   (0)| 00:00:02 |
|*  2 |   INDEX RANGE SCAN          | T_IND |  2025 |       |     4   (0)| 00:00:01 |
-------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("ID"=5)

SQL>

Conclusion: Optimizer is using full table scan for the query which returns 12508   records. At the same time, optimizer is using index scan when for other query which returns one record. This scenario, the optimizer choose the right execution plan based on the query WHERE clause.

Rules On When To Create Histograms

• First there are no rules of thumb that are always correct.
• When creating using dbms_stats, the use of the DEGREE (for parallel) can skew the value of density
• Running dbms_stats can drop histograms ,so Always check if histogram exists on table before DBMS_STATS is run.Use METHOD_OPT FOR ALL COLUMNS REPEAT to prevent deletion of histograms  data.
• Histograms are stored in the dictionary.
• There is a maintenance and space cost for using histograms.
• Only create histograms on columns that are used in WHERE clauses of queries and have a highly-skewed data distribution.Method_opt “size auto” is supposed to do just that but does not always get the second part right.

When Not To Use Histograms

• all predicates on the column use bind variables 
• the column data is uniformly distributed 
• the column is not used in WHERE clauses of queries 
• the column is unique and is used only with equality predicates
• If your database exclusively uses bind variables, Oracle recommends deleting any existing Oracle histograms and disabling Oracle histogram generation (method opt) for any future dbms_stats analysis.  This approach will use the number if distinct values to determine the selectivity of a column.

Reference :- http://rajeshwaranbtech.blogspot.in &  http://myorastuff.blogspot.in 

Automatic Storage Management (ASM) lessons 4

Using ASM Storage
We have discussed management of an ASM instance. This section covers how to actually
use ASM from an Oracle instance. You can put all sorts of Oracle-related files into an
ASM instance, including these:

• Oracle datafiles
• Database tempfiles
• Online redo logs
• Archived redo logs
• Control files
• Spfiles
• RMAN backup sets
• The flash recovery area (FRA)
• Data-pump dump sets

What Are ASM Files?

We have already created ASM disk groups. To actually use the ASM disk groups, we have

to populate them with ASM files. In this section, we will discuss what ASM files are and

then we will discuss the different kinds of ASM filenames that you might deal with.

ASM Files

ASM files are created in a number of different ways; for example, when you execute the

create tablespace command and you indicate that the resulting datafile(s) should be

stored in an ASM disk group, the result will be the creation of ASM files in that ASM

disk group.A goodly number of Oracle file types can be stored in ASM, including datafiles, controlfiles, redo logs, and archived redo logs. There are some Oracle files that cannot be stored inan ASM group. These are mostly the administrative files like trace files, the alert log, andso on.

ASM Filename Types

When a file is created on an ASM disk, the filename is generated by ASM. There is a number of different kinds of ASM filename types:

• Fully qualified ASM filenames
• Numeric ASM filenames
• Alias ASM filenames
• Alias filenames with templates
• Incomplete filenames
• Incomplete filenames with templates .

For example,

• 1. Fully Qualified ASM Filename: +group/dbname/file_type/file_type_tag.file.incarnation
  

  Example: +dgroup2/sample/controlfile/Current.256.541956473

• 2. Numeric ASM Filename: +group.file.incarnation

  Example: +dgroup2.257.541956473

• 3. Alias ASM Filenames: +group/dir_1/…/dir_n/filename
  

  Example: +dgroup1/myfiles/control_file1 +dgroup2/mydir/second.dbf

• 4. Alias ASM Filename with Template: +group(template_name)/alias
  

  Example: +dgroup1(my_template)/config1

• 5. Incomplete ASM Filename:+group
  

  Example: +dgroup1

• 6. Incomplete ASM Filename with Template: +group(template_name)
  

  Example: +dgroup1(my_template)

Note:- Incomplete ASM file names are most  commonly used by DBA.  Incomplete ASM file names are used only for file creation operations. They consist of a disk group name only. ASM will then use the appropriate default template to translate the incomplete ASM file name, as defined by its file type. For example, here is the SQL command I executed originally to create the TBSASM tablespace in the ASM1DG1 disk group:

SQL> CREATE TABLESPACE tbsasm DATAFILE '+ASM1DG1' SIZE 32M;

Defining ASM as the Default Destination for Database Files

If you decide you want to allow Oracle to create all file types as ASM file types, you can set the values of various parameters such that ASM will automatically be employed. One of the big benefits of this feature is the standardization of your database, ensuring that all files get placed where they belong and in the ASM structure to which they belong. You can define default ASM destinations be defining incomplete ASM filenames. The following database parameters take incomplete ASM filenames:

• DB_CREATE_FILE_DEST
• DB_CREATE_ONLINE_LOG_DEST_n
• DB_RECOVERY_FILE_DEST
• CONTROL_FILES
• LOG_ARCHIVE_DEST_n (log_archive_dest_format will be ignored)
• LOG_ARCHIVE_DEST (log_archive_dest_format will be ignored)
• STANDBY_ARCHIVE_DEST

Here is an example of using an incomplete name when setting the DB_CREATE_FILE_DEST

parameter so that it will use the ASM disk group +sp_dgroup1:

SQL> alter system set db_create_file_dest=’+cooked_dgroup1’ scope=both;

Creating a Tablespace Using an ASM Disk Group as the Destination

There are different ways to create tablespaces using ASM disks. In this section, we will first look at creating an ASM tablespace, allowing the default ASM disk location to be used (as a result of having set the DB_CREATE_FILE_DEST parameter as we did earlier). We will then look at how to create a tablespace datafile by explicitly referencing the ASM disk group that it is supposed to be assigned to.

Creating Tablespaces Using Default ASM Assignments

Now that you have seen how to define a default ASM location, you can use the create

tablespace command to create a tablespace that will have a file in the ASM disk group by

default, as shown in this example:

SQL> create tablespace test_rgf datafile size 100k;
Let’s see where Oracle put the datafile now by querying the DBA_DATA_FILES view:

SQL> Select tablespace_name, file_name

from dba_data_files Where tablespace_name=’TEST_RGF’;

TABLESPACE FILE_NAME

---------- ---------------------------------------------------------

TEST_RGF +COOKED_DGROUP1/11gDB/datafile/test_rgf.256.613064385

You can have a mix of ASM datafiles and normal datafiles assigned to a tablespace, as shown in this create table statement:
SQL> Create tablespace part_asm_tbs Datafile ’c:\oracle\oradata\11gDB\part_asm_tbs_01.dbf’ size 10m, ’+COOKED_DGROUP1’ size 100k;

Let’s look and see where the datafiles were created:
SQL> Select tablespace_name, file_name from dba_data_files Where tablespace_name=’PART_ASM_TBS’;
TABLESPACE_NAME FILE_NAME
--------------- ------------------------------------------------------------
PART_ASM_TBS C:\ORACLE\ORADATA\11GDB\PART_ASM_TBS_01.DBF
PART_ASM_TBS +COOKED_DGROUP1/11GDB/datafile/part_asm_tbs.256.613066047


Note:- if you drop the PART_ASM_TBS tablespace, only the ASM files
related to that tablespace would be removed from the disk when you issue the drop
tablespace command. In cases such as these, you need to make sure you include the
including contents and datafiles parameter with the drop tablespace command.

Creating Tablespaces Referencing Specific ASM Disk Groups
There are going to be many times when you will not want to define a default ASM disk
group to write all tablespaces to. In this case, you will want to reference the specific ASM
disk group that you want a datafile created in when you issue the create tablespace command. Here is an example:
SQL> create tablespace another_test datafile ’+COOKED_DGROUP1’ size 100k;

Automatic Storage Management (ASM) lessons 3

Manually rebalancing disks within ASM is typically not required since ASM will perform
this operation automatically. However, in cases where you might want to have some more
granular control over the disk-rebalance process, you can use the alter diskgroup command
along with the rebalance parameter to manually rebalance ASM disks.

When we discuss rebalancing disks in ASM, we often discuss the power level that is

assigned to that rebalance operation. Setting power with regard to a rebalance operation

really defines the urgency of that operation with respect to other operations occurring on

the system (for example, other databases or applications). When a rebalance operation

occurs with a low power (for example, 1, the typical default), then that operation is not

given a high priority on the system As a result, the rebalance operation can take some time.When a higher power setting is used (for example, 11, the maximum), the ASM is given higher priority.

Note: what is re balancing ? 

ASM ensures that file extents are evenly distributed across all disks in a disk group. This is true for the initial file creation and for file resize operations. That means we should always have a balanced space distribution across all disks in a disk group.

You can set the default power limit for the ASM instance by changing the asm_power_limit parameter. Here is an example of starting a manual rebalance of a disk group:

SQL> alter diskgroup cooked_dgroup1 rebalance power 5 wait;

In this example, you will notice that we used the wait parameter. This makes this rebalance operation synchronous for our session. Thus, when the SQL prompt returns, we know that the rebalance operation has completed. The default is nowait, which will cause the operation to be synchronous in nature. You can check the status of the rebalance operation using the V$ASM_OPERATION view during asynchronous rebalance operations.

Finally, you can also use the rebalance parameter along with the power parameter when

adding, dropping, or resizing disks within a disk group, as shown in this example:

SQL> alter diskgroup cooked_dgroup1 resize all rebalance power 5;

Note:- ASM automatically initiates a rebalance after storage configuration changes, such as when you add, drop, or resize disks. The power setting parameter determines the speed with which rebalancing operations occur.

Checking the Consistency of a Disk Group

On occasion you might wonder if there is some problem with an ASM disk group, and you

will want to check the consistency of the ASM disk group metadata. This need might arise

because of an error that occurs when the ASM instance is started or as the result of an Oracle database error that might be caused by some ASM corruption. To perform this check, simply use the alter diskgroup command with the check all parameter, as shown in this example:

SQL> alter diskgroup sp_dgroup2 check all;

When you execute the alter diskgroup check all command the results are written to

the alert log of the instance. ASM will attempt to correct any errors that are detected.

Creating ASM Disk Group Directories
When you create an ASM disk group, it includes a system-generated directory structure for
the ASM files that will be stored in that disk group. The system-generated directory structure takes on the following format, where disk_group is the root of the directory hierarchy:
+disk_group/database_name/object_type/ASM_file_name
for Eg, +DATA/oracle/datafile/system.259.724180831
The database name will be the name of the database that the data is associated with. The
object_type is the type of object being stored (for example, datafile) and the ASM_file_
name is the system-generated filename assigned to that ASM file.
ASM allows you to create your own directories within these predefined structures. This
allows you to give alias names to the ASM files that you will create. This can make working
with ASM files easier.
To create a directory structure within a diskgroup, you use the alter diskgroup command with the add directory parameter, as shown in this example:
SQL> ALTER DISKGROUP cooked_dgroup1 ADD DIRECTORY ‘+cooked_dgroup1/stuff’;

Using the ASMCMD Command-Line Utility

The ASMCMD tool is a command-line utility that allows you to manage ASM instances
and the disk structures and files within those instances. With ASMCMD, you can do the
following:

• List contents of ASM disk groupNN s
• Perform searches (like directory listings)
• Add or remove directories
• Display space availability and utilization 

Starting ASMCMD

To start ASMCMD, simply set your ORACLE_SID to +ASM and then type asmcmd from the

command line, as shown here:

C:\>set ORACLE_SID=+ASM

C:\>asmcmd

Or from Unix:

/opt/oracle>export ORACLE_SID=+ASM

/opt/oracle>asmcmd

Note:- You will need to make sure that perl.exe is in the path before you run ASMCMD. If you have installed more than one ORACLE_HOME, it may take some setup to get the environment to  set correctly. Make sure the following is set to the correct 

ORACLE_HOME, PATH, PERL5LIB,PERLBIN .

ASMCMD Commands
ASMCMD has a basic set of commands, many of which mimic Unix commands. You can
see these commands from the ASMCMD prompt if you type in help. The commands are
pretty straightforward and easy to use.Below Table  lists the different ASMCMD commands
and their purposes. Reference summary of commonly used asmcmd commands:

cd Changes the current directory to the specified directory.

du Displays the total disk space occupied by ASM files in the
specified ASM directory and all its subdirectories, recursively.

exit Exits ASMCMD.

find Lists the paths of all occurrences of the specified name (with
wildcards) under the specified directory.

help Displays the syntax and description of ASMCMD commands.

ls Lists the contents of an ASM directory, the attributes of the
specified file, or the names and attributes of all disk groups.

lsct Lists information about current ASM clients.

lsdg Lists all disk groups and their attributes.

mkalias Creates an alias for a system-generated filename.

mkdir Creates ASM directory.

pwd Displays the path of the current ASM directory.

rm Deletes the specified ASM files or directories.

rmalias Deletes the specified alias, retaining the file that the alias points to.

Overview of ASM Data Dictionary Views

Several data dictionary views exist to help manage ASM. These data dictionary views are available both when connected to the ASM instance as well as any Oracle 10g database. Each view is slightly different in its presentation depending on whether the instance you are looking at is an ASM instance or a database instance (and some views are only used in the ASM instance). The table below shows these views.

View Name	Description
V$ASM_DISKGROUP	In an ASM Instance: This view will describe a given disk group. In a Database: This view contains a single row for each ASM disk group that is mounted by the local ASM instance.Note that discovery will occur each time you query thisview. This can have performance impacts.
V$ASM_DISK	In an ASM Instance: This view describes each disk that was discovered by the ASM instance. All disks are reported, even those not assigned to disk groups. In a Database: This view describes each disk that is assigned to that database.Note that discovery will occur each time you query thisview. This can have performance impacts.
V$ASM_DISKGROUP_STAT	This view is equivalent to the V$ASM_DISKGROUP view. However, this view will not cause discovery to occur. This can help with performance, but the data in the view may not be accurate.
V$ASM_FILE	In an ASM Instance: Displays each ASM file contained in the ASM instance. In a Database: Not used in a database instance.
V$ASM_DISK_STAT	This view is equivalent to the V$ASM_DISK view. However, this view will not cause discovery to occur. This can help with performance, but the data in the view may not be accurate.
V$ASM_TEMPLATE	In an ASM Instance: Displays each ASM template contained in the ASM instance by disk group. In a Database: Not used in a database instance.
V$ASM_ALIAS	In an ASM Instance: Displays each alias contained in the ASM instance by disk group. In a Database: Not used in a database instance.
V$ASM_OPERATION	In an ASM Instance: Displays each long running operation occurring on the ASM instance. In a Database: Not used in a database instance
V$ASM_CLIENT	In an ASM Instance: Displays each database that is using at least one disk group managed by the ASM instance. In a Database: Not used in a database instance.

Basic of raids must know

Every article that i posted related to dba only, here is a diversion .Here i am giving a good notes about RAID. As we are using raids in different purpose in IT industry , a dba should know atleast the basic functioning of RAID.Hope it will helps to somebody. :)

A Redundant Array of Independent Drives (or Disks), also known as Redundant Array of  Inexpensive Drives (or Disks) (RAID) is an term for data storage schemes that divide and/or replicate data among multiple hard drives. RAID can be designed to provide increased data reliability or increased I/O performance, though one goal may compromise the other. Raid provide fault tolerance by stripping , mirroring or parity .

RAID can be implemented on a hardware or software level. On a hardware level, you can have hard disks connected to a RAID hardware controller, usually a special PC card. Your operating system then accesses storage through the RAID hardware controller. Alternatively, you can implement RAID as a software controller,IN a software RAID controller ,special program manage access to hard disks treated as RAID devices. The software version lets you use IDE hard disks as RAID disks. Linux uses the MD driver, supported in the 2.4 kernel, to implement a software RAID controller. Linux software RAID supports five levels (linear, 0, 1, 4, 5, and 6), whereas hardware RAID supports many more. Hardware RAID levels, such as 7–10, provide combinations of greater performance and reliability. Before you can use RAID on your system, make sure the RAID levels you want to use. If not, you will have for the kernel. Check the Multi-Driver Support component specify support for any or all of the RAID levels.
Note: we will get redundancy only when parity or mirroring  is present on RAID.

Commonly used RAID levels for UNIX / Linux and Windows server

RAID 0 (Striping)

This level is achieved by grouping 2 or more hard disks into a single unit with the total size equaling that of all disks used. Practical example: 3 disks, each 80GB in size can be used in a 240GB RAID 0 configuration. You can also use disk having different sizes.

RAID 0 works by breaking data into fragments and writing to all disk simultaneously. Read and write operations are improved as compared to single disk system, since the load is shared across many channel and are done in parallel on the disks.On the other hand, no single disk contains the entire information for any bit of data committed. This means that if one of the disks fails, the entire RAID is rendered inoperable, with unrecoverable loss of data.

However since there is no redundancy, it doesn't provide fault tolerance. If even one drive fails, data across the drive array will be lost.

RAID 0 is suitable for non-critical operations that require good performance, like the system partition or the /tmp partition where lots of temporary data is constantly written. It is not suitable for data storage.

Suggested application:- Video Production and Editing , Image Editing etc

RAID 1 (Mirroring) 

This level is achieved by grouping 2 or more hard disks into a single unit with the total size equaling that of the smallest of disks used. This is because RAID 1 keeps every bit of data replicated on each of its devices in the exactly same fashion, create identical clones. Hence the name, mirroring. Practical example: 2 disks, each 80GB in size can be used in a 80GB RAID 1 configuration.
Because of its configuration, RAID 1 reduced write performance, as every chunk of data 
has to be written n times, on each of the paired devices. The read performance is identical to single disks. Redundancy is improved, as the normal operation of the system can be maintained as long as any one disk is functional. Of Course, the disks must be of equal size. If one disk is larger than another, your RAID device will be the size of the smallest disk.
If there are spare disks available, and if the system survived the crash, reconstruction of the mirror will immediately begin on one of the spare disks, after detection of the drive fault.
RAID 1 is suitable for data storage, especially with non-intensive I/O tasks.

RAID4   (striping with parity) 
This RAID level is not used very often. It can be used on three or more disks. Instead of completely mirroring the information, it keeps parity information on one drive, and writes data to the other disks in a RAID-0 like way .If one of the data drives in array fails, the parity information can be used to reconstruct all data. However if more than one disks fail, whole of the data is lost.
The reason this level is not more frequently used, is because the parity information is kept on one drive. This information must be updated every time one of the other disks are written to. Thus, the parity disk will become a bottleneck, if it is not a lot faster than the other disks. However, if you just happen to have a lot of slow disks and a very fast one, this RAID level can be very useful.

RAID 5 (Stripping with distributed parity)

Raid 5 support both striping as well as redundancy in terms of parity. RAID 5 improves on RAID 4 by striping the parity data between all the disks in the RAID set. RAID-5 can be used on three or more disks, with zero or more spare-disks.

This is a more complex solution, with a minimum of three devices used. If one of the RAID 0 devices malfunctions, the array will continue operating, using the parity device as a backup. If spare disks are available, reconstruction will begin immediately after the device failure. If two disks fail simultaneously, all data are lost.Like raid 4 RAID-5 can survive one disk failure, but not two or more.

RAID 5 improves the write performance than raid 4, as well as redundancy and is useful in mission-critical scenarios, where both good throughput and data integrity are important.

In this configuration 25% of the combined dik space is used to store the parity information

And around 75% of the total disk capacity is available for use.

Linear RAID

This is a less common level, although fully usable. Linear is similar to RAID 0, except that data is written sequentially rather than in parallel. Linear RAID is a simple grouping of several devices into a larger volume, the total size of which is the sum of all members. For instance, three disks the sizes of 40, 60 and 250GB can be grouped into a linear RAID the total size of 350GB. 

Linear RAID provides no read/write performance, not does it provide redundancy; a loss of any member will render the entire array unusable. It merely increases size. It's very similar to LVM. Linear RAID is suitable when large data exceeding the individual size of any disk or partition must be used. 

RAID6

RAID-6 is an extension of RAID-5 to provide additional fault tolerance by using dual distributed parity schemes. Dual parity scheme helps survive two disk failures at a time without data loss. It extends RAID 5 by adding an additional parity block, thus it uses block-level striping with two parity blocks distributed across all member disks.  The read performance is same as RAID 5. However, its write performance is poorer than RAID 5 due to overhead associated with the additional parity calculations. But it does better than RAID 5 on file data protection because RAID 6 provides protection against double disk failures and failures while a single disk is rebuilding. RAID 5 fails to do so.

RAID 10 

RAID 10 or RAID 1+0 - Combination of RAID 0 (data striping) and RAID 1 (mirroring). 

• RAID 10 is also called as RAID 1+0
• It is also called as “stripe of mirrors”
• It requires minimum of 4 disks
• To understand this better, group the disks in pair of two (for mirror). For example, if you have a total of 6 disks in RAID 10, there will be three groups–Group 1, Group 2, Group 3 as shown in the above diagram.
• Within the group, the data is mirrored. In the above example, Disk 1 and Disk 2 belongs to Group 1. The data on Disk 1 will be exactly same as the data on Disk 2. So, block A written on Disk 1 will be mirroed on Disk 2. Block B written on Disk 3 will be mirrored on Disk 4.
• Across the group, the data is striped. i.e Block A is written to Group 1, Block B is written to Group 2, Block C is written to Group 3.
• This is why it is called “stripe of mirrors”. i.e the disks within the group are mirrored. But, the groups themselves are striped.  

RAID 01

• RAID 01 is also called as RAID 0+1
• It is also called as “mirror of stripes”
• It requires minimum of 3 disks. But in most cases this will be implemented as minimum of 4 disks.
• To understand this better, create two groups. For example, if you have total of 6 disks, create two groups with 3 disks each as shown below. In the above example, Group 1 has 3 disks and Group 2 has 3 disks.
• Within the group, the data is striped. i.e In the Group 1 which contains three disks, the 1st block will be written to 1st disk, 2nd block to 2nd disk, and the 3rd block to 3rd disk. So, block A is written to Disk 1, block B to Disk 2, block C to Disk 3.
• Across the group, the data is mirrored. i.e The Group 1 and Group 2 will look exactly the same. i.e Disk 1 is mirrored to Disk 4, Disk 2 to Disk 5, Disk 3 to Disk 6.
• This is why it is called “mirror of stripes”. i.e the disks within the groups are striped. But, the groups are mirrored.

Main difference between RAID 10 vs RAID 01

• Performance on both RAID 10 and RAID 01 will be the same.
• The storage capacity on these will be the same.
• The main difference is the fault tolerance level. On most implememntations of RAID controllers, RAID 01 fault tolerance is less. On RAID 01, since we have only two groups of RAID 0, if two drives (one in each group) fails, the entire RAID 01 will fail. In the above RAID 01 diagram, if Disk 1 and Disk 4 fails, both the groups will be down. So, the whole RAID 01 will fail.
• RAID 10 fault tolerance is more. On RAID 10, since there are many groups (as the individual group is only two disks), even if three disks fails (one in each group), the RAID 10 is still functional. In the above RAID 10 example, even if Disk 1, Disk 3, Disk 5 fails, the RAID 10 will still be functional.
• So, given a choice between RAID 10 and RAID 01, always choose RAID 10.

Before You start 

Specially built hardware-based RAID disk controllers are available for both IDE and SCSI drives. They usually have their own BIOS, so you can configure them right after your system's the power on self test (POST). Hardware-based RAID is transparent to your operating system; the hardware does all the work. 

If hardware RAID isn't available, then you should be aware of these basic guidelines to follow when setting up software RAID. 

IDE Drives

To save costs, many small business systems will probably use IDE disks, but they do have some limitations.

• The total length of an IDE cable can be only a few feet long, which generally limits IDE drives to small home systems.
• IDE drives do not hot swap. You cannot replace them while your system is running.
• Only two devices can be attached per controller.
• The performance of the IDE bus can be degraded by the presence of a second device on the cable.
• The failure of one drive on an IDE bus often causes the malfunctioning of the second device. This can be fatal if you have two IDE drives of the same RAID set attached to the same cable.

For these reasons, I recommend you use only one IDE drive per controller when using RAID, especially in a corporate environment.

Serial ATA Drives

Serial ATA type drives are rapidly replacing IDE drives as the preferred entry level disk storage option because of a number of advantages:

• The drive data cable can be as long as 1 meter in length versus IDE's 18 inches.
• Serial ATA has better error checking than IDE.
• There is only one drive per cable which makes hot swapping, or the capability to replace components while the system is still running, possible without the fear of affecting other devices on the data cable.
• There are no jumpers to set on Serial ATA drives to make it a master or slave which makes them simpler to configure.
• IDE drives have a 133Mbytes/s data rate whereas the Serial ATA specification starts at 150 Mbytes/sec with a goal of reaching 600 Mbytes/s over the expected ten year life of the specification.

If you can't afford more expensive and faster SCSI drives, Serial ATA would be the preferred device for software and hardware RAID

SCSI Drives

SCSI hard disks have a number of features that make them more attractive for RAID use than either IDE or Serial ATA drives.

• SCSI controllers are more tolerant of disk failures. The failure of a single drive is less likely to disrupt the remaining drives on the bus.
• SCSI cables can be up to 25 meters long, making them suitable for data center applications.
• Much more than two devices may be connected to a SCSI cable bus. It can accommodate 7 (single-ended SCSI) or 15 (all other SCSI types) devices.
• Some models of SCSI devices support "hot swapping" which allows you to replace them while the system is running.
• SCSI currently supports data rates of up to 640 Mbytes/s making them highly desirable for installations where rapid data access is imperative.