Ibrix Storage Cluster

Overview

IBRIX Scale-Out NAS system consists of 12 servers with large amounts of cache and 6 dual controller storage arrays. They provide a total of 224 TB high performance usable storage capacity. Its remote replication capabilities offer a higher level of redundancy for important data.

Sections

System Details
Features

• Snapshots
• Remote Replication

---

System Details

LinGA’s internal Ibrix storage cluster contains 12 servers in six storage arrays. Four are located in LinGA’s primary server facility and a separate pair are located in another building for off-site replication. Combined, the system contains 576 TB raw (224 TB usable) storage served to all LinGA compute clusters through 1Gig and 10Gig network links.

Features

Snapshots

What is a “snapshot”

The storage system maintains “snapshots”–incremental backups–of the data. This is one of the many protections against data loss and is implemented on a partition-wide scale. This means that all shares (group and personal folders) in a partition (individuals, protected) have the same snapshot schedule and that it is not possible to add more replication or remove snapshots for specific groups.

Snapshot schedule

Path	Daily  Retention	Daily   Creation	Weekly   Retention	Weekly   Creation	Snapshot  Reclamation
/protected/individuals	7	1 PM	0	N/A	Wednesday
/protected/projects	7	1 PM	0	N/A	Wednesday

 

Note About Usage (“df” vs. “du”)

Snapshots are hidden from the user for simplicity, but they do count against your storage usage. This can cause a discrepancy between the “du” (disk usage) and the “df” (disk free) commands. For storage accounting purposes, “df” reports the correct value.

Typically df reports a value 4-5 times larger than du, though the usage depends on directory contents and recent file history. As an extreme example we will use a protected folder that has had many recent changes and deletions to large files.

• The du (disk usage) command shows:

 

• The df (disk free) command shows:

 

This means that the sum of all file sizes (not including snapshot information) in this directory is 1.1GB, but that the this share is actually using 48GB on the storage system. This extreme difference is caused by deleting many large files, so they are no longer counting against du, but are stored as incremental backups for df. In a few days-two weeks the current snapshot information will be reclaimed as snapshots are removed and new snapshots will be added reflecting more current incremental backups.

Remote Replication

In addition to snapshots, shares in the “protected” partition are replicated to a co-location site in another building on the medical campus. The remote replication is a storage cluster feature that allows another layer of redundancy for the most important data.