Ibrix Storage Cluster

Overview

IBRIX Scale-Out NAS system consists of 12 servers with large amounts of cache and 6 dual controller storage arrays. They provide a total of 224 TB high performance usable storage capacity. Its remote replication capabilities offer a higher level of redundancy for important data.

Sections

System Details
Features


System Details

LinGA’s internal Ibrix storage cluster contains 12 servers in six storage arrays. Four are located in LinGA’s primary server facility and a separate pair are located in another building for off-site replication. Combined, the system contains 576 TB raw (224 TB usable) storage served to all LinGA compute clusters through 1Gig and 10Gig network links.

Features

Snapshots

What is a “snapshot”

The storage system maintains “snapshots”–incremental backups–of the data. This is one of the many protections against data loss and is implemented on a partition-wide scale. This means that all shares (group and personal folders) in a partition (individuals, protected) have the same snapshot schedule and that it is not possible to add more replication or remove snapshots for specific groups.

Snapshot schedule

Path Daily 

Retention

Daily  

Creation

Weekly  

Retention

Weekly  

Creation

Snapshot 

Reclamation

/protected/individuals 7 1 PM 0 N/A Wednesday
/protected/projects 7 1 PM 0 N/A Wednesday

 

Note About Usage (“df” vs. “du”)

Snapshots are hidden from the user for simplicity, but they do count against your storage usage. This can cause a discrepancy between the “du” (disk usage) and the “df” (disk free) commands. For storage accounting purposes, “df” reports the correct value.

Typically df reports a value 4-5 times larger than du, though the usage depends on directory contents and recent file history. As an extreme example we will use a protected folder that has had many recent changes and deletions to large files.

  • The du (disk usage) command shows:

[cjahnke@redstar cjahnke]$ du -hs
1.1G    .

 

  • The df (disk free) command shows:


[cjahnke@redstar cjahnke]$ df -h .
Filesystem            Size  Used Avail Use% Mounted on
ibx-ct4:/ibxct4fs1/protected/individuals
50G   48G 2.5G  96% /protected/individuals

 

This means that the sum of all file sizes (not including snapshot information) in this directory is 1.1GB, but that the this share is actually using 48GB on the storage system. This extreme difference is caused by deleting many large files, so they are no longer counting against du, but are stored as incremental backups for df. In a few days-two weeks the current snapshot information will be reclaimed as snapshots are removed and new snapshots will be added reflecting more current incremental backups.

Remote Replication

In addition to snapshots, shares in the “protected” partition are replicated to a co-location site in another building on the medical campus. The remote replication is a storage cluster feature that allows another layer of redundancy for the most important data.