Paging your blocks of data

Doesn't happen often, but some new products change dramatically the landscape of their technology. One good example is virtual memory, a feature present and given for granted today in any computer worth its silicon content. Try to imagine what life would be if computers had only real memory to run their applications.

Probably we would be spending gazillion of dollars on memory chips to make sure that our most memory-hungry applications could run.

Of course virtual memory freed us from babysitting RAM: in modern computers the OS keeps tracks of how much memory an application needs and will automatically swap pages of data or code between RAM and less expensive virtual memory.

The application and its users don't know and don't care where that page is, because virtual memory has made the actual location of a page of data irrelevant.

Why am I rambling about virtual memory? Because a new application from Compellent, Data Progression, promises to do for storage what virtual memory did for computing.

If you forgot about Compellent here is a link to a review of their SAN.

In essence, Data Progression makes possible moving infrequently referenced blocks of data to a different storage device, automatically and seamlessly.

To understand how this works, it's important to remember that a Compellent SAN has a rather unique architecture. For example, you can define LUNs for say 10 TB even if your SAN has actual disk space for only 1TB. As long as you don't use all the space immediately, this will work, and you can add more disk space gradually without breaking anything.

One more thing to keep in mind: When creating a LUN, Compellent will suggest to assign different tiers of storage to accommodate less frequently referenced data.

For example a fast array with 15K RPM, 36GB SCSI drives could be the primary tier for that volume, while a slower but more spacious array with 200 GB SATA drives could be the secondary tier.

You can create more than two tiers, and I believe Compellent will add other media besides disks in the future. Also, you can further discriminate where to store data according to the RAID level, for example, RAID 5 for primary, RAID 1 for other tier if this make sense with your application.

Interesting so far? Let's recap. A Compellent volume can spread across different tiers of storage, pretty much like a blanket can lay across the different steps of a stair. Great, but how does the system decides where to store data?

Another morsel of information will help understand that: Compellent keeps (again automatically) some interesting metadata for each block of data. For example, frequency of access, what tier of storage is it on, what RAID level, which volume, etc...

Time to bring all this together. Let's assume that a volume contains Office files and has tier 1 on RAID 5 SCSI and tier 2 on RAID 1 SATA.

It's intuitive that the most recently changed files will remain on the expensive SCSI drives on tier 1, while older stuff will in time migrate to the less expensive second tier.

What happens if user Jean creates a file and doesn't touch it for months? Without her knowledge, Compellent will move the blocks of that file to tier 2.

What should Jean do when she needs to update that file 10 months later? ? Absolutely nothing: Jean will be able to open her file without knowing that the system is pulling those blocks out of the secondary tier.

Will there be a delay? Compellent swears on no delays and although I haven't worked with Data Progression yet, my guess is that if there is an access time difference with more recent files Jean won't even notice.

What's the impact on your servers when using Data Progression? Again, nothing changes: the LUN is just another volume. The OS won't even know that there are several arrays or RAID levels behind that LUN.

So is this application from Compellent going to kill all the hype about ILM (information lifecycle management)?

Honestly I don't know, but Data Progression has little to do with ILM, in my view, because it lacks (at least for now) a main prerequisite: classifying data according to business criteria.

What Data Progression can do quite well, is to keep recent data on your most expensive tier of storage and automatically move old stuff somewhere else, without a dedicated babysitter and with minimal increase of administrative cost. In fact, the application starts at $10,000.

Perhaps the most likeable aspect of Data Progression is that it works without affecting users or applications, just like virtual memory.