Deduplication is an advanced data reduction technique which can have a large impact on the amount of storage space required for data.  In the case of backup, it is especially effective because typically the same data will be repeatedly sent to the backup store.   Almost every backup product on the market today offers deduplication based backup to disk (B2D), and the rest have it on their roadmaps.

Because you can configure these backup-to-disk servers to write to a Cloud Storage appliance like CloudArray (and thus replicate your backup store to Cloud Storage, B2D2C), how you configure your appliance will end up having a large impact on performance due to the unique characteristics of deduplication engines

Read the full story after the fold….

Think of the typical backup  process where a full backup is done on servers every week and incremental backups are done daily.  If you have 100 windows based servers, this would mean that you would be backing up 100 nearly identical operating system volumes every week.  You are also backing up full copies of any changed files every night, even if it’s only a 4 byte change to a 100GB file.  Once the archive bit gets flipped, that’s the signal to back it up at night.

The traditional backup process can really eat through storage.

How deduplication solves this problem.

Deduplication aimed at backup environments is typically a block level, target deduplication mechanism.  While implementations vary, the mechanism that most deduplication engines use will identify identical blocks of data and only store one copy.   As new blocks come in, they are checked against what is already in the data store and if they are duplicates, the data itself is not stored, but a small pointer to the already stored data block is.

What this means is that your backups can be compressed to at least a 10:1 ratio.  You can achieve higher rates if you have a lot of common data, like the hundred windows servers I mentioned above. Less data stored means less that has to be transferred, and improved performance overall.

If your deduplication engine is doing the deduplication at the file level (typical for a file system based device), you will see much lower deduplication rates, typically on the order of 3x instead of 10x.  You may still find it advantageous to use application block level  deduplication over file level depuplication you get through other devices.

Additionally, the whole concept of incremental backups goes away.  After the first full backup, the system is only sending changed blocks to the data store, and it can retain a very large number of restore points so you have far more than the few backup sets offered under traditional backup policies to choose from.   Every backup is a full backup, every backup is changed blocks only, and you can keep dozens of restore points.

So what does this have to do with performance?

Traditional backup applications are very nearly 100% sequential writes.   The exception to this is if you choose to do a verify operation after you write the backup to tape or disk (a sequential read operation).

In the case of deduplication, the i/o ratio has a much higher mix of random reads since before every write the system is looking to see if the block of data  exists in your datastore already.

A high percentage of random reads in a Cloud Storage environment means that you will have long wait times if the data is not cached locally, completely, and on a dedicated cache volume.  If the data is stored on a partial cache volume, or if the cache volume gets shared for a number of data sources, then you run the risk of having to constantly go back out to the Cloud to satisfy reads.  This can cause the equivalent of a lot of swapping and paging.  If there is no local cache volume, you are guaranteed to see long wait times.

Not all Cloud Storage appliances support fully sized and dedicated cache volumes that can be segregated from other data volumes or sources.  CloudArray does.

So if you are going to use the deduplication feature of a modern backup application to save space at your Cloud Storage provider, you would be well advised to use a fully sized (100% of target storage capacity), dedicated cache volume for your backup application to write to.  Either that, or plan for long backup times.

News wise, this week has been a bloggers paradise.  Not so much for vendor bloggers like me (though I am sneaking this one in – ), but if anyone doubts that Cloud computing and Cloud Storage have arrived, are viable, and will become a real force in the market, they haven’t really been paying attention.

Lets start with  the 3PAR highlights.  Three companies approached 3PAR at more or less the same time and 2 are now in a heated bidding war.  The eventual sale price could far exceed $2B.  And what does 3PAR have that is worth that kind of attention?  They have a highly scalable, easy to manage, highly efficient storage platform that is being snapped up by Cloud Service Providers and which is being well accepted in Enterprise accounts.  Clearly HP and DELL believe that the HW that provides the basis for Cloud Storage environments is critical to their future.  Cloud Storage is indeed very healthy and its future is bright.

On the other front, VMworld kicks off next week and promises to be a vPaloozah.  The hype/buzz has been building for weeks and lots of vendors will be making announcements – almost all of them having something to do with Cloud Computing and Storage.  It should be a fun week to watch and listen.  Unfortunately I have other committments and can’t attend this year (first time in 4 years), but my heart will be there.  It’s one of my favorite shows.

So the net effect of all of this is that we seem to be at that mystical tipping point.  All the concern over whether “Cloud” is real, whether it is mature, and whether it is affordable, will quickly fall away as early adopters lead to innovators and then full deployments.  Then, the hard work begins.  Should be a fun ride.

Everything about  Computing is a trade off.   The old joke is “Speed, Capacity, or Cost”, you can control 2.

Cloud Computing and Storage are no exception.    How you configure your environment can have a direct impact on your performance levels, but it can also have a direct impact on your budget.

More after the fold….

One of the defining attributes of a Hybrid Cloud Storage model is there is a local storage component which gets used as local cache or buffer.  This helps performance considerably because it is always faster to read or write to disk than to the Cloud.  Simple approaches use one big volume as a buffer (not a cache), and everything goes through it, more or less in the order you intended.  More sophisticated approaches create a real cache, with LRU/MRU policies, read ahead algorithms, and cache coherency checking.

Pure Cloud Storage clients suffer from extreme network latency, and that is usually bad for performance.  If you can afford to wait days to read or write small amounts of data, that model might work for you.  For everyone else, a hybrid configuration is the way to go.

CloudArray, TwinStrata’s breakthrough Cloud SAN software, uses a hybrid storage model with an advanced Intelligent Caching architecture.  We allow you to configure local cache, on a per volume basis if you like, on a sliding scale from a fraction of the target volume size to 100% of the target volume size.

The larger the cache relative to the target volume, the better your local read/write performance will be.  When you get to 100% of the target volume size, reads and writes occur at local disk speed regardless of the latency of your connection to your cloud Storage provider.  Some vendors use a common cache volume for all managed volumes; TwinStrata allows you to use either dedicated or shared cache pools (or both concurrently) for any volume managed by CloudArray.

Remember: Cost, Speed, Capacity:  Pick 2

One of the great things about CloudArray is it is infinitely configurable.  We support local cache disk configurations based on DAS, SAN or NAS volumes.  You can configure those connections through one port (SAN/NAS/RAID controller), or many.  You can use fast disks or slow.  You can use IDE, SCSI, iSCSI, FC, SAS, SSD, or any storage device that can mount using a standard protocol as a device to the CloudArray appliance.

That is also one of the performance configurations when you build out your configuration.  If you attach slow disks to CloudArray, with say a 40 MB/s max throughput over your local storage network, then that is the maximum write performance you will see to CloudArray.  Configure faster drives, connected over a fast network, and you will see a very high level of performance from your CloudArray appliance.  It really comes down to a quality of service decision in the end.

You really need to treat CloudArray like a physical array when configuring both it’s local cache devices and the network links to your application servers.    For applications that require less performance,  backup applications from mirrored copies of your application environment for example, you can use slower/cheaper storage for your local cache.  For those applications which are performance critical – creating remote copies of your Exchange 2010 DAG nodes for example, you want to allocate higher performance storage as your cache devices and use a larger cache device.

By doing this, you can control your costs and your performance levels on a QoS basis across all the applications you want to back up or replicate to Cloud Storage.  TwinStrata is unique in allowing you to configure cache policies and volumes either in groups or individually, and it is the only product which allows a variable caching size to be configured.

We do all of this with Security (256 bit AES encryption) and advanced target data reduction techniques.  You can download a copy and try it yourself, but keep in mind your performance will be related to the performance of the disk and network connection you allocate as cache and your choice of caching policy.

Once you cut through all the hype surrounding the benefits of Cloud Storage, specifically the economics of Public Cloud Storage, it becomes clear that there are use cases that do shine.

At the heart of the analysis are tried and true factors effecting storage costs like OPEX and CAPEX, deduplication, thin provisioning, compression, utilization, TCO, ROI, Business Opportunity costs (downtime, business recovery, business restart), etc.

Data Storage may be cheap and getting cheaper, but storing less data is always cheaper than storing more, and cutting costs – both operational and capital – is still critical.

The traditional Storage Model

There are several cost factors working against you in the traditional model:

• You tend to over configure new storage arrays, “just in case”, buying slightly more capacity than needed.  Nobody likes having to justify another capitol request because they planned poorly.  Much of this capacity isn’t needed on day 1, you are buying capacity up to a year ahead of need
• Once purchased, you will have to find room in the data center, provide power and cooling, and assign someone to administer it.
• Next, you have to have some storage admin provision the volumes, assign them to the proper ports, zone them into the proper switches, and finally hand them over to the system admins.
• Application environments consuming storage on the array tend to be over-provisioned, again because of planned growth.  If that growth never arrives, you have stranded storage that will never be used and can’t be easily reclaimed

Then, after three years, you will have to buy a replacement and migrate all the data over to the new array.   You will probably even repurchase and migrate that stranded storage as well.

But the economics of using Cloud Storage are very different.

With Public Cloud Storage, there is no capital expense to buy a new array (or a replacement array).  Cloud Storage is rapidly expandable and you pay for only what you use.  So after initial deployment, you would only have paid for the transfer cost of the initial storage and the monthly cost to host it.  As you add additional data, you pay the additional fees, but only when you actually use it.

And since it’s deduplicated/compressed and thin provisioned, you can give your App Admin their full request of local storage, and you will only be paying for what they actually use.   That’s why deduplication, compression, and thin provisioning, as well as snapshots, make so much sense to Cloud Storage deployments.

Of course, since it’s self-provisioned, the storage admin doesn’t need to make any bin file changes, or decide on spindle placement, or go through a complicated zoning assignment.  All the administrator needs to do is pick the max size of the volume to be allocated, which Cloud provider tier to use, and which host it will be assigned to.   There really aren’t any “storage administrator” skills required for this stage.

No local floor tiles or racks need to be freed up and assigned, no power needs to be run, and there are no HVAC implications.  As an added benefit, you can configure it so there is both a local and a remote copy of the data, eliminating the infrastructure that would normally be required for site to site replication.

There are no storage upgrade costs.

There is no need to spend money custom coding to Cloud Storage API’s.  Our CloudArray software can easily interface between your legacy server environment and Public Cloud Storage Providers.

You will have some additional costs for bandwidth between your data center and your Cloud service Provider, but bandwidth, like storage, is becoming a commodity and prices are dropping.

The net result is a considerable cost savings over deploying storage the traditional model.

Is it all a perfect world with Public Cloud Storage?

Of course not, there is always a tradeoff.   In Enterprise data centers, the economics tip back to maintaining local and dedicated storage, here Private Storage Clouds still offer savings over traditional storage models but you will incur infrastructure expenses.

But in small to mid-sized businesses, the economics of using Cloud Storage (Public or Private) for second and third tier data storage is very compelling.  Even in cases where you are storing backups in the Cloud, the advantages can rise to a 50% cost savings over traditional tape backup and offsite storage.  I’ll go into more detail on that in future posts.

In the meantime, if you would like more information, feel free to contact us.

Data Protection is a natural fit for cloud Storage since it affords you a low cost, offsite, service oriented storage model for a function that is critical to your business.

Data Protection isn’t  a complex operation.   It’s the process of ensuring that applications and servers can be restored in the case of a failure or loss of some kind.  Backup is a good example of a typical DP application, but certainly not the only one.  DP is a lot of things, risk reduction, securing your assets, business continuity and disaster restart.

But at its core, DP is about Continuance, Restart, and Recovery; not backup.

Traditionally, backups are spun to tape, and then shipped to offsite storage for safe keeping.    Tape  is well known and understood, but has a lot of drawbacks.

First, it’s a serial media.  You physically have to advance the tape to a specific location to begin to read off data you are looking for, so restore operations can be slow.  That’s assuming the tape is even readable (studies have claimed up to a 40% read error rate)); if either the index or data segment you are looking for is corrupt, you won’t be able to restore the data.

And of course, someone has to find the correct tape at the offsite storage location (remember the Iron Mountain tape losses?) and then ship them back to you.  All of this elongates the recovery process and  increases risk and expense.

A better way – B2D

In more advanced environments, customers will use a backup to disk (B2D) or B2D2T process in place of traditional tape backups.  This has several advantages, first, since the data you want to recover is online, it is extremely easy to find and restore just the data you are looking for.  Secondly, it is far more reliable than tape.   A 1 in 3 or even 1 in 10 chance of not being able to restore the data you need from tape is not a very good metric.

But B2D alone also has disadvantages.  First, it’s a local solution only.  Unless you replicate that disk (very expensive since you need infrastructure in two data centers), you have no offsite copy of your data.  B2D2T is an alternative here, you gain the benefits of short term storage on disk and long term archival offsite, but you are still burdened with the expenses and unreliability associated with tape.

B2D is Good, B2D2C is better

If you could extend your B2D solution to offer low cost, secure, and reliable, offsite storage at the same time, that would be the best of both worlds.  And in fact, B2D combined with CloudArray from TwinStrata enables you to do just that.

With fully cached volumes, you not only have a local copy of your disk based backups, but you gain an automatic offsite copy at your Cloud Storage Provider (public or private).   Of course it’s secure as well.

Even Better, you already have the Backup Software

Chances are, whatever backup product you are currently using to write to tape can be used to write to CloudArray storage volumes (and ultimately to the Cloud).  Almost every major backup application vendor now supports a B2D option in their software, and most will run concurrently with local tape operations.  You may have to upgrade to a later version of their product, but you won’t have to rip and replace your current backup solution.

Adding the Cloud component is easy

To configure the two to work together, all you would need to do is create a local CloudArray volume and mount it to your backup server.  The backup application can then be configured to write to this volume just like it would any local volume, and CloudArray will then copy that data safely to your Cloud Storage Provider.

If a restore becomes necessary, it would be serviced directly from the local cache, and you wouldn’t need to go out to the cloud at all.  Since CloudArray cache is persistent and can be dedicated to specific volumes, there is no risk that your backup data would be flushed out by another more active volume.

In the event of a local site disruption, the Cloud resident backups could be restored to any location of your choice, even a Cloud  Computing environment such as Amazon EC2.

Data Protection is about Recovery.  Reducing risk, and decreasing costs.  It isn’t about doing unnatural acts with Robocopy because that is the only tool that will work with your gateway software.

CloudArray can be used today with leading backup application products such as Net Backup, Backup Exec, Veeam, Visioncore, and a host of others.  Visit www.TwinStrata.com for a free trial.

TwinStrata has posted a new White Paper over on TechTarget. We’ve blogged about this quite a bit, held webinars on it, and now we’ve made it available as a stand alone white paper.

You can find it here.

Let us know what you think.

Thanks

You can watch a video of interviews with Ann from HostingCon – held in Austin, Texas last month.

]We get asked how VMware environments can take advantage of Cloud Storage often enough that I thought I’d mention a few options here for use cases that make financial and operational sense.

Backup

I’ve covered using CloudArray with VDR here before, so I won’t dwell on this other than to mention that any backup application that makes use of the new vStorage API’s,  performs deduplication and can write to disk is an excellent choice to reduce backup expenses as well as gain an offsite storage component with Cloud Storage.

Archival

Another interesting use case is archival of older VM’s, Templates, or infrequently used VM’s such as test environments.  In these cases, you can use Storage vMotion to migrate the datastore for these VM’s from your primary SAN to Cloud Storage.  They will still appear as though they are local to your ESX server, but the data will be kept resident in offsite Cloud Storage.  If you need to use them at any time, they can either be used with their datastore on Cloud Storage or they could be migrated back for the time they are needed.  You can also Clone VM’s, or Clone VM’s to a Template with the target being cloud storage volumes.

Migration/Replication

By now you are seeing that Cloud Storage can be an excellent way to store VM’s you don’t need immediate access to, either by using a backup product such as VDR or by simply archiving your VM’s.   But there is another valuable use for Cloud Storage use as well; as a migration “swing” set.  For example, if you have some VM’s that you want to transfer between datacenters, or even between ESX hosts, you can use Cloud Storage as a place to first write the VM files from one host or site, and then read them to another host or site.

Pay as you go, provision it yourself

Remember, Cloud Storage is billed on a pay-as-you-go basis.  If you need a few TB of storage to do a migration, you will only pay for the storage you actually use until you destroy it.  Additionally, it is elastic and can be self provisioned.  You don’t have to make a request to a Storage Administrator to provision and assign the storage resources.  A VMware administrator could easily manage this themselves.

Is Cloud Storage ready for VMware?  Absolutely, and so are you.

As Emily Litella would eventually say, ”never-mind”. (http://en.wikipedia.org/wiki/Emily_Litella)

I am of course referring to Cloud Security.

You hear about it everywhere, including in conferences and webinars from Cloud product vendors who feel the need to constantly reassure you, which probably just scares you even more.  It’s really becoming a virtual symphony of fear.  It’s all really, really, overheated as well.

The simple truth is that if there is such a thing as Cloud Security, it has a long way to go before it can stand on its own; well, in the sense that it is a discipline in itself with unique tools anyway. Some vendors are trying to adapt existing tools, but it is largely just a beginning effort at this stage.

The Information Security controls, processes, and policies that apply to Cloud Computing and Storage are really no different than those you should be using in your enterprise today to protect your data center and IT operations.  Just because a local site has key card access doesn’t make it secure.  Just because you store your data in the Cloud doesn’t make it insecure (far from it, it is probably more secure if you do it right).

Perhaps these vendors believe that physical security, access controls, application and systems security, regulatory compliance, or CIA (Confidentiality, Integrity, and Availability) don’t matter to your organization until you put data in the Cloud?

Yes, of course interfacing with Cloud services will open new attack vectors and expose you to new risks, and you need to pay attention to those.

But so will sending your backup tapes to offsite storage, or allowing remote access to distance workers, or allowing portable media, or smartphones, or social media, or a hundred other avenues that technology opens up.  You do have to be ever vigilant and adapt as new technologies are introduced.

A recent case in point is the story of the South Shore Hospital in Weymouth, MA.  According to reports, about 800,000 records were lost on their way to being destroyed at an offsite vendor. These records “could include people’s names, addresses, phone numbers, birth dates, Social Security numbers, driver’s license numbers, medical record numbers, patient numbers, health plan information, dates of service, and information on diagnoses and treatments. For a very small subset of people, bank account and credit card information was included”  Source. .

It’s kind of ironic that the very act of securing data by destroying it could lead to a breach. But this is the world we live in, and why we need to look at all systems, not just those that are in vogue because bloggers and vendors talk about it.

For the record, if this data had been encrypted and stored in the Cloud, we wouldn’t be reading about this right now. It never would have happened.

You don’t have to cower in fear and read every nightmare scenario a vendor gets the urge to write about.   Unless you want to of course, some of us actually enjoy that.

—–

As a footnote, if you are genuinely interested in the Security implications of Cloud Computing and Storage, the kindly folks at the Cloud Security Alliance have published a very good paper titled Security Guidance for Critical Areas of Focus in Cloud Computing.

What they have sensibly done is looked at each area of Information Security and analyzed them for applicability and relevance to Cloud computing environments. They haven’t invented anything new, and they aren’t trying to scare anyone. That’s the way it should be handled.

I’ll leave you with a quote from the paper itself:

Security controls in cloud computing are, for the most part, no different than security controls in
any IT environment. However, because of the cloud service models employed, the operational
models, and the technologies used to enable cloud services, cloud computing may present
different risks to an organization than traditional IT solutions.

Cloud computing is about gracefully losing control while maintaining accountability even if the
operational responsibility falls upon one or more third parties.