The topic of backups always seems to strike a chord with folks that work in operations. It can be a touchy subject for people that have been bitten in the past either by the lack of good backups or any backups at all. However, it’s easy to avoid bitterness if you follow sage wisdom from other sysadmins who have either already experienced the pain of not having a backup or have dedicated themselves to always having a copy of their data.
It’s not fun at all to experience data loss or to go on the old data hunt, hoping you will find what you are looking for. Here are some tips that can help system administrators protect themselves against data loss and data failure scenarios with smart data backups.
Get A Good Data Backup
This one should be obvious, but getting good backups is an invaluable asset to you as a system administrator. Often, backups are overlooked by IT staff, so by ensuring you have good backups (and testing them), you’re protecting yourself if something out of your control occurs.
As a bonus, you look like a hero when the CEO or president of the company needs data from a month ago and has no idea where to turn or to start looking. You may even help strengthen their view of IT.
Cover Yourself
Getting a good backup is difficult if the target is moving. For example, it’s extremely important to speak with the business side of the company to identify and pinpoint the key pieces of data that should be backed up. Assuming you can get this type of information from parties and owners in writing, then it should be much easier to use later on down the road to cover yourself if certain circumstances were to occur. This type of strategy works almost like an internal service-level agreement.
Some ideas for these types of backups would be things like key databases, mailboxes, mailbox databases, internal file shares and/or user PCs.
The other important detail to identify is how often these types of data should be backed up. Daily? Hourly? Some backups may only need to happen once a week. Be very thorough, so both parties are aware and understand what to expect from the backup lifecycle.
Be Thorough
As mentioned, backups are almost always a moving target, so it can be difficult to keep track of what exactly needs to be backed up. In addition to the key business data, to make your own life easier in day-to-day operations, it’s a good idea to backup all of the internal systems that are important, just in case. It is best to err on the side of over-preparation, so making sure to backup everything will often be the best decision, especially if you have the extra resources to justify it. It can be a challenging task to back up so many things; one good strategy is to create a checklist and make sure to keep it up-to-date with things that either need to get backed up or things that aren’t getting backed up but should be.
Bonus points for automating the process of backups.
Be Prepared
As the saying goes, expect the best but prepare for the worst. If at all possible, simulate failures or scenarios where data loss occurs and attempt to recover the data. These types of “drills” are great for exposing where the whole backup/restore process is broken. The more practice you get, the more prepared you will be for when the inevitable data failure occurs.
Bonus: Automate the process of restoring some files on a daily or weekly basis and have the backup report to you if something breaks. Often, backups have the tendency to work for a while and then suddenly stop working out of nowhere. Knowing when the backup breaks can be invaluable.
Different Strokes For Different Folks
Business requirements tend to vary wildly. For example, the backup and retention needs of a startup company will be far different from the needs of an enterprise company. Being able to identify these needs and quickly adapt your backup strategy (without spending too much money) is an invaluable skill to have.
The requirements of cloud-based companies tend to change much more rapidly, therefore it’s usually harder to pin down what exactly needs to be backed up. S3 object storage tends to be common because of the high uptime and data resiliency. Cold storage is also popular, with technologies like Glacier and Nearline. Another feature of cloud-based services is the nice built-in backup features. For example, RDS offers options for automatic database backups, which are very easy to manage and are usually just a few button clicks. Another common form of backups in cloud-based infrastructures is block storage imaging and snapshotting. The cost of imaging servers and drives is very cheap so there is really no excuse not to use snapshotting.
In more traditional enterprise-type backup solutions, you usually hear about SAN arrays, deduplication, and compression of backups as well as dedicated backup storage arrays and tape devices. As stated, the data requirements are often very different, so it isn’t uncommon to see things like mailboxes and databases getting backed up to a tape device and being held onto for years and years. Or things like backing up file shares for a certain number of days and then dumping to tape to be held onto for extended periods of time. Some enterprises also backup user PCs, which may be required to backup to yet another storage device.
There are many tools to address enterprise backup landscapes. For example, for VM backup and management, Veeam can be a good go-to solution. Other popular enterprise backup software includes EMC and Commvault products. Some prior experiences or lack-of-use prevent me from recommending Symantec Backup Exec and HP Data Protector.
Successful Data Backups = Knowing Your Environment
As you can see, backup architectures and solutions can quickly get complicated. Each environment requires different backup architectures, so knowing what to backup and to where is a very important skill to possess. Naturally, the skill of backups can only be grown through time and experience, so getting to know the tools well is important.
Backups are a challenge to master. It’s difficult to get backups right because they always seem to be a moving target and require seemingly constant care and attention. At the same time, doing backups correctly is extremely important and can lead to company-wide conflicts if handled incorrectly.
Backups are such a dirty job to take care of, and because of this, they are often avoided by seasoned sysadmins or handed off to somebody else. But, with a good attitude and buy-in from business personnel, owning the backup process and lifecycle is well worth the trouble of getting good at. If something disastrous were to happen and you have confidence in your backups and a good place to recover from, then you have a good chance of saving the day and many headaches.