Guest Post: This is a guest post on data backups from Josh Reichardt, DevOps Engineer with about.me and the owner of Practical System Administration, where he writes about scripting, devops, virtualization, hardware and policies. Follow him on Twitter at: @Practical_SA.
The topic of backups always seems to strike a chord with folks that work in operations. It can be a touchy subject for people that have been bitten in the past either by the lack of good backups or a complete lack of any backups at all. However, it is easy to avoid bitterness if you follow sage wisdom from other sysadmins who have either already experienced the pain of not having a backup or have dedicated themselves to always having a copy of their data.
I have experienced some stressful situations throughout my career, spending days on end digging through old tapes, hoping and praying to find one tiny file from months and/or years ago that for whatever reason is all of a sudden critical to have. It is not fun at all to experience data loss or to go on the old data hunt, hoping you will find what you are looking for.
That said, many of these tips are derived from my own personal experience working throughout the years with important company data and backups. Hopefully, some of these tips can help other system administrators protect themselves against data loss and data failure scenarios with smart data backups.
Get A Good Data Backup
This one should be obvious, but getting good backups is an invaluable asset to you as a system administrator, and can be a great bargaining piece, if necessary, for any sort of fallout or political battle that may arise. Oftentimes, backups are overlooked by IT staff, so by ensuring you have good backups (you must always test them!) you are protecting yourself and are able to deflect blame elsewhere as well if something out of your control occurs.
As a bonus, you look like a hero when the CEO or president of the company needs emails from a month ago and has no idea where to turn or to start looking, you will look like a magician and could potentially help strengthen their view of IT.
Getting a good backup is difficult, however, if the target is moving. For example, it is extremely important to speak with the business side of the company to identify and pin down exactly what the key pieces of data are that should be backed up. Assuming you can get this type of thing in writing by the key parties and owners, then it should be much easier to use later on down the road to cover yourself if certain circumstances were to occur. This type of strategy works almost like an internal SLA.
Some ideas for these types of backups would be things like key databases, mailboxes, mailbox databases, internal file shares and/or user PCs.
The other important detail to identify is how often these types of data should be backed up. Daily? Hourly? Some backups may only need to happen once a week. It is important to be very thorough so both parties are aware and understand what to expect from the backup lifecycle.
As mentioned, backups are almost always a moving target, so it can be difficult to keep track of what exactly needs to be backed up. In addition to the key business data, to make your own life easier in day-to-day operations, it is a good idea to backup all of the internal systems that are important, just in case. It is best to err on the side of being over prepared, so making sure to backup everything will oftentimes be the best decision, especially if you have the extra resources to justify backups of everything. It can be a challenging task to back up so many things; one good strategy is to create a checklist and make sure to keep it up-to-date with things that either need to get backed up or things that aren’t getting backed up but should be.
Bonus points for automating the process of backups.
As the saying goes, expect the best but prepare for the worst. If at all possible, simulate failures or scenarios where data loss occurs and attempt to recover the data. These types of “drills” are great for exposing where the whole backup/restore process is broken. The more practice you get, the more prepared you will be for when the inevitable data failure occurs.
Bonus: Automate the process of restoring some files on a daily or weekly basis and have the backup report to you if something breaks. Often, backups have the tendency to work for awhile and then suddenly stop working out of nowhere. Knowing when the backup breaks can be invaluable.
Different Strokes For Different Folks
One observation that I have made throughout the years and across various jobs and industries is that business requirements tend to vary wildly. For example, the backup and retention needs of a startup company will be far different from the needs of an enterprise company. Being able to identify the needs and quickly adapt your backup strategy (without spending too much money) is an invaluable skill to have.
Different Tools For Different Jobs
I don’t want the following to come off as a blanket statement, but more of a general rule of thumb that I have noticed in my career, so it will not apply to everyone. Startup companies, in general, tend to live in the cloud much more than enterprises. Therefore, the tools for backing up these types of environments are usually very different due to the somewhat different architecture and choices
Looking at cloud-based companies, the requirements tend to change much more rapidly, therefore it is usually harder to pin down what exactly needs to be backed up. S3 object storage tends to be common because of the high uptime and data resiliency. Cold storage is also popular, with technologies like Glacier and Nearline. Another feature of cloud-based services are the nice built-in backup features. For example, RDS offers options for automatic database backups, which are very easy to manage and are usually just a few button clicks. Another common form of backups in cloud-based infrastructures is block storage imaging and snapshotting. The cost of imaging servers and drives is very cheap so there is really no excuse not to use snapshotting.
In more traditional enterprise-type backup solutions you usually hear about SAN arrays, deduplication and compression of backups as well as dedicated backup storage arrays and tape devices. As stated, the data requirements are often very different, so it isn’t uncommon to see things like mailboxes and databases getting backed up to a tape device and being held onto for years and years. Or things like backing up file shares for a certain number of days and then dumping to tape to be held onto for extended periods of time. Some enterprises also backup user PCs, which may be required to backup to yet another storage device.
There are many many tools to address the enterprise backup landscapes. For example, for VM backup and management, Veeam can be a good go-to solution. Other popular enterprise backup software includes EMC and Commvault products. Some prior experiences or lack-of-use prevent me from recommending Symantec Backup Exec and HP Data Protector.
Successful Data Backups = Knowing Your Environment
As you can see, backup architectures and solutions can quickly get complicated. Each environment requires different backup architectures, so knowing what to backup and to where is a very important skill to possess. Naturally, the skill of backups can only be grown through time and experience, so getting to know the tools well is important.
Backups are a challenge to master. It is difficult to get backups right because they always seem to be a moving target and require seemingly constant care and attention. At the same time, doing backups correctly is extremely important and can lead to company-wide conflicts if handled incorrectly.
Because backups are such a dirty job to take care of, and, because of this, are often avoided by seasoned sysadmins or handed off to somebody else. But, with a good attitude and buy-in from key business personnel, owning the backup process and lifecycle is well worth the trouble of getting good at. If something disastrous were to happen and you have confidence in your backups and a good place to recover from, then you have a good chance at saving the day and many headaches.
Do You Need an Enhanced File Server?
SmartFile’s FileHub™ gives you an enhanced file server that consolidates legacy hardware, FTP and file sharing tools with data backup, high availability and failover options. Chat with sales to get more info.