Mitigate and Manage Big Data Disaster Effectively
It is an exciting time for conducting business operations that utilizes technology to add value to your customer’s expectation and delivery of service. All over the world, technology is being juxtaposed upon out of box thinking. As a result, we see small yet interesting streams of innovation flowing from various quarters all around the globe. Be it the rise of mobility, the value proposition of cloud computing, or even the growing influence of social media on business fortunes, it is truly awe-inspiring to see the emergence of these technology disruptions that didn’t even exist some 15-20 years back.
Whatever be the means, the end objective has always been to capture newer territories and grow business. One key business driver has been the way opportunities in tech evolution can be utilized to serve our customers better, by uncovering insights from seemingly random pieces of data. This is precisely where big data comes into the picture and changes things radically for business owners and IT heads across the world.
How well protected is your big data
As is evident, big data helps you unravel gold standard insights that help drive your business further. However, beneath the hype and hoopla around big data, is the very real need for a radical shift in work culture and technological infrastructure. Both of these crucial factors are found to be severely wanting in many organizations looking to tap into the rich value offered by big data. In this scenario, it will not be uncommon to face the possibility of a crushing data disaster in form of data outage, data corruption, or loss of data.
Do you too agree?
If your answer is ‘Yes’, then it’s time to take matters in your hands and ensure that you have a well-defined, clearly laid out, disaster recovery plan for your valuable and voluminous big data. Just having ‘Big Data’ as a buzzword flying around in your office without concrete plans to protect its structural and inherent integrity will be nothing but a white elephant proposition for your business.
What’s holding us back in ensuring top notch big data disaster protection?
A very interesting experiment was undertaken by Lockwood Lyon to understand organizational preparedness for big data disaster mitigation and recovery. He managed to uncover some startling facts:
- Many IT heads felt that big data was not mission critical, but merely to be used for analytics purposes
- Because of this, they didn’t feel the need to implement a disaster recovery plan
- The volume of data stored is humongous; this calls for huge space and infrastructure. This makes periodic backup expensive
This is precisely where most thought processes deviate from the obvious USP of big data – i.e. it is mission critical enough to drive the fortunes of business. Imagine these steps that shows how the stocks of big data continues rising from the initial set up of big data infrastructure, till it becomes absolutely essential for management decision making –
Step#1 – Your organization will setup big data analysis engine as a new initiative.
Step#2 – Your business analysts start utilizing the system by deluging the system, with their queries on data collected through processes such as data crawling.
Step#3 – Out of the many, many queries, comes some valuable information that guides the top management in reducing costs, improving margins, or enhancing efficiencies. They can now frame revised growth objectives bolstered by the presence of the on-site big data system.
Step#4 – The resulting value encourages analysts to pour in more queries and more number crunching. These single-run queries now take shape of valuable and clearly defined processes to be carried out weekly or monthly; its value increasing with each iteration.
Step#5 – This way, the management gradually becomes dependent on big data for their success in operations and strategies. This aids the management to classify this engine or system as mission critical to their revised growth plans.
Thus, considering the endgame, it is necessary to consider big data as mission critical and ensure concrete steps are taken to prevent issues with big data integrity in the event of a data disaster.
Effective big data disaster management
Below are some key points to consider when setting up big data disaster recovery and management:
- Regulatory requirement – Your big data disaster management plan needs to comply government and relevant regulatory body mandates. Some companies like knowledge and information aggregators, who look at long term trends, may keep even decades-old data, while this is not necessary for other type of companies. Blend in your critical time period for retaining data, with regulatory compliance essentials. This detail will help you to correctly determine how many years back you need to go, when preserving data.
- Recovery point – Big data is a culmination of various well thought out processes. First, the use of a web data extractor collects relevant and targeted data from diverse sources. This is then stored in a data warehouse in a planned manner. From here, it is passed through an ETL engine (extracted, transformed, and loaded) to be used by BI and big data analytics tools for uncovering insights. Now, coming back to big data disaster recovery, you need to be sure what will be your recovery point in case of an outage or data issue. Will it be the initial raw format that data comes in from various sources? Or will it be a more refined form of data that has passed through the ETL process? A larger proportion of corporate entities involved in big data disaster management will choose the second option. It is worth noting that this is just like transactional data – where the point of recovery will be nearest to when the stoppage or trouble happened in the transaction. However with big data, it is also necessary to determine the ‘form’ in which you need the data to be recovered.
- Speed of data recovery – In order to derive its full potential, management rightly needs big data analytics to be carried out in near real time. This calls for quick fire recovery in case of any trouble. IT heads might look at cloud storage options to enable this. They can also look to bolster their on-site storage choices. This can be done either with a slower media such as tape drive that has time consuming recovery, or go for continuous replication on in-memory storage on more than one data server.
- Priority vs. non priority – Big data is humongous. You need to be absolutely clear on what data takes priority for recovery, in case of an unforeseen disaster. It’s unnecessary and expensive to try and recover ALL the data at once. Mission critical, time critical, or rapidly changing data needs to be on top of your priority list, followed by other data constituents that change or get updated less rapidly. Classifying various clusters into these two types of data on priority needs to be a consensual affair. This way the top management, operations heads, IT admins, and other stakeholders can mutually decide on what should be recovered first and what can wait for a bit longer.
- Enforce data governance – The classic quip of ‘Prevention is better than cure’ becomes very applicable for big data disaster recovery. While we may have strong process oriented approach for related activities such as data crawling, we lag behind when it comes to implementing strong protocols for big data. Having this in place will help with data disaster management to a great extent. Important components worth considering are assessing the data provenance (source and metadata about the data) and then deciding on how to use the data in your analytics. This is not something that you can simply plug and play. It has to be ingrained into the way a web data extractor works. This helps make certain that you consider this crucial pointer from the very outset of data collection and later on, its analysis.
- Practice makes it perfect – All the initial brainstorming and devising of big data disaster management plans will bear fruit only with its successful long term implementation across the organization. All stakeholders (both, internal and external) and IT executives need to be taken through the plan and told of their roles and responsibilities in context of the larger picture. It needs several rounds of testing and coordination with end users (individual as well as department level) and external vendors.
The involvement and adoption will be hesitant from them initially, but don’t lose hope, because this is the foundation to your disaster management initiative success. You will see that they will eventually support this planning and preparation process when they witness the immense value provided by a properly planned big data disaster management process.
These pointers will help you ensure that your big data continues to denote integrity, accuracy, and relevance to help management with insight-based decision making. Do write in to us and let us know what consideration have you factored in for ensuring the safety of your hard earned big data insights.
Interested to protect your big data against unplanned situations? Then connect with us today and we will be glad to assist you with our industry expertise.