Table of Contents
show
The immense reach and influence of Big Data over almost all the industry verticals is not unknown. With Big Data Ecosystem, seemingly massive and complex chain of communication, comments, and brand mentions is analyzed at a granular level. The purpose of this exercise is to unlock insights that may have hitherto remained hidden from the views of the decision makers of a company. Take the case of American Express. The card company giant wanted to bring more than just trailing indicators to take its aggressive growth plans ahead. This led AmEx to invest into building a complex yet powerful predictive models that take in as many as 115 variables. The purpose of the exercise? To look at ways to enhance brand loyalty among customers and bring down customer churn with the help of Big Data Ecosystem.
This predictive analysis is one of the forms of Data Science – the field that helps extract knowledge or insights from Big Data Ecosystem (both structured and unstructured). Some other implementations of data science include statistical analysis, data mining, data engineering, probability models, visualizations, and machine learning. Data science is a part of the bigger domain of competitive intelligence, which also includes data analysis and data mining.
A look at propelling the productivity of next- gen data scientists
IBM’s Big Data Evangelist, James Kobielus had produced an interesting article that highlighted the different ways in which the productivity of next generation data scientists can be enhanced. This can, in turn, impact the fortunes of the global economy, finance, and society.
He has acknowledged the mission critical role played by data scientists in providing value to the always-on business environment. Their value spans different repeatable solution integration to help analyze the data and generate meaningful insights to help stakeholders with their decision-making process.
Why boosting the data scientists’ productivity is essential
Data scientists perform a host of varied roles and responsibilities within the entire big data ecosystem. These include tasks such as –
1. Manual
- Designing and developing statistical models
- Analyzing performance of these models
- Verifying the models with the real-world data
- Carrying out the difficult task of conveying the insights in a manner that non-data experts (stakeholders and decision makers) can understand
2. Automated
- Initiation, brainstorming, and research on client business and intelligence gathering
- Data discovery
- Data profiling
- Sampling and organization of data
As is evident, these tasks call for a set of human capital expertise that cannot be found in one single individual. A team of people who are experts in different niches has to be built. More importantly, they have to be aligned such that the business objective of having a team of data scientists is met amicably and without any politics. And this can be achieved by having a robust set of processes and protocols that need to be followed by every single one within the team.
However setting up and enforcing these protocols doesn’t necessarily mean a dip in the productivity of the data scientists. James takes a look at the real life examples where different processes have been set up to ensure optimum productivity of the data scientists within complex team environments. One instance that he has specifically mentioned in this context is Ben Lorica from O’Reilly. This article seeks to offer the below advantages in productivity to the data scientists:
- The provisions of an off- the- shelf API that can be made available to tackle various main and sub steps of the data analysis and visualization domain. Streamlining the end to end process of machine learning processing can help at every single milestone of the project can exponentially improve the reduction in time and cost. And this reduction is far more than the cost involved in on- boarding the software into your organization’s existing systems.
- Data types such as multimedia (audio, video, content) play a pivotal role in streaming media and cognitive computation. With automated machine learning, the absorption and analysis of these types of data can be done easily. Ben suggests going with sample pipelines for speech and computer vision and data loaders for other types of data.
- Applications can help in fast- tracking the training, usage, and perfection of the statistical and predictive models. Examples of such scalable machine learning algorithms include the Spark-based runtimes.
- The productivity of data scientists can also be enhanced by smartly extending the processing pipelines of multifunctional machine learning projects. Examples of such components include incorporating and loading libraries and optimizers. Other instances of these components include the diverse array of data loaders, featurizers, and memory allocators.
It also talks about designing, clearly defining, and setting up error bounds to help check the efficacy of the machine learning projects. With the help of this effort, the actual performance can be measured against pre-defined benchmarks. In addition, it can help in fine- tuning the model if there is a significant diversion of the actual performance of the model from the expected outcomes.
This is one example of the efforts going on worldwide in different organizations to catapult the productivity of data scientists. With these efforts they perform their roles within deeply complex environments that touches multiple personnel, processes, protocols, and expectations.
How to add more to the value provided by data scientists
James then goes on to highlight the ways in which data scientists can rally excel at their jobs and do remarkably well with the data analytics and visualization niche. There are two aspects – one is the technology itself (in the form of solutions like Hadoop, R, Python and Spark) and the other is the team of experts that form touchpoints for data scientists (data application developers, modelers, data engineers, senior management, and ETL experts). Both of them should work in tandem to provide an environment that fosters higher productivity for the data scientists. James has listed quite a few ways to achieve this.
1. Ease of working with multiple data sets –
Take the case of a medical center. It can maintain and store millions of records for thousands of patients. These may include structured as well as unstructured data (pathology images, physician notes etc). A typical Big Data Ecosystem implementation would be to create a Hadoop data lake and harness the data for further use. Another example can be of social media posts and comments that are taken and stored in data clusters. A data scientist must be able to acquire data from such diverse data sets easily. Some of the examples include – data lakes, data clusters, cloud services.
2. Excel in work responsibilities –
Data analytics, predictive modeling, machine learning, data mining, and visualization. These are just some of the many functions a data scientist is involved in. Quite naturally, he/ she would have to do a plethora of activities to carry out the job. This may include one or more of data discovery, aggregation of similar data, weighting of data to match universe, prepare and curate models for deeper insight generation, and formulate, test, and validate a hypothesis. Be it simple structured data or more complex multi-structured data, the productivity environment needs the data scientist to excel across different job responsibilities.
3. Hands on experience –
Provide the data scientists every scope to implement their working knowledge of the big data analytics applications. These may include R, Python, Spark, and Hadoop.
4. Extend their versatility –
As mentioned earlier the data scientists has to interact with many experts in his/ her day to day roles and responsibilities. These include data application developers, modelers, data engineers, senior management, and ETL experts. The touchpoints need to share knowledge on libraries and templates that can help ease the working, and comprehension of topics like machine learning, statistical exploration, neural networks, data warehousing, data transformation, and data acquisition.
5. Monitoring the progress –
A data scientist provides a lot of weightage to devising, designing, and putting into action, processes for handling large- scale data sets to be used for modeling, statistical research and data mining. He/ She also does a lot of ancillary functions like business case development, interaction with third party vendors, managing the lifecycle of the entire data analysis project keeping the team well aligned till the very end, and interacting with stakeholders with regular updates on the progress of the project. Under a conducive environment, a data scientist must be able to track, enforce, and verify the correct functioning of the various components that allow him/ her to do the job right. These components include libraries, modeling, tech integrations, data, algorithms, and metadata.
With these helpful pointers, James brings out the ways in which Increasing the significance of Data Scientists in the Big Data Ecosystem can be made possible.
Planning to acquire data from the web? We’re here to help. Let us know about your requirements.
