Using Data Crawling, DaaS & Analytics to exploit the Internet of Things
Considering the exponential explosion of Internet-enabled devices and the quantum of data spewed by these devices, it is no surprise that the Internet of Things (IoT) and Big Data feature at the top of the closely-watched Gartner hype cycle which has been known to give a good indication of the specific stage that a particular technology (or product) trend is at.
Similarly, the strategic importance of Big Data analytics for businesses can be gauged from a recent study by General Electric and Accenture that delved into the impact of the convergence of Big Data analytics and the Internet of Things According to this study, such convergence is altering the competitive landscape of entire industries. Organisations that do not develop an appropriate strategy to leverage Big Data analytics could risk their growth, a fact acknowledged by almost 89% of the participants in the study. This in turn has lead to many DaaS providers to offer innovative web scraping and data crawling solutions, thus supplementing the data-driven future of IoT.
These DaaS providers are a key link in the Big Data and analytics chain and function to bring data (unstructured or structured) from the web. Scraping URLs and extracting e-commerce data feeds is just the tip of the iceberg in the Internet of Things.
The consumer and Industrial Internet of Things
IoT is naturally associated with consumer applications and in all probability the consumer segment will see its greatest impact. According to David McLauchlan of an IoT analytics company, “for the first time, companies that produce consumer goods have a window into how customers are using their goods; where; and even what kind of issues those customers are running into.” The result is that data analytics becomes useful and presents opportunities for companies that traditionally wouldn’t have anything to do with Big Data.
The range of industries that are impacted by the concept or philosophy of IoT is staggering: from insurance firms to power/ utility companies and autonomous vehicle manufacturers, to cite a small example. Enterprises across domains are now looking to scrape valuable data off the Internet and gain the analytics advantage on the data crawled and extracted.
Popularly, data aggregation now demands URL scraping for Travel Websites, reviews, Hotel Pricing details, all of which is use in Data analysis for business Intelligence
The world of Industrial Internet of Things (IIoT) just became real. Writing on Wired.com, Matt Cicciari draws an interesting distinction between the consumer and industrial IoT. “Unlike consumer-based IoT that is trying to devise a way to make your world a better place by telling you when your washing machine needs service or letting you control all of your home’s systems while you are away, the IIoT is working to make our collective world a better place by improving the monitoring, control and safety of everything around us. In short, reducing risk and improving the reliability of our massive industrial systems.”
In the automotive industry itself, IoT-related improvements is projected to result in $19 trillion in economic benefits with enormous value generated by Big Data analytics. As Andreas Mai, director of Smart Connected Vehicles at Cisco Systems is quoted in this article by Drew Robb, “Big Data is the fuel of the connected vehicle. It is analytics which gives you the true value certainly one of the.”
The data management challenge
Nevertheless, such a wide canvas (in terms of industries impacted) presents data management challenges that create the opportunities for Big Data analytics to flourish. “A key challenge with IoT is data management: determining what type of data is important, what should be transmitted immediately, what should be stored and for how long, and what information should be discarded. Otherwise, you could end up with an almost infinite pile of data to analyse, when only a relatively small portion is of real importance. Some data just needs to be read and thrown away,” says Moin Khan, Executive Director of Product Marketing Management at AT&T in the same article above. This view is echoed by Rob Rich, managing director of the TM Forum’s Insights Research, who says that “… while operators see the value of analytics for IoT, they are still wading through the sheer volume of it. I think a lot of focus is around how much data you need to keep.”
These statements clearly point to the need for making significantly more progress on the software side of things. The current discussion on IoT is perhaps dominated by hardware or devices, but as many organisations already realise the greatest value will be derived when the software – including data analytics capabilities- side of the equation becomes equally, if not more, powerful. Matt Cicciari in his article on Wired.com also highlights the urgent need for software to be part of the discussions on the IIoT.
The need for standards and a strategic imperative
The realm of IoT also needs to have some standards in place and the need for interoperability, sooner rather than later, write Thomas Davenport and Sanjay Sarma. They call for a slightly different approach to IoT than when one of the earliest precursors to the IoT took off: RFID. One of these is the need to “leverage bottom-up adoption as well top-down standard setting.” Businesses will also have to use a “more carrots than sticks” ploy to speed up widespread adoption of IoT, they say.
In conclusion, it is imperative for organisations to estimate the effect of the Internet of Things on their business as it necessitates the creation of a robust infrastructure to support the volume, velocity and variety of data that they will have to contend with. While there is a not-so-insignificant cost associated with it, the price the business pays for not doing so could be many times higher.