Web Scraping reveals top tech trends and company’s media mentions in 2017
Last Updated on by
Web data extraction or web scraping can be a great business tool for trend spotting via media monitoring. Essentially, leading media outlets can be tracked to unveil the top buzzwords and the number of mentions companies (including their products) garner over specific time period. We wanted to apply this method to understand the tech landscape and its coverage in 2017. Hence, we deployed PromptCloud’s in-house web crawler to extract the article titles from two popular outlets (TechCrunch and VentureBeat) and performed text mining on the dataset to uncover the top buzzwords, companies and products.
The dataset is very simple and it contains only the following three fields:
- Article title
The last article was crawled on 7th December.
The goal of the analyses can be divided into two sections:
- Top companies/products that were covered by media over the year
- Top tech trends over the year
Let’s now get to the interesting part of the study, i.e., the result and the context associated with the findings.
Top companies and products
Given below is the tabular representation of the top companies and products that garnered maximum number of mentions:
|This year focus was on AI and Machine Learning, be it Google Photos, Google Assistant, G Suite or Google Home. They also introduced Google Lens which uses a machine learning technology to identify the objects around the world. Clearly, they are on their path to become “AI-first” company.|
|Apple||Apple announced Homepod speaker to take on Amazon Echo and Google Home. Launched new ARKit to let developers build augmented reality apps for the iPhone. This year we also saw launch of iPhone 8 and iPhone X apart from other Apple products. The news is that they are integrating machine learning and augmented reality with the iOS 11.|
|The company announced tons of its new projects and features in 2017. Camera’s augmented reality platform- suite of creative tools. Also launched Facebook spaces-a social virtual reality platform that relies on connecting your Facebook account with Oculus gear.|
|Amazon||The main areas of focus was the security, as Amazon strive to create environment for their customers. They released slew of security features for AWS apart from various other AWS services centered around advanced analytics, machine learning, AI, NLP and more. Their retail business also soared with the highlight being acquisition of Whole Foods.|
|Uber||This year Uber acquired selfdriving truck company Otto which landed in legal trouble with Alphabet’s Waymo. They also started the year with a $20 million fine from the Federal Trade Commission (FTC). Aprt from that they got into workplace related controversies after their former engineer Susan Fowler highlighted company’s adverse culture in a blog post. Other highlights include their merger with Chinese rival Didi Chuxing and announcement of flying car plans for Dubai and Texas by 2020.|
|Microsoft||Microsoft is introducing artificial intelligence and machine learning (ML) services within the Microsoft ecosystem including the Visual Studio Tools and Microsoft Azure. Now Azure IoT Edge Public Preview is available, enabling Artificial Intelligence (AI) and advanced analytics to be built and deployed on the “Edge”. Azure Machine Learning is being updated to provide integration with the Azure IoT Edge to deploy Machine Learning models on “Edge” devices.|
|From the alarming possibility of Russia manipulating US public opinion, to the #MeToo hashtag that acted as a catalyst for women to speak up about sexual assaults, Twitter has been at the center of memes and movements that gave the society jolts in 2017.|
|Tesla||Designed and built as the world’s first mass-market electric vehicle -Model 3, it is a critical step in its mission to accelerate the world’s transition to sustainable energy.|
|iPhone||This is the flagship Apple product that managed to remain in the top 10 by the virtue of new product launches and major overhauls.|
|Android||This mobile OS developed by Google stays at tenth position owing its massive adoption by the major device manufacturers. The iconic phone maker Nokia also came up with new Android devices.|
The other notable companies and products that could not make it to the top 10 are Samsung, Alexa, Snapchat, YouTube, iOS and Netflix. Let’s now check out how the media mentions of the top five companies changed over the four quarters.
Clearly Google is leading the pack with highest number of mentions and Apple got closest to Google in the third quarter. This can be mostly attributed to the Apple event that was held in September. Uber got into the top five surging past Microsoft mostly because of the newsworthy controversies — from bribery and lawsuits to leadership change and hostile workplace culture.
Top tech trends
Given below is the tabular representation of the top 10 tech trends in 2017:
|app||Clearly exponential increment in the usage of various types of apps starting from web and mobile to connected products and AR/VR devices made sure that this is numero uno tech trend.|
|data||Data has already become a major growth driver for most businesses. There was a shift towards stateful applications that leverage big data, and the incorporation of machine learning approaches which use large of amounts of historical data to better understand the context of newly arriving streaming data. Incorporation Internet of Things (IoT) and wide adoption of AI techniques makes data a top trend which will continue its position in 2018 as well.|
|mobile||Location based services will continue to evolve and will provide users real time information. A greater number of apps will start using AR thus making applications more effective and productive. Android instant apps will become a common trend. Health, education, security, smart home companies and many more have started using mobile enabled technologies and applications.|
|video||In 2017, bandwidth dedicated to video traffic was estimated to make up 74% of internet use. Hence, live streaming will undergo strong growth. More broadcasters are moving into live streaming of Over The Top video. Virtual reality, especially VR streaming, is becoming more widespread. Virtual Reality video is likely to be the future platform of choice. In the meantime, 360° video is a bridge technology that is coming into its own.|
|car/selfdriving car||Connected cars have already arrived and self-driving cars are not far away. The upcoming driverless cars will need drastic changes to our streetscape, including fresh traffic management strategies and a review of the parking spaces.|
|cloud||2017 saw a rise in cloud-run business apps which now have a share of 70% in the enterprise market. There is also a gradual shift towards hyperconverged cloud infrastructure (HCI) which provides the ease of managing integrated technologies using a common toolset. Another interesting trend is the increasing adoption of cloud containers like Linux among top companies like Google, Facebook, Microsoft as well as smaller organizations alike.|
|smart (devices/home)||Smart devices like Amazon Echo and Google Home went mainstream in 2017 with the companies making these devices available globally as well as improving upon their existing features and adding new ones. Amazon’s cloud cam was one of the hottest smart home devices launched this year. It not only worked as a home security camera with unlimited storage but also came with the provision of letting amazon personnel drop your shipments inside your house by working together with Amazon’s own smart lock.|
|social||Augmented reality, live videos and stories were some of the top trends among social media users in 2017. There has been a rise in brand participation in messaging apps like Facebook Messenger, Whatsapp and the like. Influencer marketing gained more traction during 2017 and is one of the go to medium of marketers. After failing to make any significant growth in 2017, Twitter increased the character limit of tweets from 140 to 280.|
|security||2017 saw businesses taking data security more seriously than ever before. WannaCry was one of the biggest cyber threats that we saw in 2017 which rendered critical facilities like hospitals, government agencies and individual computers useless, causing widespread harm. Security has indeed become a rising concern for enterprises holding huge amounts of data and the efforts to tackle this has already begun.|
|machine learning||The machine learning space saw groundbreaking innovations and development particularly focused at areas like space exploration, healthcare, finance and autonomous driving. Autonomous driving is fast gaining momentum with top players like Tesla, Uber and Google spending significant amount of resources to achieve perfection in this field.|
Given below is the chart that shows the movement of the top ten tech trends over the four quarters.
Looking at the top three trends, i.e., app, data and mobile, it is evident that first quarter gave maximum spike for ‘app’ and ‘mobile’ with data remaining consistent (since last quarter has data till 7th December).