To conduct extensive website searches for new teams across tournaments, age group, state, age, class and season. Minute details like the team name, division, city, state, manager were also required. Additionally, tabulated Game and team scores, synchronized to suit the game and team data were required. This vast database of list of teams constantly needed to be updated. Critically, time was of utmost importance to keep game scores up-to-date with tournament progress. Client had already “dumped a lot of money into developing this website” and “wasted too much time on this already with companies that say they can do it”. Client was disappointed with sub-par extractions and incomplete data deliveries.
The sooner we proved our product and technical proficiency, the better chances of converting the client. The way forward was to prove this with a proof.
From the website, we processed the combo boxes correctly and iterated the extraction from team through states and scores and other details. Initial data quality made an impression on the client. We then hit a snag, we found many teams in the event results and score data but didn’t get the team info. Meticulously, we followed links on the event results score data for that team and then got the team information. Separate CSV files for team data and game data was compiled for the proof. Credit also goes to the client’s timely feedback on the crawling process. In fact, client had discovered solutions to some of the problems, but these were mitigated without any hand-holding much to the delight of the client. Data delivery (format, mechanism, and quality) impressed the client to come on board for regular crawls.
- Successfully crawling disparate data schema independent of sources and search results
- Crunching large data volume helped unlock insight improving pipelines, crawl schedules and data delivery
- The competitiveness of this challenge was another takeaway. To know that the client tried many other DaaS providers and tools proves that USP for a DaaS provider lies in technical competence.
- Sports and stats go hand and hand, which is perhaps why the sports is ahead of the curve in adopting analytics to improve performance. This was a great project to gauge where we stood when it came to catering to large and dynamic advanced analytics-driven data-requirements.