Photo Credit: RPI
- Data is everywhere, but not all data is created equally. About 2.5 quintillion bytes of data are generated each day.[1] However, the vast majority of this “Big Data” is unstructured and unlabeled, preventing businesses and institutions from harnessing the full power of Analytics and AI to drive decision making.
- Because of its unstructured nature, Big Data requires significant preprocessing, the scale of which prohibits all but the largest of organizations from building and refining novel datasets that save companies like Netflix $1 billion a year in customer retention.
- It is estimated that “a third of business intelligence professionals spend 50% to 90%” of their time preprocessing data with the remaining time left for model building and evaluation.[2] Given that a 3TB Big Data project costs $1 million for each month it is active, $500,000 per month is out the door before the “real” analysis work begins.[3]
- This may explain why 60%-85% of big data projects are never completed, despite that enormous 130% ROI such projects yield organizations compared to competitors not using big data.[4][5]
- The “Big Data” structuring problem is relatively new, which is one of the reasons that it remains open-ended. Right now the solution is to throw more processors and people at it.
- However, with the rise of IoT and Web 4.0, companies need to structure and mine this data efficiently to remain competitive in the 21st century. The current data engineering techniques and platforms are incapable of doing this efficiently, making the need for a new solution urgent.
Sources:
- https://techcrunch.com/2017/07/21/why-the-future-of-deep-learning-depends-on-finding-good-data/
- https://www.zdnet.com/article/big-datas-biggest-problem-its-too-hard-to-get-the-data-in/
- https://www.cooladata.com/blog/true-cost
- https://www.techrepublic.com/article/85-of-big-data-projects-fail-but-your-developers-can-help-yours-succeed/
- https://existek.com/blog/big-data-solutions-development-cost-example-software/
johnfantell 