Brain Storm for Big Data – Data Mining

Big Data

Big Data, With the developing technology, the number of smart devices and applications used daily is increasing rapidly. Depending on this increase, problems arise regarding the storage and management of large amounts of data to be stored.

Since 2010, in parallel with the development of Cloud Technology, although the storage problem has been largely solved, problems related to the processing of this stored big data and making it interpretable in a meaningful way have begun to manifest themselves.

In this article we will brainstorm together on Big Data. As an example, let’s talk about a few companies that use Cloud Technology to the fullest. If we take Facebook as the first example.

Facebook.Com, one of the leading users of cloud technology, provides most of its server and storage infrastructure from Akamai Corp.

In fact, I came across such an incident recently, when I followed the virtual Windows 2012 Data Center Edition updates through Firewall, I witnessed that Microsoft was pulling the update files from Akamai.Net. Moreover, I saw that Akamai was downloading from its servers deployed in Turkey.

When I resolved the DNS address, I confirmed the IP address as Facebook, which means that Microsoft and Facebook meet on the same physical server but are broadcasting from different virtual servers.

To briefly summarize cloud technology and continue with our article, these are virtual systems that give you the right to use all these things in a certain capacity and for a certain period of time by developing only applications and software without having any system hardware and network infrastructure.

In essence, there are virtual servers running on Cluster servers brought together in a Grid structure. According to the terms of the contract, all authorization and management of this virtual server is left to you and it is provided to serve in the internet environment by assigning the requested bandwidth.

Thanks to cloud technology, IT companies, which have the opportunity to invest more and develop more applications with falling costs and increasing price advantages, have started to accumulate data on a huge scale. However, after a while, despite storing huge amounts of data, this data could not be processed sufficiently, resulting in mountains of worthless data-trash. Then the profession known as“data mining” emerged.

Since data mining requires both system expertise, network expertise and even software development expertise, the number of Data Miners is very small. The world’s leading universities opened departments on this profession and started education and even gave their first graduates. However, universities in our country have not made any breakthroughs in Data Mining.

Data mining and big data management will be the most sought-after professions in the IT sector worldwide, especially in developed countries, for the next 20 years. Businesses and organizations are currently busy storing piles of worthless data. Unless you process this data, you will not go one step further than just being a data porter, so to speak.

IBM’in 2015 Big Data Raporu

It is stated that all data production worldwide will increase by 35 ZB (1 zettabyte = 1.073.741.824 terabytes) annually in the next 5 years and when indexed to the total, it corresponds to a 60% slice.

In addition to this, the number of media files (high-resolution images, audio and HD video files) will also increase in size and is expected to reach 2.7 ZB annually. In addition, if we evaluate the total increase on a daily basis, it is estimated that there will be approximately 5 EB (1 Exabyte = 0.1 Zettabyte) in 2 days all over the world.

Within this big data, the increase in the number of mobile and smart devices is thought to be caused by image and media files. Social media directs users to this, and for this reason, its share in big data will increase steadily in the coming years.

IBM’s 2011 Cost Analysis on Big Data

In 2011, there was a daily data increase of 100 TB, 294 billion e-mails, 230 million tweets and 4.8 trillion ad impressions. With the integration of mobile smart devices into the shopping system, $2.1 billion was spent on ads on mobile devices in 2011, while $83.2 billion was spent on ads displayed on personal computers in 2012.

In 2011, since mobile usage was not as widespread as it is today, there was a gap between spending and investments. Today, however, mobile usage and spending have almost caught up with online usage and closed the gap. The main factor is that users are turning more to mobile apps to spend money and shop.

Special product discounts offered by shopping companies for mobile applications have been effective in directing users to online mobile shopping. Google Seo has preferred mobile application and mobile appearance(bootstrap).

If we take the Video Sharing site Youtube; It is one of the sites that most successfully applies data mining on Big Data. It has a robot that analyzes the videos you watch at the upload stage and determines whether there is inappropriate content or copyright.

Similarly, another robot application determines whether the video sent with the application is action, dramatic or artistic, and while categorizing it, it takes into account the parameters entered by the users as well as the data detected by the robot and the category is determined in this way.

Likewise, the Linked.in social business network makes offers to you based on your knowledge and skills, your education, your work environment and many other parameters. While you can’t access most of these offers with the Standard Entry Package, premium users who pay a license fee can access a variety of offers.

Sometimes it is a job offer, sometimes it is an offer for education, friends, community or line of work. Linked.in is a system that best applies data mining and pays off economically.

I regret to say that Data Mining is still in the model-project or initial stage in our country. For now, the most successful e-government project known as www.turkiye.gov.tr, the Ministry of Justice Uyap project and the Identity Sharing System of the General Directorate of Population of the Ministry of Interior can be counted among Public Institutions.

Data Mining in Private Institutions is mostly seen in the Banking and Online Shopping Sector. Banks make applications such as discounts and money-points according to the type of shopping made by credit card users.

In 2015, the shopping-point-discount application called HOPI can be called the most successful application in data mining. It provides store or customer-specific discounts and points according to the purchases made by member customers.

It should be ensured that departments on Data Mining, which is still in its infancy in our country, are opened in universities or at least applied courses are created, and new training models in this direction should be created in private Informatics Academies.