BigData, Author at Outsource Bigdata Blog | Page 4 of 5

Master Data Management in Big Data perspective

Master Data Management (MDM) is a method to define and manage all critical data of an organization to one file i.e. master file to provide a single point of reference. To define and manage those critical data, MDM includes the processes, governance, policies, standards and tools. The benefits of MDM increases by increasing number of department, resources and related data. So, Master data is a subset of Big data and while analysis MDM provide a starting point.  Applying MDM gives many benefits while leveraging Big Data.

“The world of big data is a world of unknowns, and you need to somehow anchor it to the stuff you do know and trust — that’s the relationship between big data and master data,” said Ted Friedman, an analyst at consulting and market research company Gartner Inc.

So, MDM put high value while leveraging big data and companies are starting to see. According to survey of The Information Difference Ltd., an MDM consulting and Research Company in London. Only 17% of the 209 corporate respondents in North America, Europe and Asia said they expected big data applications to generate new master data in their organizations. But 59% said they thought MDM hubs and big data systems could be linked together for business uses, including the ability to use master data to automatically detect customer names in sets of big data.

There are several reasons why organizations enhance their Master Data Management (MDM) with Big Data:

More Effective Analytics – Using MDM with big data comes with some great benefits. It provide the basic framework for performing analytics. It brings different new and old information for leveraging. Thus data comes in huge quantity and which makes the analysis more effective.

More arranged data – MDM gives a better arrangement of complex data than normal management of data. Anyhow analyst need to arrange data in right format and MDM help them a lot in this purpose.

Source of Big data – MDM is the subset of big data. It having a lots of information from different sources. So it can be used as the source of big data.

Decision making – Companies use MDM management to keep all records or data which comes from different internal and external sources. Applying big data analytics in MDM gives a better understanding about the company requirement as well as it helps in decision making.

Data quality – The benefits of using master data management with big data is data quality. Whatever data we will use for big data analysis if it is coming from MDM then definitely the quality of data will be far better than those data which are coming from different external sources without applied MDM

Generally MDM holds the internal most trusted data of any organization where as big data contains internal as well as external data – social media, mobile, cloud and many other sources. MDM strategies for Big Data indicate a transformation of the applicability of MDM – and increased customer or product centricity and personalization. Organizations that combine big data with master data management started gaining gaining advantages.

Hadoop for Ecommerce data processing

Retailers always want real time or near real time analysis of huge data sets that change rapidly or have a very short life, for example web shopping cart. We know that Ecommerce companies sit on huge amount of data due to a large number of transaction & inventory. And for that, retailers leverage Hadoop technology for quick and large volume data processing.

Data processing is a process of manipulating the stored data for further use. Stored dump data need to be converted into meaningful and can be used for decision support. So, after processing the data, it can be fit for different purpose as per requirement. After processing, data format may change, means data may be modified and it cannot be the same that it was earlier.

Hadoop is one of the highly used platform for big data processing. Hadoop has established itself as the highly demanded tools in big data sector. Hadoop is used for data storing as well as data processing. For both purpose it is having different part inside it- for data storing HDFS is there and for data processing MapReduce is there. With the help of Hadoop, retailers started shifting their focus on individual marketing by giving customized retail experience.

Hadoop is the widely used framework for big data processing and MapReduce is the most important massive data processing tool for ecommerce data processing. Once Gartner had predicted “Hadoop will be in most advanced analytics products by 2015” and now we can see that their prediction became close to 100 % correct. There are many reports published on Hadoop which convey about the importance of Hadoop in Big data. Some of them are:

A report of Technology Research Organization says that “The data market currently with the fastest growth are Hadoop and NoSQL software and services”.

According to the Big Data Executive survey “Almost 90% organisations which are leveraging big data have embarked on Hadoop related projects and thus Hadoop skills are in huge demand”.

These are some survey reports that convey the importance of Hadoop in ecommerce data processing.

Now we will see that how and why we use Hadoop for data processing. First see the answer of How?

Hadoop is an open source data management technology which having both data storing capacity as well as data processing. Hadoop distributed file system i.e. HDFS is used for data storage and MapReduce is used for data processing. Whenever data come in Hadoop it break all data in small chunks and store it on different clusters across the server. After storing data, MapReduce job runs according to the requirement.

Now we will answer the question of why i.e. Why ecommerce uses Hadoop for data processing?

Using Hadoop, ecommerce companies process data to utilize big data insight to ensure high profitability. Some of the area where they use analysis result that comes after data processing are:

  • Personalized marketing
  • Fraud detection
  • Improved customer service
  • Dynamic pricing

These are the few areas where Hadoop helps ecommerce sector to ensure high value service.

Hadoop having some advantages that make it better from other tools. It is based on distributing computing concept that makes it different from others. Due to its scalability and effectiveness, companies are heavily adopting Hadoop for data processing.

Web Data Mining: Explore immense Automation Potential using Python and R

Today, massive amount of data is uploaded in web-world creating huge new and exhilarating business opportunities to small and medium size companies. However, collecting all of the required data is only one part of the storyline. Mining and converting these data into actionable is where real business value lies. The overall goal of web data mining process is extract information from various web sources and transform it into an understandable structure for further processing. The task of Data mining is to mine or analyse a large quantity of data using automatic, semi-automatic or manual ways.

In general, there are mainly two ways for data mining. First one is traditional way, manual data mining and other one – we could call automated or semi-automated data mining. Manual mining is a time taking and long process where mining is done one by one or say- by each record or piece of information. It needs a lot of time and a lot of efforts. Whereas automated web data mining is a process wherein most of the repeated tasks can be converted into simple logic based script which can scrap all web data as desired. As its name shows that things can be automated – say close to 100%. Whole mining process runs automatically using some algorithm. It is the approach that companies prefer to use for web data data mining.

To make the web data mining process automated, we can follow different methods based on the web structure, data format and size; it may require a custom made language script -script can be in R, Python, etc. and API. We can define process using any scripting language and run it for sourcing the entire data from websites. It will scrap all data whatever we instructed in the code.

We could leverage Python or R. Both are well-known for scripting – data mining especially used in Big data projects.  The beauty of python is that it is a user friendly language. A person from non-technical background can quickly learn and understand it without much difficulties. Another benefits of using Python is that ‘number of lines in script. If you are writing any web scraping script in Python, then it will be completed maximum in few hundreds of line whereas if you will choose java or any other language then it can go to thousands of lines. Due to these advantages Python is preferred in data mining for exploring big data potential. One another benefit of using Python in data mining is Python modules. There are many Python modules are available for data mining which can be easily implemented during the scripting.

For huge volume of web data mining, it is always good to go for automation using custom made script so that time and effort can be saved substantially. Data mining will act as a phase in which we could get data for processing and later for analysis. So it also comes under the big data collection phase. So big data tools can be easily implemented on it.

Broadly speaking, there are three different steps in web data mining. One is identifying the source or sources of data or web pages, mine the desired data and save it into the data processing environment using the right tool; and finally, process the data for decision support. Data identification is the first step in mining process. Here we identify the data in the web page/s that we want to mine. In second step, we check the data pattern i.e. in all pages’ data is showing in same manner and in the source code ‘class’ name of data is same or not. If the class name of data in each web page are same then only we can go for automation otherwise to run an automation job will be really a tough call. Third and last one is writing code and run job. Here we use big data potential like Python. We write code and using the class of that particular value i.e. available in page source code. Input will be anything like url, id etc. but using that it must be redirected to the data page.

Nowadays, everyone prefers to collect web data using automated mining. It is cost saving as well as time saving. For automated web data mining we can use any programming language and different APIs. Python and R seem to be most preferred language for web data mining and also considered as preferred tools for big data projects.

Small Business, Small Investment and Big Data

Big data is only for Big business.” “For leveraging big data, large investment is needed.”

These are some of the myths about big data by which we often come across. Big data doesn’t mean that it is only for big business – and it is not. As we know that leveraging big data is very vital for companies for sustainable growth. Leveraging big data having countless benefits in smaller companies too. But unfortunately many small companies are missing these benefits of utilizing big data because many of them believe that leveraging big data is too costly. But the reality is often different. For leveraging big data, you don’t need to invest a huge money.

A recent survey from Gartner Inc. found that 73 % of respondent have invested or plan to invest in big data in next two years. It is 9% more than the last year record. While the number of business that said they have no plans for big data investment also decreased from 31 % to 24 %.

A SAS report published with the title “Big Data: Harnessing a Game-Changing Asset” showed that 73 % of people surveyed said that collection of data increased “somewhat” or “significantly” over the previous the previous year. So It doesn’t matter that you are a small or big company just go for big data.

Now, let us discuss that how small companies can leverage big data by small investment.

Use ‘unused’ data– If you are a small organization and you think that you don’t have enough data to analyse; then no need to worry, try to find unused data in your organization. This data can be the system information, review mails, sensor data, vendor data, customer transaction data, PDF files, etc. By analysing these data, you will get valuable insight for your organisation.

Vendor selection- You can take a wise decision on the topic that how to analyse big data, means by developing internal capability or outsourcing to a right big data vendor. Look at your need and then take a decision. If outsourcing big data to a vendor having low cost than developing internal capability; then go for a right vendor. If you want to focus on the core of your business; then also go for a suitable vendor.

Always start small- Always start with small. Don’t go for a large investment. First look for POC (Proof Of Concept) and then; invest in whole project. It will minimise your risk.

Tools and techniques– For big data processing, a large number of tools and technology is available. Try to find the appropriate and economic technology. For ex- Storing big data, you don’t need to invest on the hardware you may go for Amazon Web Services. Here you will have to pay according to the usage – as-a-service or pay and use.

Vendor location selection– Find big data partner from those part of world where cost is less and availability of high skilled resources are more. Distance between you and your big data partner doesn’t matter today.


In essence, if you don’t have enough high value customers your business will fail. The same applies if you spend too much money in big data for acquiring those customers or optimize the business value chain. As technology advances, big data is becoming an essential part for small businesses. To be in market competition, it is paramount for all type and size of business to invest in big data.

Small Companies & Big data processing: Way forward

The importance of Big Data is increasing in every passing day. Each and every organization wanted to implement the insight of big data to their business. It is not only applicable to large size of organizations but also for small and medium size companies are also expected to leverage big data for their business to propel growth.

A response from a recent Nielsen poll of 2,000 small businesses in the U.S.A, 41 percent think conducting market research is too costly, and 42 percent say they just don’t have the time. And even more surprisingly, 35 percent went so far as to say they’ve never even considered it.

Research shows that, the companies that have the best data and ability convert them for decision making is likely to win the battle. In other words, big data will play a key role in deciding the winner.

There are many rumours about the usage of Big Data to small and medium size companies i.e. Big Data is not applicable for small and medium size companies, cost of leveraging big data is too high and many more. The fact is that Big data is equally important for big as well as small companies. The cost of leveraging big data is not too high.

Let us discuss some area in small companies where big data insights can be implemented:

Information about customer insights: – By leveraging big data, you may know more hidden insights about the customers’ requirements. As big data brings a lot of information about the customer choice, you may provide your products to the customer according to their current and potential needs. Especially retail sector is using this concept in their business than other sectors and results are visible in our daily shopping experience. And today’s business, it is totally dependent on customer satisfaction – all about customer experience management and ensure collect data at each customer touch point and convert them to informed decision making. This can make customer feel special and delighted.

Better product / service offerings: – Either you have a small company or big company; you must know about your product and customer’s requirement and match of both. If you are offering a high quality product but it’s not according to the customers need; then it will give you zero value. So, to know about the customers need is the first step towards offering any product and for this purpose use of big data can tell you a great story.

Continuous improvement: – Once your product came to market then you need information about that product from customers i.e. feedback. You could only work on the same product again if feedback are positive otherwise you need to revise your plan and product. Big data help you in this purpose to find information as feedback from different sources.

Value chain optimization: – Customers requirement, their preferences, demand forecasting, source of raw materials, processing, supply chain management, vendor management, sales and marketing, human resource support, etc. all these make an eco-system for a company. To be precise, companies having capability to capture data for the entire value chain and leverage them for decision making can improve the way it runs. Big data can better help you to collect these data points and can transform the way to minimize cost and maximize the efficiency of the value chain.

Marketing: – Almost every big company’s decisions are data driven. Strategic plans or especially marketing plan is drafted on the basis of information obtained after leveraging big data and advanced analytics. Data driven decision are more powerful and economic rather than that decision which has taken without it.  This can be also applied in small smart companies and definitely help keeping your business in front line.

In summary, companies – small, medium or big do not think a lot about their cost and own capability of big data; just get into it. In upcoming years, big data to be an important tool for survival and growth of any company.

How Small and Medium level companies can leverage Big Data?

Big Data’ is the word which appears on everybody’s lips these days. In recent years, there has been a huge hype of ‘Big Data’ which is use to analyse by different companies and vendors to capture meaningful insights from a vast amount of data that can be used to improve business and decision making. When the data is too big, and too diverse to handle in standard database; then it is called big data.

As it is clear from the above that big data is huge collection of data. So, it is impossible to use all that data at a time. You can, however, make use of a small portion of the data that is beneficial for your business.

Unfortunately, many small and medium size companies are missing out on the benefits of utilizing big data because they believe that leveraging big data is too costly and too complex. But the truth is that big data is neither too costly nor too complex. Small and medium size companies can also leverage big data because Big Data solutions have become much more affordable in recent years.

There are many ways by which small and medium level companies can leverage big data. Here are just a few ways small and medium size companies can leverage big data towards their success.

Look for unused data– Small and medium size companies should look for those data which never used in the entire value chain. These data can be their feedback by customers, emails, vendor transaction, and many more. By leveraging these data, we can get some meaning insight that can be used to improve the business – the way it runs and operate.

Look for affordable and effective big data partner– Small and medium level companies can outsource their data to a suitable third party vendor for processing because developing internal capability may not be a wise decision for them at the initial stage. So find an affordable and effective big data partner for your success.

Go for small– If your company is small and medium level; then go for a small start towards big data. After getting some quick positive results from the vendor then only go for big implementation.

Look for the necessity – Small and medium size companies can go for leveraging only those part of data, where analysis is required. Analysing the whole data can be a waste of time and money for them.

Location– Cost is an important factor for any size companies including small and medium size companies while leveraging big data. So, find suitable big data vendors – today there are good small players in the marker out there where you get what you wanted at lesser cost at higher quality and quick turnaround. For eg: leveraging big data is costlier in USA rather than leveraging in India.

Adopt new tools and techniques- For collecting and leveraging big data; use new tools and techniques from the market – essentially freeware.

Every business needs to know the way to success and increased ROI. Leveraging big data is useful for all level of companies. The point is – how you are using and implementing it. According to a survey of Gartner, investment in big data technologies continues to expand every year. They found that 73 % of respondents have invested in big data or have plan to invest in big data in next two years. So, if you are a small or medium size company and thinking that big data is not beneficial for me then for sure you are leaving your boat.

Big data analytics and Competitive growth

In today’s tough competition, big data analytics is playing a vital role in deciding the winner of the market place. Today’s business is mostly centred on the customers, especially retail sector. So, companies prefer to craft personalized marketing strategy according to the changing behaviour of customers. If ‘Technology’ brought big data concept; then ‘Marketing’ used it most for the organization’s growth.

In a recent survey from accenture and general electric showed that 87% of enterprises believe that with the use of big data, the competitive landscape of their industries will be changed in upcoming few years.

89% of them believes that companies without using big data analytics strategy in the next year, they will lose their market and be as competitive.

73% of the companies are investing twenty percent of their technology budget in big data analytics.

From these results we can imagine the role of big data analytics in an organization. Let’s see how big data analytics can help in competitive growth.

Customer insight – As we know that today’s business is customer centred and customer is the focus. So, companies collect data about their customers from various customer touch points and after analysing those data they understand more insight about customers. Companies who missed these valuable analysis result is nowhere in the competition today.

Personalized and dynamic Marketing– For better increased sale, companies need personalized marketing strategy and it is only possible when companies have detailed information about their customers or potentials.

Improved of customer experience– Big data helps companies to improve their customer experience not only in after sale but also to provide new products to the customers according to their needs.

Innovative ideas– Big data helps companies to develop next generation of products and services. For this, manufacturers are using data obtained from sensors embedded in products to create innovative changes new product development after sales services. Analysis of those data helps them to do some innovative way to serve their customers and delight them. If companies not using these benefits and serving them in traditional way; then they are likely to lose their customer base. Getting a new customer is far tough than holding an existing one. So, you always need to do something new for your existing customers. NOKIA is one of the victim of this case.

Minimize risk- A decision based on the real time data is always far better than that decision which is based on historical data – meaning way old data. If you have no information about the current market and trends, then you are making a decision then there is no guarantee that it will work. But after analysing the behaviour of market and making a decision; then it is likely to have less risk.

Improve internal capabilities – Use of big data make an organisation well organised. Companies can know about their internal capabilities using the analysis of these data and can make decision for the improvement.

In summary, the insights that we collect from big data and analytics can be used in different industries and can be paramount in exploring competitive growth. In the upcoming years we will be the witness of big data and analytics to decide the winner of the competition. So, organizations that are not leveraging big data and advanced analytics for decision making; then get on it; else you may lose your position in the game.

Big data and Advanced Analytics: Made for each other

Before knowing the relation between big data and advance analytics let us look at both i.e. big data and advance analytics.

Recent research by analyst firm International Data Corp. (IDC) reported that the global amount of digital data will grow from 130 Exabyte’s to 40,000 Exabyte’s by 2020. The old data processing technologies like RDBMS are simply not capable to process data in such a large amount; so, a new trend came “Big Data”. In simple words big data is a collection of huge amount of data coming from different sources.

Then, what is advanced analytics?

Advanced analytics is a grouping of different analytic tools used to predict the future outcomes. Predictive analytics, data mining, big data analytics and location intelligence are just some of the analytic category that falls under the banner of advanced analytics. Advanced analytics is widely used in industries including marketing, healthcare, risk management, support and economics. In other words, it is used in almost all sectors where the analysis of data is needed. For this purpose, we need a team of experienced statistician, analytics tools and data visualization to perform and give results.

Big data is plenty big, and it’s going bigger. There is no use of any data when we don’t get any relevant information from them. Big data is a huge collection of structured / semi-structured/ unstructured data type. By using advance analytics, companies figuring out how to turn that data into values for different purpose like marketing, sales, etc. Without advance analytics it is impossible to deal with big data. Companies also know this fact so they focus equally on both the side.

According to a report by BMO Capital Markets report $50 billion are spending by the marketers on big data and advanced analytics. Big data and advanced analytics can transform your business. The point is – it may not be worth using any single of them to get result because advanced analytics is the key to reveal the secret of big data.

Big data can help you out when you have ability to ask right business questions. The tool by which you will search and find the answer of that business question in big data is called advanced analytics.

Advanced analytics can include a lot of mathematical calculations and application of business knowledge on the historical data and based on that try to predict the possible future outcome – more accurately. Advanced analytics techniques allow us to design precise model of real surroundings around us. These models help you to take better decision making.

To summarize, advanced analytics is the process of analysing a large data sets containing different variety of data types i.e. Big Data to reveal useful information, to uncover hidden patterns, market trends, future outcomes predictions and other useful business solutions. So, we can say it all in one line – ”Big Data and Advanced Analytics are made for each other”.

Big data for data monetizing: Recent trends

Big data is a fast emerging concept that totally transformed the business – the way it runs in almost all industry and sector. Now it has been covered a big area of business. The main part of big data is to store the vast amount of data and analyse them to get some valuable information.

Data monetization means to get some beneficial information from that huge amount of data to generate profit for your organisation. According to an Economist Intelligence Unit survey those firms who used big data analysis result for decision making got 5-6% of improvement in their performance and other 41% company expect improvement in their business within next 3 years due to big data. These data are enough to prove that big data can be monetized.

Now, let us discuss some recent trends in big data.

Cloud computing – These days, cloud computing is one of the hottest trends in big data field. For storing big data, we need a large storage capacity. We need at least some TB of space. Cloud computing allow you to store the data in service provider’s server. It is of flexible size i.e. if the amount of data increase then its space can also increase according to that by allotting more space. Amazon Web Services is an example of cloud computing.

Hadoop – Big data means Hadoop? Hadoop is a framework on which big data processing occurred. It consists two core parts in it i.e. HDFS and Map Reduce. HDFS is storage part, used to store data and Map Reduce is processing part.

Security – Bank fraudulent reports show that – lost opportunity is multi-billion dollars every year and we could minimize it leveraging big data. Big data analytics fills the security void. It helps to find the security gap by analysing the patterns. Big data is frequently used in fraud detection.

More predictive analysis– Big data helps companies in decision making. As we saw earlier that with the help of big data analysis companies got 5-6% of improvement in their performance. With the use of big data, companies are able to serve their customers better and increase their revenue.

Apache Spark– It is a new technology in big data that works 100 times faster than Hadoop. It saves both, your time and your money. Effectively, companies moving to Spark technology to resolve big data problems.

IoT– At present Internet of Things like- sensors, smart machines, connected devices, etc. is capable to generate a huge amount of real- time data that helps companies to find the proper and basic information about their customers. This information helps them to make customized strategy.

According to a prediction by Gartner “By 2020, information will be used to digitalize or eliminate 80% of business processes and products from a decade earlier and by 2017, more than 30% of enterprise access to broadly based big data will be via data broker services, serving context to business decisions.

Another prediction by Gartner states that by 2017, more than 20% of customer-facing analytic organisation will provide product tracking information leveraging the IoT.

These are the recent trends in big data through which big data can be monetized. Monetizing big data is the best way to increase the revenue or EBITA.

7 ways to monetize Big Data

Data monetization means generating more and more profit from Big Data. Today, every business owner is trying to focus on Big Data to get values from it. These values help them out to make business strategies and increase ROI. By the analysis of the data, organizations come to know more about their customers, vendors, and their behaviour, pattern, etc. This information helps them to make their strategy according to the needs and demands of the customer – effectively serve the entire stakeholders better.

A Big data related survey on July 2015 showed that 60% of CEOs around the world now use data analysis to run their decision making process.

In July, 2015 an Economist Intelligence Unit survey surveyed upon 600 CEOs around the world and of different sectors about the use of big data in their organization. This survey showed that 60% of CEOs across the globe now use data analysis to govern their decision making process. 75% of them was ready to accept that their organization should be data-driven. 90% decision was better in last three years when they had made it after analysing the different and relevant information. 42% of CEOs was facing difficulties while dealing with unstructured data. This survey result is enough to make things clear – Big Data is how much important.

Every company is trying to monetize data in different ways. Here, there are 7 ways that can look at data monetization.

Look at your business – various data sources– Take a deep breath and find all sources of data within your organization. There are always some data which won’t be used by the organization due to different reasons like highly unstructured but if you are capable to analyse those data

Search potential application of the data – Let us not leave any data sources unturned. Today, technology evolved a lot and there are various ways and means to convert any structured and unstructured data in to meaningful. So, look at data sources and try improving the way business run, customers are treated, vendors are considered, etc.

Assemble the right team– For analysing data you need to have a well organised and skilled team. Monetizing the data is mostly depending upon the analysis result and ability to generate insight out of data. So, get the right people on boat.

 Look for your company size– For monetizing big data- size of your company also matters. If your team size is small, then look for a vendor who can help monetizing the data.

Analytic skills- Your team need to have good analytic skills because big data not just about the size of data it is all about converting the data – identify hidden pattern and leverage them for decision support. Also, experience in working on different big data tools and technology regarding the analysis of data is one of the important way to monetize big data.

Look for the patterns – identify and prioritize areas to monetize. Here patterns refer to the type of data and hidden pattern. While analysing data, similar pattern to be analysed in separate attempts. It can save your time and effort and you can expect more benefits from the data.

Leverage internally or sell the data to external parties– On which role of big data analysis your company is playing. Here the ‘role’ refers as a consumer of data, an aggregator or a creator of new data product. Which one will suit better to your organization go for that, it will help you lot in monetizing big data.

Data by nature may not be helpful unless processed. Data in a proper context make sense and provides information. 10rs doesn’t make any sense but 10rs for a ‘Dosa’ in Kerala in 2015 make sense and also gives an information. Same story is about Organization data too. Those unstructured data never going to help you to make profit but putting those data in a proper context can help monetize.

Page 4 of 5«12345»