However, companies that manage to draw insights from this pile of unorganized information can gain a critical competitive advantage. Analyzing this data efficiently allows them to identify issues and opportunities quickly, and pilot activities in a data-driven manner.

In the last decade, data scientists have developed powerful algorithms to process Big Data and make it intelligible. They now can find patterns, trends, and relationships that will give them insights into their performance. 

With predictive analytics, for instance, they can forecast performance and market changes based on existing and previous data. In this article, we study how this branch of Big Data analysis will impact marketing activities in 2023. Let’s dive in!

How can predictive analysis be helpful to marketers?

In competitive environments, businesses must always be one move ahead of market trends and preferences. This is made possible by predictive analytics. This method of analyzing Big Data helps marketers evaluate customer trends, identify potential changes in the market, and run marketing campaigns more effectively. 

Big Data marketing uses predictive analysis to understand customer behaviors and future trends, allowing them to plan their campaign accordingly and seize opportunities in real-time. Of course, it’s not an exact science, no algorithm can predict future events with 100% accuracy. But it still can give an idea of what might happen in the future with a certain degree of confidence.

Big Data and predictive analytics can be beneficial for marketing as follows:

• Effective decision-making — Marketers get fresh information and make decisions based on data instead of assumptions.
• Cost saving — Marketers save on paid campaigns and can optimize spending by analyzing the results of previous campaigns.
• Productivity improvement — Marketers can evaluate activities that bring in more results and run operations more effectively.

There are many more use cases for predictive data analysis in marketing, read this article for more practical applications.

What data is used for predictive analysis 

Big Data analytics is the challenging process of sifting through enormous amounts of data to find actionable information. It involves applying advanced analysis tools to very large, diversified Big Data sets, which can range in size from terabytes to gigabytes and contain organized, semi-structured, and unstructured data from many sources.

Depending on the industry, marketers can deal with a large amount of very disparate data. The Internet of Things (IoT), handheld devices, media platforms, and artificial intelligence (AI) are driving sources of data to become more complicated than those for traditional data. 

Data is usually collected in three forms, which are:

Structured Data — It is a type of data that corresponds to a specific, predefined format. Because of its predictable structure, it requires less processing and is easy to organize, search and analyze.

Examples of structured data: Any relational data that can be stored in a table with rows and columns.

Unstructured Data — This type of data does not have a pre-defined data model. It usually corresponds to factual data and comes in greater volume. Although it’s more difficult to process unstructured data using conventional techniques, it provides better insights and more opportunities.

Examples of unstructured data: Videos, audio, social media posts, etc.

Semi-structured Data — This data cannot be stored in a table but has some properties that make it easier to analyze. Semi-structured data has to be pre-processed before analysis.

Examples of semi-structured data: Emails, dates, credit card numbers, geolocation, etc.

How to turn data into an exceptional product or service?

A product or a service that uses behavioral data to guide business choices and development is referred to as a “data product.” Data products are created by turning the data resources (that an organization already owns or that may be obtained) into a product intended to assist a user in solving a particular issue. 

Updating and continual feedback are essential to creating exceptional products. Quantitative and qualitative data are gathered and analyzed for more intelligent and strategic judgments. In this way, data is used to improve products. However, there are different degrees of data-oriented product design and development. 

The influence of data on product development

A data-driven approach leads to very customer-centric products. It required an experimental mindset where the designers generate ideas and experiments, but ultimately the customer behavior drives the orientation to take. The main objective is to generate data and then let this data make decisions and help uncover customer opportunities.

What’s in it for you?

We can say that data is an essential resource in the globe for organisations, and businesses to gain a competitive advantage. With the help of the knowledge acquired by predictive analysis, marketers are better able to anticipate future events and develop effective marketing plans.

Big Data and predictive analytics for marketing in 2023 are more challenging than ever before. Since there is so much data available, predictive analytics cannot be used by marketers without powerful marketing tools and measurement skills. If you’re looking for a partner to assist you on your Big Data journey, reach out to the Mowgli team.

Big Data refers to the massive volume of data that companies collect and store every day. This data comes from two major sources. First, from customer interactions with the business, but data also comes from internal sources such as the different tools and technologies used by modern businesses to run operations.

There are many use cases for Big Data to have a significant impact on how we manage organizations and create value for customers.

So how will Big Data impact tomorrow’s business? Let’s dive into how related technologies will play a crucial role in business decision-making.

Big data trends in Marketing applications for 2023

To combine and analyze the data collected by businesses, scientists use top-notch algorithms that can handle a large amount of information and make sense of it. Using these data models, they can draw out actionable insights that help pilot day-to-day business operations.

Learn more about Big Data use cases for marketing in this article.

Big Data in Decision-Making: How It Will Play a Crucial Role

Modern businesses are looking for innovative and performant methods to better lead operations. From distributing financial and human resources to crafting a relevant offer, Big Data is one of the better ways to glean useful information and support proper decision-making. 

So, let’s explore how big data plays a crucial role in decision-making.

Get the bigger picture

Companies collect data from so many different sources: websites, apps, customer service, social media, and more. This data comes in massive volumes and in different formats that are challenging to combine and analyze as a whole. 

The more this data grows, the more challenging for a business to transform it into valuable insights. But with Big Data algorithms, companies can finally make sense of unstructured data and get the bigger picture of their market, customers, and processes. They can better understand the customer journey as a whole and improve the quality of marketing campaigns.

Understand the market

A business’s ultimate goal is to have enough customers that buy, use, and recommend their solution to sustain growth and profitability. To achieve this, you must have a product in perfect adequation with your market and achieve your product-market fit.  

This means ensuring there is demand for your solution, understanding the market trends, analyzing competitors, customer behaviors, and so on. In some markets, this information can represent a massive volume of data that would hardly be possible to make sense of without the use of Big Data. 

Big Data also helps companies better understand customers and their feedback. For instance, Natural language processing allows marketers to analyze language and find patterns in order to evaluate how they perceive the company or products.

Improve operational efficiency

Leaders and managers have understood for a long time that decisions should not be based on assumptions but on tangible data. This is also true for internal strategic decision making: how to allocate budget, resources, and how to prioritize activities.

Leaders can also use Big Data to improve day-to-day operations. Project management tools, CRMs, and any other business software used to run the company also produces key data. Analysing this data can help companies improve operational efficiency and identify opportunities for improvement.

Big data trends: machine learning and IoT 

Big data trends: machine learning and IoT

Self-driving cars and IoT

Tesla has always been a data-driven company. Elon Musk, its founder, is a disruptive entrepreneur who is always developing top-notch strategies to collect, analyze, and make strategic decisions based on data. 

Tesla uses car sensors to collect data about the vehicle and the consumers. The company has gathered over 100 million miles of information on roads. This allowed them to create maps that are said to be more accurate than any other GPS.

Data is used in all areas of Tesla: car performance, customer experience, satisfaction, and product developments. Tesla uses Machine Learning to power its self-driving software, with data from surrounding vehicles and other Tesla cars in the area. It uses sensors, cameras, and radars in the car to collect data, and combines this information with external data such as GPS or weather information. This data is analyzed using Big Data models and results in a decision that translates into the autopilot software. 

Keeping employees happy with sentiment analysis

IBM uses Sentiment Analysis to understand how employees feel about the company and their jobs. All 380,000 employees in over 150 countries have access to an internal social networking platform where they publish content, exchange comments, and chat in smaller groups. 

Big Data gives IBM social listening capabilities: they can monitor the networking platform and identify trends and alerts. For instance, it can help them identify recurring complaints and take action accordingly in a timely manner. 

For the sake of privacy, IBM claims that only public posts and comments are collected and analysed, no email or private chats are included in the process.

Preventing customer churn with predictive analysis

Predictive analysis can help companies forecast when a customer is about to churn – i.e. stop doing business with your company. These models can identify dissatisfaction or loss of interest in your solution so you can take action before it’s too late.

Preventing churn is key for businesses that have a model based on recurring revenue such as SaaS companies. If you want to learn more about predictive analysis, check out this interview with one of our Big Data experts.

Get started with Big Data

At Mowgli, we work with Big Data experts to help businesses make the most of their data. Chances are you already collect tons of information about your company and customers. What to do with it? Reach out to our team for a personalized audit and see how you can leverage Big Data to drive efficiency and revenue for your business. 

Conversion rate optimization (CRO) for data driven processes

As an e-commerce business your main goal is to sell. There are many techniques, tools, ideas, strategies, and thoughts on how you can get more sales. In this article, we will look at what big data and CRO actually are. We will dive into how they complement each other, and how you can utilize both to eventually create more conversion for your business.

CRO explained

CRO stands for Conversion Rate Optimization, but what is this exactly? CRO is a process in which you increase the percentage of purchases or any other desired action (for example subscribing to your email list) on your website or application. This process involves improving your website as well as the user experience of your website. The hypotheses created after this analysis will then have to be tested with a split A/B test. These tests will then provide the information you need to optimize. This constant process has the aim to keep on identifying optimization opportunities.

Do you want to read more about improving your CRO, read more here.

Big data explained

Everything we do online leaves a digital trace. The more people that are leaving traces online and the more data that is stored, the more information there is to work with. It is called big data for a reason. In 2018 there were around 33 zettabytes (ZB) in the world and this number is predicted to grow to 175 ZB in 2025. That is a mindblowing amount of data. Simply having a phone in our pocket already leaves a trail of information. It traces for example where we are, how many steps we take, and when we unlock our phones.

Big data allows companies to make decisions based on facts instead of assumptions, and this is a very powerful way of moving forward and creating new strategies. But how do companies make sense of this vast amount of information? First, you have to figure out what key information you need; and then where you can obtain this information. We will look at it below.

Big data approaches for more informed decisions

Let’s look at where you can find the information you need to create a data strategy that will help optimize your business performance and CRO. In order to move forward, we advise you to look at this process strategically. We will look into five pillars you can use to gain information that help you better understand your customers, internal processes, ways to monetize, and your products and/or services.

1. Better understanding your customers

You can use data to get to know your customers better. This data might be gathered directly from your website, or based on your target market. Investigating your target market as well as market trends will allow you to understand the wants and needs of your customer better. That is why we always recommend first looking at the data of your target market and/or customers.

2. Getting insights on your products and/or services

You have created your service and/or your product. Whether it has been created recently or a while ago, it can not hurt to gather data on the function and experience of your product and/or service. This is because the world is rapidly changing and there might be ways to make your product and/or service smarter, better, and more optimized. You might get insights about an additional value you can add, that you didn’t know about if you hadn’t looked at new data.

3. Improve internal processes

Your business has internal processes. Whether you work for yourself or you have multiple people working for you, things are done in a unique way in your business. But is this the most functional and right way? Data will tell you. Researching internal processes of operations, marketing, and manufacturing processes will give you information that might indicate necessary changes to some of these processes for the better. A couple of changes can save you a lot of time and money. Definitely worth looking into.

4. Use data to monetize

As an e-commerce business you want to make money. Rather than guessing what works and what doesn’t, we recommend looking at data to see if you are monetizing your business in the right way. Where are you making money and where are you losing it? Look at your business as an asset and use your data to make your asset grow even more.

5. Bring the data together

Once you’ve looked at all the categories above, it is important to write down which parts are interesting to research for you right now. You can divide them into interesting research topics for long-term and for short-term gains. Try to set goals that are concise and measurable. The information that you want to gather should be easy to obtain and to decipher.

Which business challenge would you want to address? What measures of success can you identify? How can you be more successful? In order to get to know about this, which data do you actually need? Setting measurable goals will help you with the next steps; to obtain the data and make sense of the information in front of you.

CRO and big data together

Data has become one of the most valuable and important business assets for companies. We have looked into where you can search for data and which questions you should ask yourself to set the goals you would like to measure. After this, it is important to look at any data gathering complications. What are the laws around data gathering and how can you best use technology and skills to obtain the right information?

After having the right data in front of you, you want to look at implementation. How do you translate the data into a strategy? What are the most important things to look at? What changes will optimize your website and/or application? How can you improve your CRO (read more about CRO for E-commerce businesses here)? Let’s look at how you can calculate a conversion rate and make sense of the numbers in front of you.

How to calculate conversion rates to improve CRO?

You can calculate the conversation rate by dividing the number of conversions by the number of visitors on your website. You can then multiply the results by 100 to get a percentage. We advise you to calculate this number before and after you implement the data analysis changes. This way, you can make sense of the numbers in front of you. Then what is a good conversion rate? We aim somewhere between 1 and 2% for e-commerce businesses.

Do you want to know more about improving your CRO rate?

Do you want to learn more about data strategies and improving conversion rates? Reach out and get in touch today and book a call with us. We’ll be happy to assist you in further growing your business.

The introduction of technologies and methodologies for the analysis of massive amounts of data has changed the way companies do business. Customer data is collected by companies (not just web companies) to discover the links between different events and predict future behaviors in order to gain a competitive advantage over their competitors. 

In marketing, data analysis is now a common practice, because it allows businesses to drive company growth by building more lasting relationships with customers. They do so through the collection of factual information about their tastes, interests, and habits. The ability to extrapolate and analyze a huge amount of data – both structured and unstructured – continues to help companies scale their businesses.  The question of how to manage information that is often sensitive has gathered increasing attention.

In this article, we will discuss what a business can do to ensure to protect the privacy of its customers. We also look at how an organization can establish analysis systems and knowledge generation that account for ethical principles that respect and protect internet users.

What is data ethics?

For the majority of companies operating online, data management has become a strategic pillar of the business. A data-driven approach helps to make informed decisions, based on objective facts and not on personal feelings. Collecting accurate and fresh customer data is critical in the fast-paced digital world we live in today. This data helps to understand KPIs, generate statistics, and identify technical problems that may be occurring during the company’s life.

Big Data is a powerful tool, especially in companies where advanced analytics are used to manage production or sales. However, dealing with behavioral information can raise ethical issues that must be addressed to help find the balance between improved user experience and the protection of users’ privacy.

When we speak of data ethics, we refer to those principles that guide the correct use of data from a moral standpoint. Determining what is the right (and wrong) way to collect, use, store, and analyze data collected through digital means has an impact on both individual users and society as a whole. Specifically, it’s important to consider who has ownership of the data, how transparent the communication regarding its use is, whether users have given their consent, and how well privacy is protected.

A first step toward the establishment of a fixed set of principles valid for every business, was made by the European Union, which introduced the General Data Protection Regulation (GDPR) in 2016. It helped identify ethical parameters to follow when collecting data and extracting knowledge. Let’s take a look at the key principles of data ethics.

Customer data protection as a service: six GDPR principles of data ethics

The new European General Data Protection Regulation (GDPR) states that the processing of all personal data must be aligned with the principles defined in the regulation. As part of the effort to implement the regulation, it is important to understand the six key principles of the GDPR. These principles form the basis of the GDPR requirements – let’s understand what they are and what a company needs to account for in order to comply.

1. Transparency and lawfulness

Companies that collect customer data need to make sure that their users know for what reasons information is handled and processed. To legally collect data, an organization must make its privacy policy public, explaining with maximum clarity the purpose of the data collection and how the company intends to use the information.

2. Specific reasons for collection

The processing of personal data must be limited to the legitimate purpose for which such personal data was originally collected. In other words, a company should make clear in advance why it is collecting data and, if a user agrees on such terms, use the data only for the purpose that is specified. This effectively prevents the processing of personal data outside of the legitimate purposes for which the customer data was collected.

3. Process only the customer data that is necessary

This principle is known as “data minimization” and means that when collecting data, a company may only request personal data that is absolutely necessary for the purpose that is specified. For the GDPR, it is not allowed to request or store data that is not essential for the stated goals of the company. It is important to limit data analysis to an anonymized data set, a data set for which consent has been obtained, or for which there is a clear legitimate processing purpose.

4. Data should be accurate

Organizations should take measures to make sure the data collected is always accurate and updated. Companies are asked to ensure that customer data is kept as accurate as possible at all times and that users from which the data has been collected can update their data when necessary. People have the right to request that their personal information is canceled or rectified if inexact or incomplete, and following such requests the company owner has 30 days to fix the information.

5. Limited retention

Organizations should delete personal data when it is no longer necessary for their purposes. This is not straightforward – the timing changes from company to company and from industry to industry –  and must be determined in accordance with applicable laws, which may sometimes require that personal data be retained for a longer period than the originally intended processing purpose.

6. Security

Personal data must be processed in a manner that ensures appropriate security of personal data, including protection against unauthorized access and from accidental loss, destruction, or damage. The GDPR does not specify how such measures should be implemented and while the most common method remains encryption, companies are free to choose systems suitable to their objectives.

Ethical by design in the digital economy: building trust with customers

The discussion surrounding data ethics is still evolving and with the introduction of artificial intelligence systems to manage large amounts of information, issues related to algorithm bias are becoming increasingly important. It’s not just humans who are responsible for the correct management of customer data, but also those who program machines to process such data.

“Ethical by design” means that machine-learning algorithms should be created with ethical principles in mind in an attempt to contrast bias that can be present – perhaps unconsciously – in human programmers. There are three steps to avoid AI bias, which should always be part of the process of designing ethically:

• Train algorithms with datasets that are representative of the population. Machines learn based on the data sets they are fed with. By using information that originates from a small section of the population, algorithms may produce results that favor one group over the other. A system that traces the origin of data sets can help identify potential biases.

• Check feedback loops. While machines can be trained through selected datasets, once they are put to work they continue learning through users’ feedback. If an action is repeated by many users, the machine will interpret such action as predictable, favoring some results over others. If, for instance, photographs of a specific body type tend to be “liked” more often on a social media platform, an algorithm may begin showing that body type more often to please the audience, excluding other images.

• Human intervention. It’s important to think about how humans and machines can work together, rather than expecting the algorithm to run on its own. Systems known as “human-in-the-loop” can provide insights into ways in which humans can control how the machine behaves and intervene when it becomes necessary to correct the algorithm.
  

No algorithm is perfect but addressing potential issues related to bias before they impact users can help prevent ethical issues that may arise due to unconscious bias.

Rethinking customer data: some final words

Data analysis has become essential for any business operating online, but with an increasing amount of personal information circulating globally, the question of how to ethically manage such information has gathered increasing attention. How data is collected, used, and shared can impact behaviors, decisions, and perceptions of both individuals and communities. Therefore, it is important to consider ethical principles when collecting information and designing algorithms.

In this article, we have explored the key principles to keep in mind when working with data, but the discussion around ethics remains open. What’s important is to remember that behind every piece of data there are people, and just like you and me, they like to be treated ethically

Do you need help with collecting, understanding, and analysing data for business growth? Get in touch today and let’s discuss your project.

People analytics: a data-driven approach to reduce gender inequality

The vast majority of data that we have collected, and continue to collect globally, has been gathered on men. This applies to all sorts of data ranging from economic, to medical, urban, and lifestyle data ‘The data gap is continuing to grow as researchers carry on ignoring the pressing ethical need to include female cells, animals, and humans, in their research’ Caroline Criado Perez says in her book Invisible Women. It is important to reduce gender inequality by utilizing data in a more inclusive way.

Examples of the gender data gap can range from small inconveniences for women to influential issues that truly impact the way females live. Small inconveniences are for example the fact that bullet-proof vests are not created for women and often don’t fit properly. A somewhat larger problem is that car-crash dummies are based on a male posture, which makes it statistically more dangerous for women to drive a car. Let’s look into three issues that influence the gender data gap in tech.

1. Most computer programmers are male

The US Bureau of Labor Statistics mentions that even though the population of the US is half male and half female, 78.8% of computer programmers are male. This means that the people who are creating our online world are not representative of our society. The question that arises is how predominantly men, who are not representative of the entire population, can effectively create over 2/3 of the websites, applications, and systems that everyone uses. A way to combat this is to encourage more females to dive into the world of coding, programming, and computers.

2. Data bias is created when AI systems are based on gender inequalities

Research shows that the data that AI uses are heavily male-dominated, which is considered to influence the gender gap. This is because one study found out that AI not only reflects on this data, but also amplifies the data that is already in their system. This means that AI uses data that will lead into even stronger sexist assumptions. Therefore, it is extremely important that if we want to close the gender  gap that not only we, but also the machines that we feed data, are using data on both males and females.

3. Private companies are not showing their algorithms

Companies such as Apple, Google, and other big tech companies all have their data protected and they don’t share their algorithms. This means that these companies have a free card when it comes to handling data biases. There is no way for third-parties to examine the biases that are in the systems of these tech companies. This is a pressing issue that is only growing with Artificial Intelligence being used more often to influence our decisions such as with the process of hiring employees.

How can we make data more inclusive?

To create a world in which we can overcome the gender data gap, humans have to take the responsibility that we, and the machines that we use, are using data that are also including minorities. On top of that, we can focus on hiring and training more females to be able to work in tech companies. We can also ask tech companies to be more transparent when it comes to sharing how they handle their biases. Let’s look into three companies that are already working on closing the gender data gap.

Organisations who are leading the way to gender equality and fair pay

Even though, for many organisations, there is a long way to go to reach a point in which there is less gender inequality present in business and tech, some companies are already taking great initiatives to get to this point. Let’s look into three inclusivity examples from PayPal, IBM, and Asana.

1. PayPal: training all their employees on ‘conscious inclusion’

If you work at PayPal you will probably be included in their training that is all about conscious inclusion. PayPal mentions that: ‘Diversity is the catalyst of innovation. We’re committed to empowering talented people from every background and perspective to thrive’. On top of that, PayPal focuses on organising events for women in which knowledge is shared through hackathons and other meet-ups. Their focus on creating more opportunities for women as well as educating everyone in the company about inclusivity is a great example of reaching more awareness for the gender data gap.

2. IBM: believer of inclusivity driving better outcomes

At IBM they believe in putting more attention to representing more women in the world of Artificial Intelligence. This is because they believe that inclusivity and gender equality are drivers of better outcomes. Examples of the leading ladies at IBM are Annie Shu, who helps teenagers to get their first job by using a friendly avatar and Melissa Dorey who delivers AI-powered answers to many employees. Letting women lead in AI-driven projects helps to close the gender data gap and IBM is taking the lead into improving this.

3. Asana: focuses on building, recruiting, and thriving

Asana’s inclusivity strategy is focused on the pillars building, recruiting, and thriving. They build by creating equal opportunities for both males and females, including a 16-week paid parental leave for new parents. They recruit by seeking employees even in underrepresented communities and they focus on including people from different backgrounds. They thrive through initiating open conversations, offering free therapy, and making people feel they can be their best selves at work. This focus is helping both males and females to feel that they can thrive at work, especially taking into account that parents can take their time coming back to work after having a baby.

Do you want to know more about communicating inclusivity?

Do you want to look into ways to communicate diversity and inclusivity? We can help with the branding strategy for this as well as other aspects that come into play. Get in touch today and book a call with us. We’ll be happy to assist you in further growing your business.

In this interview, we discuss use cases for predictive analysis and best practices to get your company started with data analytics projects. Let’s dive into it!

What is predictive analysis? Why is it useful?

Predictive Analysis is a category of data analytics aimed at making predictions about future outcomes based on historical data. By predicting what will happen in the future, we can prepare ourselves for future developments and behaviours or forecast the effects of different strategies and decisions.

In data-driven marketing strategies, predictive analytics is used to determine what is likely to happen in the future based on historical user behaviour and past campaign data. For example, these models can be used for segmentation based on past engagement, to identify upsell opportunities, or even to determine the chance of a prospect actually purchasing your product or service.

On the other hand, it can also help determine how good a strategy is performing. Let’s take the example of Covid-19 and how it impacted the way companies sell their products and services. Many businesses decided to invest in their online strategies, and many saw positive results with increasing online sales. However, can we really say that they are actually selling more? 

To answer this question with certainty, they cannot compare their sales with the revenue they had just before the lockdown. They need to compare what would have happened if the lockdown never existed. That’s where machine learning and more specifically predictive analysis is used: you can take historical data, on the revenue for example. You can use this to project them in the future to predict what would have happened if businesses were running as usual, without Covid-19 and a lockdown. Only from there can you really compare the two results and determine with accuracy whether the online strategy is performing better.

Can you give us some other examples of how predictive analytics is used in a business environment?

Let’s say that we have a dataset of company X describing sales of various products. Company X has many shops. There are about 25,000 monthly historical sales records and information about the sales price and the shops’ location in the dataset. Our goal is to make a six-month forecast of the sold volume of a product sold by a store.  

In addition to the data provided, we will include additional variables that could help predict our outcome, such as special days like holidays, volume sold in the entire industry, seasons, trends from social media platforms, and so on.

After some additional data transformation, we will use all the data except for the last six months to train a Machine Learning model. The model will learn historical patterns from the data and the relationship between variables. We will then test the predictive ability of the model on the last six months of data. If we are satisfied with the results, we can then do a forecast in the next six months.

Can you tell us about a predictive analysis project you have worked on?

Yes, several. From predicting the probability of floods to forecasting sales for a company. 

For one of my clients, I had to develop an automated system to predict the effect of concurrent Instagram ads every month. The company wanted to continually monitor and evaluate many Instagram ads strategies every month for roughly a year.  The goal was to predict short term effects on sales and attribute the observed increase or decrease in sales to one of the strategies. As you can imagine, this is not only about forecasting sales in the future, but also disentangling the effect of one strategy from the other and defining causal relationships between an ad and the volume of sales.

In this case, we were able to determine the right strategy to increase the revenues over a short term and keep them steady over a medium period. This was a big win for the company!

What are the main challenges of such a project?

Predicting the future is not an easy task. It is challenging to make detailed and long term forecasts. Imagine that we would like to forecast the daily number of clients for the next year! Another challenge is data availability. We need a lot of historical data to use the current state of the art models.

Additionally, uncertainty is another challenge. The example from above represents a basic setting. But in some cases, we might want to include and quantify the uncertainty in our prediction. We would like the model to output a full probability distribution over the full range of our outcomes. In most Machine Learning models, you get back a single real number as the expected outcome. In the classification case, as we have seen for sentiment analysis, most of the models are probabilistic by definition. But this is not the same for regression models. Typically, regression models return a point estimate. But today there are ways to output a full probability distribution using Probabilistic Regression Models.

Another challenging case is with streaming data. In this situation, data arrives sequentially and at high speed. The analysis must take place in real-time. Classic examples of such situations are stock price data or weather data. In these cases, we will need to use algorithms that dynamically adapt to new patterns in the data called Online Learning Algorithms.

What is the average cost of a project?

The cost is conditional on how often the forecasting should be done. The prediction models can be run once a year, twice a year, every month or even every hour, depending on the objectives of the company.

So you have an idea, projects usually range between 6K€ up to 25K€.

In our last discussion, you mentioned that companies should improve their processes of data gathering and storage before launching a data analysis project. What are your recommendations to do so?

In 90% of the cases, companies have much data but are scattered and disorganised. Most of the time, the biggest problem is data quality and data storage. For example, I had a client with unique data on farms all over Southeast Asia, but the data was scattered everywhere. It took me two weeks to fix all the GPS locations, which was a lot of work.

In other cases, the data is perfectly structured, seven years of data stored nicely and easily accessible, but the quality of the information itself, the single datum, was a mess. For example, when I was working with sensor data from a weather station, and the sensor was very unreliable. Plants overtook the weather station and the sensor itself was covered, so the data was not usable.

In the case of purely digital data, the most common problem is accessing the data. It can get complicated to access the data from a Facebook group, a business account, or an Instagram account. The regulations have gotten more strict over the years as these companies try to ensure data privacy. But obviously, those are your data, and you can request and access them. The advantage of this third party data is that it’s really well organized because it’s not the business that’s managing them. Someone else is doing a great job at handling them, and you can access them and use them almost immediately.

Don’t get me wrong, in any data science project; the data you receive is never perfect, meaning they are not already perfectly in shape to conduct analysis. It’s part of my job to restructure the data so that the model can understand it. However, as a data scientist, you’re a bit reluctant to work on cleaning the data. It’s the business’s job to keep them clean, and then we can use them in a project.

Thank you very much Ahmed, we hope to welcome you again on our blog for more insights on data science and marketing!

If you’d like to know more about Ahmed or his company, checkout his LinkedIn and Decatab website.

Are you interested in starting a data science project for your company? Get in touch with us to start making data-driven decisions for your business.

Continuing with our series on Data Science, today we are meeting with Ahmed Hammad: passionate data scientist and researcher specialized in business intelligence and environmental analysis. In this compelling interview, Ahmed will allow us to take a closer look at sentiment analysis, teaching us how to capture emotional responses toward our products or services. Decatab, his company, makes use of artificial intelligence (AI) to provide monitoring and evaluation solutions for environmental projects. 

We’ll discuss Data Science and sentiment analysis along with their impact on businesses. Through sentiment analysis and more generally Natural Language Processing, businesses can get an understanding of people’s opinion toward a given product, branding or market strategy from text only. 

Intrigued? Check it out!

Ahmed Hammad Interview: data science, sentiment analysis and human communication

First, a few words about Decatab. You have funded your company with the idea of providing high-tech solutions for environmental programs monitoring. How did you come up with it? What is the story behind Decatab?

I’ve been freelancing since the second year of my bachelor degree. At the time, I was working in the field of program evaluation for my university – collecting data, organising them and using econometrics to analyse the effect of a given program. Later during my PhD I decided to set up my own company while I was doing research for organisations and for my university. I was working on Natural Language Processing and I had the chance to work with the UK parliament. I developed this tool to analyse electronic petition data. They had an online petition system; they were not extracting all the valuable information from it. So I developed this dashboard based on Natural Language Processing to extract more insights. How do people react to this policy? What is the positive or negative impact of this new petition? 

A petition is like a post basically. I would monitor how people are talking about it, not only the number of comments, the numbers of likes or the times they tweeted, retweeted or whatever. It’s about what they are talking about. I was using Natural Language Processing to follow the variations in the topics and how they are reacting to a certain petition.

The main goal I see in any business or government is always to monitor and evaluate, it’s just that they use different terms.

Then I started being more involved in environmental projects and I liked it a lot. It’s easy for me to switch from one field to another because my knowledge is more abstract. I’m not specialized in one single technique that can be applied only in one single field, so I can easily jump from one field to the other. But I felt that the environmental field was the right choice for me and I believe you have more space for creating and innovating. 

What makes data science different from other statistics fields? How do you explain the recent boom of data science in businesses?

Data science is a very high-level data analysis in which huge volumes of data are used to extract knowledge and insights to solve critical issues or find answers to our questions . Automation is also another important aspect, and that’s one of the main reasons for the big boom of artificial intelligence. It can automate many things, many tasks. And, when there is automation, businesses get very interested.

Natural Language Processing vs. sentimental analysis. How are they linked? What is the difference between the two?

There is no versus. Natural Language Processing is a field of artificial intelligence that deals with the interaction between computers and humans on the communication level. Speech recognition and translation from one language to another are some of its applications that we use in our daily lives.

Sentimental analysis is a specific application, or subfield, which involves the understanding and the extraction of sentiment or feelings from mostly text. How do people feel toward a given piece of information? It could be about a product in the simplest case, a review or a comment. How do they react to this? 

Natural Language Processing is among the most complicated fields in machine learning and artificial intelligence because as humans, communication is the most complicated thing we do. Nothing or noone else communicates with our level of complexity. 

Sometimes [Natural Language Processing] can really drive you crazy. I remember that I was working on topic modelling, so extracting topics from sentences or documents. And I see the answer in front of me, I read it, it’s there! And the model doesn’t get it, it just doesn’t get it! I have tried something like seventy thousand ways and it still doesn’t get it. That’s why sometimes you need to take a break.

[Laughter]

So when you’re in a situation like this, where the model just doesn’t get it, what are your options? 

It’s a trial and error process. 

People see machine learning and artificial intelligence solutions as this big box where you just dump everything and it gets you all the answers that you want, but it doesn’t work like this. There are a lot of trials and errors, optimization, changing the parameters. Because these models have a lot of parameters, a slight change in a parameter can give completely different results. That’s why they are data-hungry. It takes a lot to learn the patterns behind the data and to find the optimal parameters and therefore, this also requires a lot of processing power if you want to speed up the process. Now with the boom of cloud computing you can access a massive amount of processing power so everything is possible.

The funny thing is that all these models were imagined back in the 90s 80s event 70s, at least the theory of it. It was just impossible to process all the data because of the amount of computational power it requires, and later it was just crazy expensive.

Now that companies have the ability to use this technology, how can they take advantage of it and make it work for their business?

Sentiment Analysis is the ‘computational’ study of an individual’s opinions, attitudes and more in general emotions toward a given topic or event. These opinions, attitudes or emotions take the form of reviews or comments (text). I have found that sentiment analysis is used a lot as a monitoring tool to track the progress of campaigns or as one of the main KPI to pick a new branding strategy.

You usually don’t do only a sentiment analysis. At the end, sentiment analysis by itself gives you a very good KPI. It’s very nice, it’s straightforward, but then, obviously, you want to have a deeper understanding. For example, some people have a positive sentiment, some have a negative sentiment about say a new product. What’s the negative part about? It’s extremely valuable to know what people don’t like about it, if I’m selling goods or a service.t. Because they are going to tell you, they are writing about this. So I can link the negative sentiments with their correspondent topics. Then I can tell you “it’s the handle of the mug, the customer service, the layout of the page, the payment system” or anything else. You can connect all these things and then you have an amazing picture. You can extract really useful information pretty quickly..

However, you can do crazy things with Natural Language Processing. Find things that are not even there. Think about that: if you only have a text and you don’t have information about who wrote that text, there are models that allow you to estimate the range of the age, the level of education, and the gender of the author. Purely from the text! It’s insane, when you think about it . You don’t have this information, and even if you don’t have it you can extrapolate it with a certain degree of reliability. 

And how do you go about training a model?

Basically, you need to have a data set with examples. Imagine a CSV file with two columns: one contains sentences, and in the other we have for each sentence whether it’s positive or negative. This is a training data set, something from which the model can learn. 

The simplest way is to divide the data set 50/50. You train the model on 50% of it, and then you test the model on the other 50%. This way you can see how the model performs, how much is the accuracy. If it’s not good enough, you start the loop again: you keep changing, trying other models, trying other parameters and so on, until you get good results. But you need to restructure the data in a way that the model can understand and learn from it. 

You can work on words or letters or full sentences. You will need to have one column for each word. Imagine that if you take just a review page, you have a crazy number of words, more than one thousand words, so you have more than one thousand columns.

Then you can model in different ways. Words are converted to numbers most of the time. Let’s say you have N letters in the alphabet – “A” becomes “1”, “B” becomes “2”, spaces become “0” and so on. Everything is transformed into its numerical correspondence to be understood by the model.

What about the level of accuracy? When the model gives you an opinion or a sentiment, does it come with a confidence level?

Yes, it always comes with it. Sentiment analysis is a classification task. You want to classify a piece of text within n categories. Often, these categories are  ‘positive’,’ negative’ and ‘neutral’. But then you can also use  happy’, ‘angry’, and so on. It’s a classification task so it can be literally any category . 

When you evaluate your model’s accuracy, you want to know the fraction of predictions the model got right. If the accuracy is very low I will need to train the model again with different algorithms or parameters.

Given that we got good results on the test data set – say we got an accuracy of 97% – we can be confident in our model’s performance on new data. But we should also consider that most classification tasks are done with so-called probabilistic models that provide an idea about the uncertainty associated with a given prediction. So every prediction comes with a probability of falling in each class or category. For example, I’m sure at 90% that this sentence falls in category A which is “positive”. This is the single prediction.   

As we mentioned, human communication can get very complex. How can a machine understand all the subtleties, irony for example? Is this handled by the machine learning models?

Yes, there have been a lot of studies in the last few years.

Understanding irony has been a challenge in the field. It can be a huge problem, especially for service companies. So if the model is not trained to understand irony, we could wrongly classify a negative comment as positive. It’s quite a challenge but today we can deal with it.

That’s amazing. Well Ahmed, thank you very much for your time, I really appreciate you answering all our questions. We will be meeting again very soon to talk about predictive analysis.

My pleasure!

If you’re interested in knowing more about Ahmed’s or his company, checkout his LinkedIn and Decatab website.

In the next article of this series, we’ll be meeting again with Ahmed to explore predictive analysis, or how you can predict future events for business purposes.

Think about the data companies have been collecting for the past 20 years: number of transactions for each physical point of sales, customer data through fidelity cards, offline advertisement campaigns, and so on.

Now think about the data collected today: digital data. This includes information about online purchases, behaviors of website visitors, click rates, use of social media, geolocation and much more. The volume and variety of this data is beyond comparison and offers opportunities never seen before.

The growth of e-commerce platforms, social media and the Internet of things (IoT) for example is a wealth of information for those who know how to manage and process this data.

If you are not already incorporating Big Data Analysis in your marketing strategy, you are missing out on opportunities to make better and faster decisions for your business.

Making sense of Big Data with Analytics

Big Data refers to the humongous and ever growing volume of data generated by digital consumers and collected by businesses. The particularity of this data is that it can be structured, unstructured, heterogenous and so complex that manual analysis by humans is impossible. 

For a time, this massive amount of data was incomprehensible and even though companies were collecting and storing data, they did not have the processing power to restore them into comprehensive information. 

Recently, the emergence of more sophisticated technologies enabled us to extract valuable business information from Big Data collected over time. Big Data Analysis refers to the techniques, processes, tools and technologies to make sense of this massive amount of data. 

It is an advanced analytical system that detects hidden patterns, correlations and other meaningful insights, enabling to pilot corporate activities in a data driven manner. This new form of business intelligence helps with decision making, improves precision and reaction time to market changes.

How to use Big Data analytics to drive your marketing growth

Consumers create data with every action they take online. For your marketing team, this is a great opportunity to better understand target audiences and improve the value of customer interactions.

Big Data analytics calls for a new approach to marketing research and customer segmentation. Two major customer data source are available: 

• First party data owned by your company: it comes from customers who interacted with your company and lives in your CRM, website or social media analytics tools;
• Second party data owned by other corporations: companies now make a business of collecting and exchanging customer data (Google or Facebook for instance).

With more data available, along with the right tools to analyse and combine information, marketers can identify and target prospects more effectively. The increased granularity in customer profiles and data analysis in near real time lets us deliver a highly personalized experience to drive high ROI marketing campaigns.

However, a large volume of data does not automatically lead to a better marketing strategy or an increase of sales. Big data is the raw material essential to conscious decision making.

It is not the data itself that matters but the way is it manipulated to serve your marketing needs.

There are 4 different methods to approach Big Data analytics based on your business case and objectives. They are used at different stages of building your data driven marketing strategy.

1. Descriptive analysis: What happened?

Descriptive analysis is the most common form of data analysis. As its name implies, it is used to give an overview of past events in your company. 

Think monitoring dashboards, sales reports or marketing campaigns analysis. Descriptive analysis uses historical data collected by your company and presents it in a comprehensive format so it can easily be interpreted by a business audience.

This type of data analysis allows you to learn from your past behaviors and better understand how your company performs in a given activity.

On its own, descriptive analysis is not used for business insight but lays the foundation for more actionable prescriptive or predictive analysis. 

2. Diagnostic analysis: Why did it happen?

Diagnostic analysis is used to understand the root cause of an event or phenomenon. If your company has identified a particular problem, a diagnostic analysis will provide in depth insight regarding what caused the issue in the first place.

Let’s say the descriptive analysis of your website checkout process has highlighted a high rate of abandoned carts on your website. Is it because of the shipping costs, a website error, delivery times? There are so many possible causes it can be a daunting task to consider them all. A diagnostic analysis will study visitors behavior in depth and point you to the exact reason why they don’t proceed to purchase so you can quickly fix the issue and increase the conversions of your online shop. 

3. Predictive analysis: What might happen next?

Predictive analysis leverages artificial intelligence (AI) and machine learning (ML) to produce business projections. Massive amounts of data are fed to statistical models trained to recognise trends and patterns. Based on recent and historical data, the model is able to forecast future business trends. 

Predictive analysis is not an exact science, no algorithm can predict future events with 100% accuracy. The results are given with a level of probability that the prediction will eventually happen. The purpose is not to predict what WILL happen but rather what MIGHT happen in the future. It extrapolates data to predict a trend line.

Big Data marketing uses predictive analysis to understand customer behaviors and future trends, allowing them to plan their campaign accordingly and seize opportunities in real time.

4. Prescriptive analysis: What actions should you take?

Prescriptive analysis takes Big Data analytics to the next level. It is used to predict the possible outcomes of different choices of action. 

After your predictive analysis, your team has probably produced various plans of action to take advantage of the upcoming trends identified. Prescriptive analysis allows you to choose the most efficient course of action for your business to reach its desired outcome.

It is a form of predictive analysis, but this time it forecasts the effect of multiple decisions you might want to take. Prescriptive analysis is very complex to implement and few companies are currently using it but we believe it is the future of Big Data Analytics.

Big Data and data science prediction: the impact on sales

Working hand in hand with data scientists that understand your business has become essential to stay ahead of competition. The ability to capture and analyse data efficiently allows companies to identify problems or opportunities to seize and take informed decisions.

Making an educated use of data has a direct impact on your sales performance. Here are a few examples of how companies leverage Big Data analytics to increase sales:

Adjust price in real time: Find the right price point for a product or service to take advantage of every sales opportunity and ensure maximal profitability. Dynamic pricing techniques are used to adjust the price of a product based on multiple factors such as customer interest level, seasonality, stock available or competitors’ price and promotions. 

Reduce customer churn: Predict customer behavior to improve retention. Predictive analysis can study changes in customers interactions with your company or product and identify any signs of them stopping the commercial relationship. Reduce customer churn before it happens by giving customers at risk the attention they need.

Generate and prioritize leads: Target buyers that are actively looking for offers similar to yours by crossing data on existing customer profiles and prospects online activity. Predictive lead qualification creates a score based on ease of conversion and profit potential -aka lead scoring- to automatically assess and prioritize leads.

Repeat sales and cross sales: Identify customers more likely to buy multiple products. Find patterns in buyers behaviors and profiles to determine which customers are more likely to purchase your other products. 

At Mowgli we can help you elaborate a better marketing and sales strategy based on data. We work with top notch data scientists to guide you through any stage of your Big Data journey. If you would like to know more about how we can assist, please get in touch!