Continuing with our series on Data Science, today we are meeting with Ahmed Hammad: passionate data scientist and researcher specialized in business intelligence and environmental analysis. In this compelling interview, Ahmed will allow us to take a closer look at sentiment analysis, teaching us how to capture emotional responses toward our products or services. Decatab, his company, makes use of artificial intelligence (AI) to provide monitoring and evaluation solutions for environmental projects.
We’ll discuss Data Science and sentiment analysis along with their impact on businesses. Through sentiment analysis and more generally Natural Language Processing, businesses can get an understanding of people’s opinion toward a given product, branding or market strategy from text only.
Intrigued? Check it out!
First, a few words about Decatab. You have funded your company with the idea of providing high-tech solutions for environmental programs monitoring. How did you come up with it? What is the story behind Decatab?
I’ve been freelancing since the second year of my bachelor degree. At the time, I was working in the field of program evaluation for my university – collecting data, organising them and using econometrics to analyse the effect of a given program. Later during my PhD I decided to set up my own company while I was doing research for organisations and for my university. I was working on Natural Language Processing and I had the chance to work with the UK parliament. I developed this tool to analyse electronic petition data. They had an online petition system; they were not extracting all the valuable information from it. So I developed this dashboard based on Natural Language Processing to extract more insights. How do people react to this policy? What is the positive or negative impact of this new petition?
A petition is like a post basically. I would monitor how people are talking about it, not only the number of comments, the numbers of likes or the times they tweeted, retweeted or whatever. It’s about what they are talking about. I was using Natural Language Processing to follow the variations in the topics and how they are reacting to a certain petition.
The main goal I see in any business or government is always to monitor and evaluate, it’s just that they use different terms.
Then I started being more involved in environmental projects and I liked it a lot. It’s easy for me to switch from one field to another because my knowledge is more abstract. I’m not specialized in one single technique that can be applied only in one single field, so I can easily jump from one field to the other. But I felt that the environmental field was the right choice for me and I believe you have more space for creating and innovating.
What makes data science different from other statistics fields? How do you explain the recent boom of data science in businesses?
Data science is a very high-level data analysis in which huge volumes of data are used to extract knowledge and insights to solve critical issues or find answers to our questions . Automation is also another important aspect, and that’s one of the main reasons for the big boom of artificial intelligence. It can automate many things, many tasks. And, when there is automation, businesses get very interested.
Natural Language Processing vs. sentimental analysis. How are they linked? What is the difference between the two?
There is no versus. Natural Language Processing is a field of artificial intelligence that deals with the interaction between computers and humans on the communication level. Speech recognition and translation from one language to another are some of its applications that we use in our daily lives.
Sentimental analysis is a specific application, or subfield, which involves the understanding and the extraction of sentiment or feelings from mostly text. How do people feel toward a given piece of information? It could be about a product in the simplest case, a review or a comment. How do they react to this?
Natural Language Processing is among the most complicated fields in machine learning and artificial intelligence because as humans, communication is the most complicated thing we do. Nothing or noone else communicates with our level of complexity.
Sometimes [Natural Language Processing] can really drive you crazy. I remember that I was working on topic modelling, so extracting topics from sentences or documents. And I see the answer in front of me, I read it, it’s there! And the model doesn’t get it, it just doesn’t get it! I have tried something like seventy thousand ways and it still doesn’t get it. That’s why sometimes you need to take a break.
[Laughter]
So when you’re in a situation like this, where the model just doesn’t get it, what are your options?
It’s a trial and error process.
People see machine learning and artificial intelligence solutions as this big box where you just dump everything and it gets you all the answers that you want, but it doesn’t work like this. There are a lot of trials and errors, optimization, changing the parameters. Because these models have a lot of parameters, a slight change in a parameter can give completely different results. That’s why they are data-hungry. It takes a lot to learn the patterns behind the data and to find the optimal parameters and therefore, this also requires a lot of processing power if you want to speed up the process. Now with the boom of cloud computing you can access a massive amount of processing power so everything is possible.
The funny thing is that all these models were imagined back in the 90s 80s event 70s, at least the theory of it. It was just impossible to process all the data because of the amount of computational power it requires, and later it was just crazy expensive.
Now that companies have the ability to use this technology, how can they take advantage of it and make it work for their business?
Sentiment Analysis is the ‘computational’ study of an individual’s opinions, attitudes and more in general emotions toward a given topic or event. These opinions, attitudes or emotions take the form of reviews or comments (text). I have found that sentiment analysis is used a lot as a monitoring tool to track the progress of campaigns or as one of the main KPI to pick a new branding strategy.
You usually don’t do only a sentiment analysis. At the end, sentiment analysis by itself gives you a very good KPI. It’s very nice, it’s straightforward, but then, obviously, you want to have a deeper understanding. For example, some people have a positive sentiment, some have a negative sentiment about say a new product. What’s the negative part about? It’s extremely valuable to know what people don’t like about it, if I’m selling goods or a service.t. Because they are going to tell you, they are writing about this. So I can link the negative sentiments with their correspondent topics. Then I can tell you “it’s the handle of the mug, the customer service, the layout of the page, the payment system” or anything else. You can connect all these things and then you have an amazing picture. You can extract really useful information pretty quickly..
However, you can do crazy things with Natural Language Processing. Find things that are not even there. Think about that: if you only have a text and you don’t have information about who wrote that text, there are models that allow you to estimate the range of the age, the level of education, and the gender of the author. Purely from the text! It’s insane, when you think about it . You don’t have this information, and even if you don’t have it you can extrapolate it with a certain degree of reliability.
And how do you go about training a model?
Basically, you need to have a data set with examples. Imagine a CSV file with two columns: one contains sentences, and in the other we have for each sentence whether it’s positive or negative. This is a training data set, something from which the model can learn.
The simplest way is to divide the data set 50/50. You train the model on 50% of it, and then you test the model on the other 50%. This way you can see how the model performs, how much is the accuracy. If it’s not good enough, you start the loop again: you keep changing, trying other models, trying other parameters and so on, until you get good results. But you need to restructure the data in a way that the model can understand and learn from it.
You can work on words or letters or full sentences. You will need to have one column for each word. Imagine that if you take just a review page, you have a crazy number of words, more than one thousand words, so you have more than one thousand columns.
Then you can model in different ways. Words are converted to numbers most of the time. Let’s say you have N letters in the alphabet – “A” becomes “1”, “B” becomes “2”, spaces become “0” and so on. Everything is transformed into its numerical correspondence to be understood by the model.
What about the level of accuracy? When the model gives you an opinion or a sentiment, does it come with a confidence level?
Yes, it always comes with it. Sentiment analysis is a classification task. You want to classify a piece of text within n categories. Often, these categories are ‘positive’,’ negative’ and ‘neutral’. But then you can also use happy’, ‘angry’, and so on. It’s a classification task so it can be literally any category .
When you evaluate your model’s accuracy, you want to know the fraction of predictions the model got right. If the accuracy is very low I will need to train the model again with different algorithms or parameters.
Given that we got good results on the test data set – say we got an accuracy of 97% – we can be confident in our model’s performance on new data. But we should also consider that most classification tasks are done with so-called probabilistic models that provide an idea about the uncertainty associated with a given prediction. So every prediction comes with a probability of falling in each class or category. For example, I’m sure at 90% that this sentence falls in category A which is “positive”. This is the single prediction.
As we mentioned, human communication can get very complex. How can a machine understand all the subtleties, irony for example? Is this handled by the machine learning models?
Yes, there have been a lot of studies in the last few years.
Understanding irony has been a challenge in the field. It can be a huge problem, especially for service companies. So if the model is not trained to understand irony, we could wrongly classify a negative comment as positive. It’s quite a challenge but today we can deal with it.
That’s amazing. Well Ahmed, thank you very much for your time, I really appreciate you answering all our questions. We will be meeting again very soon to talk about predictive analysis.
My pleasure!
If you’re interested in knowing more about Ahmed’s or his company, checkout his LinkedIn and Decatab website.
In the next article of this series, we’ll be meeting again with Ahmed to explore predictive analysis, or how you can predict future events for business purposes.
Our free video audit will take a look at your website and the UX-design of your website. We will send you a short video with an analysis from one of our talented team-members.
