I worked on the Biden dataset first and planned to implement the same steps on the Trump dataset given everything went well the first time round. parallel, for parallel processing of memory consuming functions. stringr, for stripping symbols from tweets. tm, wordcloud, syuzhet are for text mining processes.
But I soon realised that although I understood the fundamentals, I would need to follow a different set of steps tailored to the dataset I was dealing with.įirst, all the necessary libraries were downloaded to run the various transformation functions.
Every dataset is different – I followed a clear guide on sentiment analysis posted by Sanil Mhatre. But I continued to use it for improved efficiency even after moving to RStudio Server, as it still proved to be useful.ģ. Parallel processing – I first used the ‘parallel’ package as a quick fix for memory problems encountered creating the corpus.
Clean text tm r data frame corpus mac#
I used RStudio Server on my Mac to access a larger CPU for the size of data at hand.Ģ. Memory space – Your laptop may not provide you the memory space you need for mining a large dataset in RStudio Desktop. But before I begin, there are some things I had to think about for processing this type of data in R:ġ. I will outline the process of transforming the unstructured tweets into a more intelligible collection of words, from which sentiments could be extracted. There was a total of 1.72 million tweets, meaning plenty of words to extract emotions from. These tweets were collected using the Twitter API where the tweets were split according to the hashtags ‘#Biden’ and ‘#Trump’ and updated right until four days after the election – when the winner was announced after delays in vote counting. I found the Twitter data on Kaggle, containing two datasets: one of tweets made on Donald Trump and the other, Joe Biden. My main aim was to perform sentiment analysis on these tweets to gain a consensus on what US citizens were feeling in the run up to the elections, and whether there was any correlation between these sentiments and the election outcome. There were over a million tweets made about Donald Trump and Joe Biden which I put through R’s text mining tools to draw some interesting analytics and see how they measure up against the actual outcome – Joe Biden’s victory.
Which brings us to my analysis here on a dataset of tweets made regarding the US elections that took place in 2020. Text mining can also help us extract sentiment behind tweets and understand people’s emotions towards what is being sold. Text mining uses NLP techniques to transform unstructured data into a structured format for identifying meaningful patterns and new insights.Ī fitting example would be social media data analysis since social media is becoming an increasingly valuable source of market and customer intelligence, it provides us raw data to analyse and predict customer needs. Unstructured data needs to be interpreted by machines in order to understand human languages and extract meaning from this data, also known as natural language processing (NLP) – a genre of machine learning. So, I decided to experiment with some data in the programming language R with its text mining package “tm” – one of the most popular choices for text analysis in R, to see how helpful the insights drawn from the social media platform Twitter were in understanding people’s sentiment towards the US elections in 2020.
Since 80% of data out there is in unstructured format, text mining becomes an extremely valuable practice for organisations to generate helpful insights and improve decision-making. As data becomes increasingly available in the world today the need to organise and understand it also increases.