Feature Extraction on Sentiment Attitude Values to Better Predict the Stock Market Using Twitter Sentiment

Behavioral economics suggests that emotions can affect an individual’s decision making. Recent research on this idea’s application on large societies hints that there may exist some correlation or maybe even some causation relationship between public sentiment—at least what can be pulled from Twitter—and the movement of the stock market. One major result of consistent research on whether or not public sentiment can predict the movement of the stock market is that public sentiment, as a feature, is becoming more and more valid as a variable for stock-market-based machine learning models. While raw values typically serve as invaluable points of data, when training a model, many choose to “engineer” new features for their models—deriving rates of change or range values to improve model accuracy.
Since it doesn’t hurt to attempt to utilize feature extracted values to improve a model (if things don’t work out, one can always use their original features), the question may arise: how could the results of feature extraction on values such as sentiment affect a model’s ability to predict the movement of the stock market? This paper attempts to shine some light on to what the answer could be by deriving TextBlob sentiment values from Twitter data, and using Granger Causality Tests and logistic and linear regression to test if there exist a correlation or causation between the stock market and features extracted from public sentiment.