Finding an Edge: Data Analysis of the Stock Market

Share:
Indexes
  1. Alpha Vantage — The historical data alpha beast for stock tickers and ETFs

The London skyline was a blur of lights and energy as the sun began to set, but one person was not interested in the beauty of the city. He had a mission — to uncover hidden secrets and the truth behind the numbers…

Puzzling Sherlock Holmes — Image made with Lexica Aperture

James Bond(ians) and Sherlock Holmes(ians) — Frames to work with

James Bond movies — James Bond movies, starting with “Dr. No” in 1962 and continuing with the latest film, “No Time To Die” in 2021¹ — and Sherlock Holmes novels — written by Sir Arthur Conan Doyle and first published in 1892 and more recent adaptations like the TV show “Elementary”² — are two of the most iconic, thrilling and popular entertainment genres. These genres have captivated audiences for decades with their thrilling blend of action, espionage and high-tech gadgetry, to the ingenious detective work of remarkable powers of observation, deduction and forensic skills, keeping audiences on the edge of their seats through a common theme: uncovering secrets and mysteries that lie beneath the obvious, overcoming obstacles of all kinds, and following a path of patterns to the very core.

Just as James Bond is a witty, clever and action-packed character who uses the most modern tools, Sherlock Holmes — no less witty or clever — is also a well known master of deduction, the ability to use logic and a keen, astute observation of the smallest details, the ability to think outside the box and introduce unconventional methods to solve difficult and often seemingly impossible cases.

Although they represent two distinctly different strategies for solving crimes: one through physical force or intellect, and the other through deductive reasoning. This makes them perfectly complementary characters in terms of style and approach. They have become archetypes, or rather blueprints, for any kind of mystery-solving detective approach today, and their influence can be seen in many other forms of popular culture, including video games, comics, and novels, which often use their characters as inspiration.

or…

Welcome to the world of exploratory data analysis.

Futuristic data science analysis setup — Image made with Lexica Aperture

EDA — Crime-Solving Data Analysis

In the same way that James Bond and Sherlock Holmes use their unique skills and strategies to uncover secrets and solve puzzles, Exploratory Data Analysis (EDA) is a powerful tool for discovering patterns, gaining insights and identifying trends in data, providing a deeper understanding of the underlying information by using a combination of various graphical techniques and statistical methods to explore the properties of a dataset. Having become a popular — and increasingly so — tool over the past few decades, EDA is being used in a wide range of fields, from healthcare to marketing and last but not least finance and economics and is becoming an essential skill for anyone looking to make sense of the vast amount of information available today that can be difficult to make sense of.

EDA is not only a powerful tool for understanding data in general, but it is also particularly useful and widely used for making sense of the intricacies of the complex and dynamic equity and derivatives markets that drive prices, the clearest mirror, the most vivid manifestation of human behavior — always enigmatic, ever puzzling. With EDA as a key tool, analysts and investors can put the pieces together to gain a more complete understanding of the market, and make more informed and strategic decisions about buying and selling stocks. With EDA, you have the tools to navigate the complex and dynamic world of the stock market and increase your chances of success, providing a more holistic and accurate picture of the information.

Think of EDA as a thrilling detective work, where you dig deeper and deeper into the data, following leads and eliminating suspects, until you finally uncover the hidden truth. It’s an iterative process, similar to a private investigator’s case file, where you can use a variety of techniques, each adding a new piece of information to the investigation. And just like a detective, you will stumble across new questions and theories that need to be tested and falsified with further data analysis or experiments, the initial step before the big showdown takes center stage, a showdown between the secret agent and archenemy, the nemesis — the data analyst the big players in the stock market.

Just like in any action-packed spy thriller or puzzle solving mystery.

“When you have eliminated the impossible, whatever remains, however improbable, must be the truth.”

Observation is the key, identification of patterns is the goal, logical reasoning and the scientific method are the means, falsification of hypotheses is the procedure.

NASDAQ by bfishadow on Flickr, CC BY 2.0 <https://creativecommons.org/licenses/by/2.0>, via Wikimedia Commons

The stock market

As the most liquid and risk-free capital market in the world, the stock market is one of the most important economic institutions in the world, a place where people can speculate on the future direction of a company’s stock. If you think a certain company will do well in the future, you can buy its shares and hope to make money if that prediction comes true, but it is also a place where anything can happen.

It is like a black box in aviation, shrouded in complexity and unpredictability. Like a black box, the stock market operates on a set of rules and inputs, but its inner workings and decision-making processes remain largely elusive to the average person. Often influenced by a wide range of factors, the market can be highly volatile and subject to sudden changes in direction, with prices soaring and plummeting like a rollercoaster. This volatility can make it difficult for investors to predict market movements and can lead to high levels of risk due to the underlying nature of uncertainty.
However, it’s important to note that the stock market is not a closed system, and with the vast amount of information available to the public, it’s a playground for data analysis.

The S&P 500, also known as the Standard & Poor’s 500, is a stock market index made up of the 500 largest publicly traded companies in the United States. It is considered a leading indicator of the overall performance of the US stock market. The companies included in the index are selected on the basis of their market capitalisation and financial stability. Companies in the S&P 500 include Apple, Microsoft, Amazon and Coca-Cola, and cover a wide range of sectors including technology, healthcare, finance and consumer goods. As the S&P 500 is broadly used as a benchmark for index funds and exchange-traded funds (ETFs) by tracking the performance of the index, it is also important to note that the S&P 500 is a capitalisation-weighted index, which means that companies with a higher market capitalisation (value) have a greater impact on the performance of the index.

Motivation

First, as an enthusiast of stock market analysis, I have always been drawn to the works of Nassim Nicholas Taleb and Edward Thorp, two of the most prominent figures in the field of finance and probability. Among others, Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets³ by Taleb is a classic in the field of finance and provides a comprehensive understanding of the role that chance and probability play in the stock market.

Similarly, Thorp’s A Man for All Markets⁴ is a fascinating account of his journey as a mathematical prodigy and successful investor. It highlights the importance of using quantitative models and tools, such as probability and statistics, to make wise investment decisions. Both authors have also shown me the power of thinking outside the box and using unconventional methods to beat the market.

Not only have these traders made significant contributions to the field of stock market analysis, but these authors and their books have been instrumental in shaping my understanding of the stock market and have inspired me to delve deeper into the world of stock market analysis, chance, probability and randomness.

They taught me the importance of understanding the underlying patterns and trends in the market, the potential pitfalls of relying too heavily on past performance and conventional wisdom, and inspired me to further explore the world of stock market analysis from the perspective of (non-)randomness.

As an avid follower of futurology, the stock market and the future itself, I am eager to apply the lessons learned to my own research. My focus will be on intraday trading, and I plan to show retail traders how to develop a winning strategy with a highly probabilistic outcome.

I believe that by understanding the underlying principles of chance and probability, we can gain a better understanding of the stock market and develop more effective trading and investment strategies both today and in the future.

Why do I believe this? Let’s list some of the strategies used by retail traders and some of those used by hedge funds and investment banks.

Analyzation and indicators commonly used by retail traders:

  • Moving Averages
  • Relative Strength Index (RSI): magnitude of momentum
  • Stochastic Oscillator: magnitude of momentum
  • Bollinger Bands: momentum and volatility
  • MACD (Moving Average Convergence Divergence): trend-following momentum
  • Fibonacci Retracements: predictions on “numerical” patterns like sea waves
  • Candlestick Charts: a mode of visualization support and resistance levels
  • Pivot Points: calculation of potential levels of support and resistance
  • Ichimoku Clouds: yet another indication of support and resistance levels
  • Volume Analysis

Strategies commonly used by hedge funds:

  • Long/Short Equity: risk-avoiding, smoothed and balanced diversification of the portfolio
  • Event-Driven: exploiting inefficiencies in the market
  • Global Macro: profiting from large-scale movements in the global economy.
    Arbitrage: exploting inefficiencies or reversion to the mean strategies
  • Distressed Investing: aggressive betting on turn-arounds
  • Quantitative/Systematic: algorithmic trading based on mathematical models

Do you see a difference?

Goals

The S&P500, in the form of the SPY-ETF (Spy: Spyder Exchange Traded Fund emitted by the State Street Global Advisors Trust Company in 1993⁵), will therefore be the main subject of some of the forthcoming articles, but the analyses and concepts are generally applicable to any other asset, with a particular focus on intraday trading.

Throughout the articles, I will provide readers with code sequences in Python and its powerful libraries for data analysis and visualization, and I will illustrate concepts and applications to real-world trading scenarios, eg. stock prices, trading volume, economic indicators, news articles, sentiment analysis and other conditions that can affect the market. We will also explore the work of legendary traders and provide examples and case studies.

This series of articles aims to demystify the stock market by using scientific methods to understand and navigate it. We will cover key concepts such as odds, probabilities and randomness in trading and explore the different analytical methods and detective techniques used, most of which are already known and used by proprietary traders, investment banks and hedge funds, with a focus on discretionary intraday trading.

By highlighting the importance of understanding the underlying patterns, trends and correlations in the data, traders can identify potential profit opportunities and risks that need to be mitigated. Traders can also use this information to develop their own profitable and effective strategies for skewed risk/reward ratios. This exploratory data analysis will yield powerful, yet often simple, insights into the financial markets.

Like any other data scientists conducting EDA, we have to keep an open mind and go wherever the data, the hints, the clues lead us. For this reason, this series of articles will not have a clearly structured outline to tell the reader which step, which aspect we’re going to cover next.

The methodology may or may not include some, all or none of the following steps:

  • Collecting and cleaning historical stock market data from various sources such as aggregated, ICE, Yahoo Finance or FRED
  • Exploring the data using visualization techniques such as line plots, bar plots, and heatmaps to identify patterns and trends
  • Using statistical techniques such as descriptive statistics, correlation analysis, and hypothesis testing to quantify the relationships between different variables in the search for frequencies, specific chracteristics and in particular outliers
  • Building and testing quantitative models using the programming language Python
  • Using machine learning techniques such as regression analysis, decision trees, and neural networks to predict future market movements (I must admit though I’m not a big fan of certain types of machine learning approaches)
  • Interpreting the results and drawing conclusions to inform investment decisions and develop strategies

Just the starting point is clear: as shown in my previous article, after collecting the SPY data, we will next explore and visualise the overnight gaps as a starting point, and proceed step by step from there.

Science Cats — Image made with Lexica Aperture

Lookout

A master of data analysis is on the case. Join me as I embark on a thrilling data adventure into the mysterious world of the stock market. Armed with my trusty Python and expertise in data analysis, I am determined to uncover hidden patterns and correlations that could reveal the truth. But be warned, this journey is not for the faint of heart. The world of investing and trading is full of surprises, secrets to uncover, and mysteries to solve, or isn’t it? As we delve deeper into the SPY-game, we must be prepared to face the unexpected and formulate new ideas and hypotheses. This is not a mission for those who lack courage in the face of uncertainty, as randomness and volatility are inherent in the stock market…

But for those who are willing to take on the challenge, the rewards can be great. Remember, this is not investment advice.

p.s.: Have any ideas on what to “discover” in future articles or want to know or discuss anything in specific? Let me know in the comment section or via pm.

Thanks for your time and I hope you have enjoyed reading this article and hopefully you have learned something. If you have anything to add, to correct or to ask? Don’t hesitate to contact me and let’s discuss!

If you’re interested in reading articles about stocks, data science, python et al., consider following me on Medium.

Cheers,
Constantin

Acknowledgements

I would like to thank my mentor for always pushing me to look deeper into the data and to never stop thinking in terms of probabilities.

References

¹ Wikipedia contributors. (2023, January 22). James Bond. Wikipedia. https://en.wikipedia.org/wiki/James_Bond
² Wikipedia contributors. (2023, January 22). Sherlock Holmes. Wikipedia. https://en.wikipedia.org/wiki/Sherlock_Holmes
³ Taleb, N. N. (2005). Fooled by randomness: The hidden role of chance in life and in the markets (Vol. 1). Random House Trade Paperbacks. https://amzn.to/403JVwn
⁴ Thorp, E. O. (2017). A Man for All Markets: Beating the Odds, from Las Vegas to Wall Street. Simon and Schuster. https://amzn.to/3HpyO8P
Spy: SPDR® S&P 500® ETF Trust. State Street Global Advisors. (n.d.). Retrieved January 17, 2023, from https://www.ssga.com/us/en/intermediary/etfs/funds/spdr-sp-500-etf-trust-spy