Deep Learning vs. Traditional Methods: Enhancing Stock Return Forecasts in Japan's Financial Landscape | Papers With Backtest: An Algorithmic Trading Journey

Description

Are you ready to unlock the secrets of stock market prediction using cutting-edge technology? In this episode of Papers With Backtest: An Algorithmic Trading Journey, we delve deep into the transformative paper "Deep Learning for Forecasting Stock Returns in the Cross-Section" by Abe and Nakayama, where the potential of deep learning techniques is put to the test in the realm of Japanese stock performance. This episode is a must-listen for algorithmic trading enthusiasts and data scientists alike, as we dissect the intricate methodologies that bridge finance and technology.

Our discussion centers around a comprehensive dataset that encompasses constituents of the MSCI Japan Index, enriched by 25 standard financial factors tracked over a significant period from December 1990 to November 2016. We explore how these inputs serve as the backbone for predictive modeling, and how deep neural networks (DNNs) stack up against traditional machine learning methods like support vector regression (SVR) and random forests (RF). The insights gained from our analysis reveal that deeper neural networks generally outperform their shallower counterparts, providing a fascinating glimpse into the future of algorithmic trading.

Throughout the episode, we scrutinize various neural network architectures and their effectiveness in enhancing predictive accuracy and achieving superior risk-adjusted returns in simulated trading strategies. The conversation takes a critical turn as we emphasize the often-overlooked impact of transaction costs in real-world applications, a crucial factor for any algorithmic trader aiming for profitability. As we navigate through the complexities of stock return forecasting, we also suggest intriguing avenues for future research, including the potential of recurrent neural networks and other advanced architectures that could revolutionize the field.

Join us as we reflect on the robustness of deep learning advantages in stock prediction, and what this means for the future of finance and algorithmic trading. Whether you’re a seasoned trader or a curious newcomer, this episode is packed with insights that could reshape your understanding of market forecasting. Don’t miss the chance to elevate your trading strategies with the knowledge shared in this enlightening discussion on Papers With Backtest: An Algorithmic Trading Journey.

Tune in now and discover how deep learning is not just a buzzword but a game-changer in the world of stock market predictions!

Hosted on Ausha. See ausha.co/privacy-policy for more information.

Transcription

Speaker #0
Hello, welcome back to Papers with Backtest podcast. Today we dive into another algo trading research paper.
Speaker #1
We do indeed.
Speaker #0
We're going to unpack deep learning for forecasting stock returns in the cross-section by Abe and Nakayama from 2018.
Speaker #1
A really interesting one.
Speaker #0
Yeah, this paper looks into whether using deep learning can, you know, give us a better edge predicting Japanese stock performance over the next month.
Speaker #1
Exactly. It's fundamentally asking, can these newer, more complex machine learning tools actually beat the traditional methods, especially for forecasting returns.
Speaker #0
Right. So for you listening, the goal today is figuring out if this deep learning approach really is a better way to predict stock moves compared to older methods and, well, other ML techniques too.
Speaker #1
Yeah, we're going to really focus on what this means for building actual trading rules and strategies, and importantly, the backtest results the authors found.
Speaker #0
Okay, so first things first, what data were they using?
Speaker #1
They used a pretty solid data set. It was constituents of the MSCI Japan Index. And they looked at 25 different factors. You know, the usual suspects, book-to-market ratio, earnings-to-price, past returns, like the last month's return.
Speaker #0
Just standard factors mostly.
Speaker #1
Pretty standard, yeah. Yeah. Listed in their table one. And the data covered a decent timeframe, December 1990 through November 2016.
Speaker #0
That's quite a stretch. Now, I saw something about a lag, a four-month lag.
Speaker #1
Yeah. Yes. That's quite important. They built in a four-month delay.
Speaker #0
Why is that?
Speaker #1
Well, it's about realism. The idea is even if data is released, it takes time for investors to actually get it, digest it, and act on it. So this lag tries to mimic that real-world delay.
Speaker #0
Makes sense. So the core problem was predicting next month's return.
Speaker #1
Using those 25 factors, but not just the latest values, they looked at the factors from the last five time points.
Speaker #0
Five points. How far apart?
Speaker #1
Three-month intervals. So looking back over what, about 15 months of factor history?
Speaker #0
Okay, so capturing some trend information potentially, and they pre-processed the data.
Speaker #1
They did. They rescaled both the inputs, the factors, and the outputs, the returns, by ranking them. Yeah, ranking across all the stocks at each point in time. Kind of normalizes things, helps the models focus on relative performance rather than absolute values. Got it.
Speaker #0
Okay, let's get to the good stuff. The trading rules and the back tests. They looked at different neural network setups.
Speaker #1
They did. They compared deep neural networks, DNNs, with eight layers, DNN8. and five layers DNN5.
Speaker #0
Against shallower ones.
Speaker #1
Yeah, exactly. Against a three-layer network, NN3. And they tested that NN3 both with and without something called dropout.
Speaker #0
Ah, dropout. That's for preventing overfitting, right?
Speaker #1
Precisely. It randomly ignores some connections during training, makes the network a bit more robust, hopefully less likely to just memorize the training data.
Speaker #0
So what did they find when they compared these deep versus shallow networks? How did they perform?
Speaker #1
Well, looking at their tables four and five, the general trend was Deeper seemed better.
Speaker #0
More layers, better predictions.
Speaker #1
Generally, yes. Performance measured by rank correlation, they called it ERR and directional accuracy, tended to improve as the networks got deeper.
Speaker #0
And COERA is?
Speaker #1
Rank correlation. Basically, how well the model's predicted ranking of stocks matches the actual ranking based on future returns. Higher is better.
Speaker #0
And directional accuracy?
Speaker #1
Simply, the percentage of times the model correctly guessed if a stock would go up or down relative to the median. Okay. And interestingly, even shallow networks with a similar number of parameters didn't quite match up to the DNNs.
Speaker #0
So it wasn't just about the number of knobs to turn, but the structure?
Speaker #1
It seems that way. The depth itself appeared to add value. Right. And the directional accuracy was statistically significant for all the network types, which is encouraging.
Speaker #0
Suggests they were definitely picking up something. Right.
Speaker #1
And the mean squared error, the loss function, was generally lower for the deeper networks too.
Speaker #0
Which... Specific models did best. I see DNN83 mentioned for CR.
Speaker #1
Yeah, DNN83 hit the highest CR at 0.0591.
Speaker #0
Now, 0.0591, that sounds quite small.
Speaker #1
It does, but in this field, forecasting cross-sectional returns, even a small consistent edge like that can be meaningful. It's notoriously difficult.
Speaker #0
Okay, fair enough. And the directional accuracy for that model?
Speaker #1
For DNN83, it was about 52.6% for tertiles, splitting stocks into three groups. And a bit better, 53.4% roughly for quintiles, so five groups.
Speaker #0
Still only slightly better than a coin flip, but statistically significant, you said.
Speaker #1
Exactly. Better than chance, consistently.
Speaker #0
Okay, so DNNs show promise, but they compared them to other ML methods too.
Speaker #1
They did, yeah. Crucial comparison point. They used support vector regression, SVR, and random forests RF as benchmarks. Pretty standard powerful ML techniques.
Speaker #0
And did they tune those models properly?
Speaker #1
Seems so. They mentioned testing 24 different setups for SVR and 37 for random forests to try and find the best configuration for each.
Speaker #0
So what were the best results for SVR and RF?
Speaker #1
For SVR, the best TOR was around 0.0569 directional accuracy, similar to the DNN, so maybe slightly lower, around 52.5% and 53.3%.
Speaker #0
And random forests?
Speaker #1
RF did a bit better on TOR, 0.0576. Directional accuracy was quite close to the best DNNs, actually, around 52.6% and 53.4%.
Speaker #0
Okay, so RF looks pretty competitive there. How did the DNN stack up overall against these?
Speaker #1
This is where it gets interesting. Several of the top DNN patterns, like DNN 81, 83, 84, and 51, they generally outperformed SVR on most metrics. And against random forests, it was closer. But some DNNs showed slight advantages, especially in TRR. Specifically, DNN84 actually beat the best RF model on all the metrics in their comparison table, table six.
Speaker #0
Really? So a specific deep learning setup actually pulled ahead of a well-tuned random forest.
Speaker #1
In their tests, yes. It suggests maybe the deep networks were better at capturing some complex, nonlinear stuff in the data that even RF couldn't quite get.
Speaker #0
Fascinating. OK, prediction accuracy is one thing, but what about actual trading? Did they simulate a strategy?
Speaker #1
They did. A classic long-short portfolio.
Speaker #0
How did that work?
Speaker #1
Simple idea. Use the model's predictions to rank the stocks. Buy the top-ranked ones, sell the bottom-ranked ones. Equal weighting.
Speaker #0
OK. And they looked at turtle and quintile portfolios again?
Speaker #1
That's right. Splitting the stocks into three groups or five groups based on the predictions. One important caveat, though.
Speaker #0
Let me guess. Transaction costs.
Speaker #1
You got it. They didn't include transaction costs in this simulation, so real-world results would likely be lower.
Speaker #0
Understood. But still, what did the simulated performance show? The return versus risk?
Speaker #1
That's the key metric, right? The return-risk ratio. Looking at Table 8, the highest RR ratios, both for Turtle and Quintile portfolios, came from the DNN models.
Speaker #0
Ah, okay. Which ones?
Speaker #1
DNN83 had the best RR for the turtle portfolio at 1.24, and DNNN51 topped the quintile portfolio with 1.29.
Speaker #0
So the deep learning models deliver the best risk-adjusted returns in simulation.
Speaker #1
That's what their results suggest. And what's interesting is, remember, random forests sometimes had slightly higher directional accuracy. Yeah. Well, despite that, some DNNs had better risk-adjusted returns. It implies maybe the DNNs were better at predicting the magnitude of the returns for the winners and losers, not just the direction.
Speaker #0
That makes a difference for portfolio performance.
Speaker #1
Absolutely.
Speaker #0
Did they try combining models and an ensemble?
Speaker #1
Yes, briefly. They combined the best SVR, RF, and their top DNN, DNN83, with equal weights.
Speaker #0
Ensembles often boost performance, right?
Speaker #1
Often, yeah. The idea is different models capture different things. In this case, the ensemble did get the highest COR.0604, so slightly better rank prediction.
Speaker #0
But what about the trading simulation, the RR?
Speaker #1
That's the interesting part. According to Table 8, the improvement in directional accuracy and, crucially, the return risk ratio was pretty limited compared to the best individual DNNs.
Speaker #0
So the ensemble didn't quite deliver the knockout punch in terms of risk-adjusted returns here.
Speaker #1
Not in this specific setup, no. The best individual DNNs still held the top spots for RR.
Speaker #0
Okay, so let's try to summarize the takeaways on the trading rules and backtests. It seems like deeper networks generally did show an advantage here.
Speaker #1
Yeah, that seems to be the main thrust. Deeper networks generally perform better than shallower ones for predicting Japanese stock returns in this cross-sectional study.
Speaker #0
And they held their own or even slightly outperformed standard ML techniques like SVR and random forests.
Speaker #1
Right. Particularly when looking at the simulated long-short trading performance. Yeah. The DNN-based strategies yielded the highest risk-adjusted returns.
Speaker #0
Even though the ensemble improved correlations slightly. It didn't beat the best individual DNNs on that key RR metric.
Speaker #1
Correct. It suggests that maybe just averaging those specific models wasn't the optimal way to combine them. Or perhaps the individual DNNs were already capturing most of the predictable signal effectively.
Speaker #0
Now, the authors did mention limitations, right? Like focusing on standard feedforward networks.
Speaker #1
They did. They pointed out that things like recurrent neural networks, RNNs, might be interesting to explore because they're specifically designed for time series data.
Speaker #0
Which stock factors often are, even if used cross-sectionally here.
Speaker #1
Exactly. And they also suggested just exploring a wider range of deep learning architectures in general. There's a lot more out there now than there was even in 2018.
Speaker #0
So there's definitely room for more research, maybe exploring how these models handle different market regimes or incorporating transaction costs more explicitly.
Speaker #1
Absolutely. And I think a key question going forward is always, how much of this improved performance is genuinely capturing economic signal? versus potentially overfitting complex patterns that might not persist, it's the perennial challenge.
Speaker #0
A very important point. It leaves us wondering how robust these deep learning advantages truly are over the very long term.
Speaker #1
Definitely food for thought.
Speaker #0
Okay, well that wraps up our deep dive for today. Thank you for tuning in to Papers with Backtest podcast. We hope today's episode gave you useful insights. Join us next time as we break down more research. And for more papers and backtests, find us at https.paperswithbacktest.com. Happy trading.

Chapters

Introduction to the Podcast and Paper Overview
0sec
Understanding the Dataset and Factors Used
3sec
Exploring Neural Network Architectures and Backtest Results
52sec
Comparing DNNs with Traditional Machine Learning Methods
2min
Simulating Trading Strategies and Performance Analysis
4min
Key Takeaways and Future Research Directions
8min

Description

Tune in now and discover how deep learning is not just a buzzword but a game-changer in the world of stock market predictions!

Hosted on Ausha. See ausha.co/privacy-policy for more information.

Transcription

Speaker #0
Hello, welcome back to Papers with Backtest podcast. Today we dive into another algo trading research paper.
Speaker #1
We do indeed.
Speaker #0
We're going to unpack deep learning for forecasting stock returns in the cross-section by Abe and Nakayama from 2018.
Speaker #1
A really interesting one.
Speaker #0
Yeah, this paper looks into whether using deep learning can, you know, give us a better edge predicting Japanese stock performance over the next month.
Speaker #1
Exactly. It's fundamentally asking, can these newer, more complex machine learning tools actually beat the traditional methods, especially for forecasting returns.
Speaker #0
Right. So for you listening, the goal today is figuring out if this deep learning approach really is a better way to predict stock moves compared to older methods and, well, other ML techniques too.
Speaker #1
Yeah, we're going to really focus on what this means for building actual trading rules and strategies, and importantly, the backtest results the authors found.
Speaker #0
Okay, so first things first, what data were they using?
Speaker #1
They used a pretty solid data set. It was constituents of the MSCI Japan Index. And they looked at 25 different factors. You know, the usual suspects, book-to-market ratio, earnings-to-price, past returns, like the last month's return.
Speaker #0
Just standard factors mostly.
Speaker #1
Pretty standard, yeah. Yeah. Listed in their table one. And the data covered a decent timeframe, December 1990 through November 2016.
Speaker #0
That's quite a stretch. Now, I saw something about a lag, a four-month lag.
Speaker #1
Yeah. Yes. That's quite important. They built in a four-month delay.
Speaker #0
Why is that?
Speaker #1
Well, it's about realism. The idea is even if data is released, it takes time for investors to actually get it, digest it, and act on it. So this lag tries to mimic that real-world delay.
Speaker #0
Makes sense. So the core problem was predicting next month's return.
Speaker #1
Using those 25 factors, but not just the latest values, they looked at the factors from the last five time points.
Speaker #0
Five points. How far apart?
Speaker #1
Three-month intervals. So looking back over what, about 15 months of factor history?
Speaker #0
Okay, so capturing some trend information potentially, and they pre-processed the data.
Speaker #1
They did. They rescaled both the inputs, the factors, and the outputs, the returns, by ranking them. Yeah, ranking across all the stocks at each point in time. Kind of normalizes things, helps the models focus on relative performance rather than absolute values. Got it.
Speaker #0
Okay, let's get to the good stuff. The trading rules and the back tests. They looked at different neural network setups.
Speaker #1
They did. They compared deep neural networks, DNNs, with eight layers, DNN8. and five layers DNN5.
Speaker #0
Against shallower ones.
Speaker #1
Yeah, exactly. Against a three-layer network, NN3. And they tested that NN3 both with and without something called dropout.
Speaker #0
Ah, dropout. That's for preventing overfitting, right?
Speaker #1
Precisely. It randomly ignores some connections during training, makes the network a bit more robust, hopefully less likely to just memorize the training data.
Speaker #0
So what did they find when they compared these deep versus shallow networks? How did they perform?
Speaker #1
Well, looking at their tables four and five, the general trend was Deeper seemed better.
Speaker #0
More layers, better predictions.
Speaker #1
Generally, yes. Performance measured by rank correlation, they called it ERR and directional accuracy, tended to improve as the networks got deeper.
Speaker #0
And COERA is?
Speaker #1
Rank correlation. Basically, how well the model's predicted ranking of stocks matches the actual ranking based on future returns. Higher is better.
Speaker #0
And directional accuracy?
Speaker #1
Simply, the percentage of times the model correctly guessed if a stock would go up or down relative to the median. Okay. And interestingly, even shallow networks with a similar number of parameters didn't quite match up to the DNNs.
Speaker #0
So it wasn't just about the number of knobs to turn, but the structure?
Speaker #1
It seems that way. The depth itself appeared to add value. Right. And the directional accuracy was statistically significant for all the network types, which is encouraging.
Speaker #0
Suggests they were definitely picking up something. Right.
Speaker #1
And the mean squared error, the loss function, was generally lower for the deeper networks too.
Speaker #0
Which... Specific models did best. I see DNN83 mentioned for CR.
Speaker #1
Yeah, DNN83 hit the highest CR at 0.0591.
Speaker #0
Now, 0.0591, that sounds quite small.
Speaker #1
It does, but in this field, forecasting cross-sectional returns, even a small consistent edge like that can be meaningful. It's notoriously difficult.
Speaker #0
Okay, fair enough. And the directional accuracy for that model?
Speaker #1
For DNN83, it was about 52.6% for tertiles, splitting stocks into three groups. And a bit better, 53.4% roughly for quintiles, so five groups.
Speaker #0
Still only slightly better than a coin flip, but statistically significant, you said.
Speaker #1
Exactly. Better than chance, consistently.
Speaker #0
Okay, so DNNs show promise, but they compared them to other ML methods too.
Speaker #1
They did, yeah. Crucial comparison point. They used support vector regression, SVR, and random forests RF as benchmarks. Pretty standard powerful ML techniques.
Speaker #0
And did they tune those models properly?
Speaker #1
Seems so. They mentioned testing 24 different setups for SVR and 37 for random forests to try and find the best configuration for each.
Speaker #0
So what were the best results for SVR and RF?
Speaker #1
For SVR, the best TOR was around 0.0569 directional accuracy, similar to the DNN, so maybe slightly lower, around 52.5% and 53.3%.
Speaker #0
And random forests?
Speaker #1
RF did a bit better on TOR, 0.0576. Directional accuracy was quite close to the best DNNs, actually, around 52.6% and 53.4%.
Speaker #0
Okay, so RF looks pretty competitive there. How did the DNN stack up overall against these?
Speaker #1
This is where it gets interesting. Several of the top DNN patterns, like DNN 81, 83, 84, and 51, they generally outperformed SVR on most metrics. And against random forests, it was closer. But some DNNs showed slight advantages, especially in TRR. Specifically, DNN84 actually beat the best RF model on all the metrics in their comparison table, table six.
Speaker #0
Really? So a specific deep learning setup actually pulled ahead of a well-tuned random forest.
Speaker #1
In their tests, yes. It suggests maybe the deep networks were better at capturing some complex, nonlinear stuff in the data that even RF couldn't quite get.
Speaker #0
Fascinating. OK, prediction accuracy is one thing, but what about actual trading? Did they simulate a strategy?
Speaker #1
They did. A classic long-short portfolio.
Speaker #0
How did that work?
Speaker #1
Simple idea. Use the model's predictions to rank the stocks. Buy the top-ranked ones, sell the bottom-ranked ones. Equal weighting.
Speaker #0
OK. And they looked at turtle and quintile portfolios again?
Speaker #1
That's right. Splitting the stocks into three groups or five groups based on the predictions. One important caveat, though.
Speaker #0
Let me guess. Transaction costs.
Speaker #1
You got it. They didn't include transaction costs in this simulation, so real-world results would likely be lower.
Speaker #0
Understood. But still, what did the simulated performance show? The return versus risk?
Speaker #1
That's the key metric, right? The return-risk ratio. Looking at Table 8, the highest RR ratios, both for Turtle and Quintile portfolios, came from the DNN models.
Speaker #0
Ah, okay. Which ones?
Speaker #1
DNN83 had the best RR for the turtle portfolio at 1.24, and DNNN51 topped the quintile portfolio with 1.29.
Speaker #0
So the deep learning models deliver the best risk-adjusted returns in simulation.
Speaker #1
That's what their results suggest. And what's interesting is, remember, random forests sometimes had slightly higher directional accuracy. Yeah. Well, despite that, some DNNs had better risk-adjusted returns. It implies maybe the DNNs were better at predicting the magnitude of the returns for the winners and losers, not just the direction.
Speaker #0
That makes a difference for portfolio performance.
Speaker #1
Absolutely.
Speaker #0
Did they try combining models and an ensemble?
Speaker #1
Yes, briefly. They combined the best SVR, RF, and their top DNN, DNN83, with equal weights.
Speaker #0
Ensembles often boost performance, right?
Speaker #1
Often, yeah. The idea is different models capture different things. In this case, the ensemble did get the highest COR.0604, so slightly better rank prediction.
Speaker #0
But what about the trading simulation, the RR?
Speaker #1
That's the interesting part. According to Table 8, the improvement in directional accuracy and, crucially, the return risk ratio was pretty limited compared to the best individual DNNs.
Speaker #0
So the ensemble didn't quite deliver the knockout punch in terms of risk-adjusted returns here.
Speaker #1
Not in this specific setup, no. The best individual DNNs still held the top spots for RR.
Speaker #0
Okay, so let's try to summarize the takeaways on the trading rules and backtests. It seems like deeper networks generally did show an advantage here.
Speaker #1
Yeah, that seems to be the main thrust. Deeper networks generally perform better than shallower ones for predicting Japanese stock returns in this cross-sectional study.
Speaker #0
And they held their own or even slightly outperformed standard ML techniques like SVR and random forests.
Speaker #1
Right. Particularly when looking at the simulated long-short trading performance. Yeah. The DNN-based strategies yielded the highest risk-adjusted returns.
Speaker #0
Even though the ensemble improved correlations slightly. It didn't beat the best individual DNNs on that key RR metric.
Speaker #1
Correct. It suggests that maybe just averaging those specific models wasn't the optimal way to combine them. Or perhaps the individual DNNs were already capturing most of the predictable signal effectively.
Speaker #0
Now, the authors did mention limitations, right? Like focusing on standard feedforward networks.
Speaker #1
They did. They pointed out that things like recurrent neural networks, RNNs, might be interesting to explore because they're specifically designed for time series data.
Speaker #0
Which stock factors often are, even if used cross-sectionally here.
Speaker #1
Exactly. And they also suggested just exploring a wider range of deep learning architectures in general. There's a lot more out there now than there was even in 2018.
Speaker #0
So there's definitely room for more research, maybe exploring how these models handle different market regimes or incorporating transaction costs more explicitly.
Speaker #1
Absolutely. And I think a key question going forward is always, how much of this improved performance is genuinely capturing economic signal? versus potentially overfitting complex patterns that might not persist, it's the perennial challenge.
Speaker #0
A very important point. It leaves us wondering how robust these deep learning advantages truly are over the very long term.
Speaker #1
Definitely food for thought.
Speaker #0
Okay, well that wraps up our deep dive for today. Thank you for tuning in to Papers with Backtest podcast. We hope today's episode gave you useful insights. Join us next time as we break down more research. And for more papers and backtests, find us at https.paperswithbacktest.com. Happy trading.

Chapters

Introduction to the Podcast and Paper Overview
0sec
Understanding the Dataset and Factors Used
3sec
Exploring Neural Network Architectures and Backtest Results
52sec
Comparing DNNs with Traditional Machine Learning Methods
2min
Simulating Trading Strategies and Performance Analysis
4min
Key Takeaways and Future Research Directions
8min

Embed

Introduction to the Podcast and Paper Overview

0sec

Understanding the Dataset and Factors Used

3sec

Exploring Neural Network Architectures and Backtest Results

52sec

Comparing DNNs with Traditional Machine Learning Methods

2min

Simulating Trading Strategies and Performance Analysis

4min

Key Takeaways and Future Research Directions

8min

Transcription

Speaker #0
Hello, welcome back to Papers with Backtest podcast. Today we dive into another algo trading research paper.
Speaker #1
We do indeed.
Speaker #0
We're going to unpack deep learning for forecasting stock returns in the cross-section by Abe and Nakayama from 2018.
Speaker #1
A really interesting one.
Speaker #0
Yeah, this paper looks into whether using deep learning can, you know, give us a better edge predicting Japanese stock performance over the next month.
Speaker #1
Exactly. It's fundamentally asking, can these newer, more complex machine learning tools actually beat the traditional methods, especially for forecasting returns.
Speaker #0
Right. So for you listening, the goal today is figuring out if this deep learning approach really is a better way to predict stock moves compared to older methods and, well, other ML techniques too.
Speaker #1
Yeah, we're going to really focus on what this means for building actual trading rules and strategies, and importantly, the backtest results the authors found.
Speaker #0
Okay, so first things first, what data were they using?
Speaker #1
They used a pretty solid data set. It was constituents of the MSCI Japan Index. And they looked at 25 different factors. You know, the usual suspects, book-to-market ratio, earnings-to-price, past returns, like the last month's return.
Speaker #0
Just standard factors mostly.
Speaker #1
Pretty standard, yeah. Yeah. Listed in their table one. And the data covered a decent timeframe, December 1990 through November 2016.
Speaker #0
That's quite a stretch. Now, I saw something about a lag, a four-month lag.
Speaker #1
Yeah. Yes. That's quite important. They built in a four-month delay.
Speaker #0
Why is that?
Speaker #1
Well, it's about realism. The idea is even if data is released, it takes time for investors to actually get it, digest it, and act on it. So this lag tries to mimic that real-world delay.
Speaker #0
Makes sense. So the core problem was predicting next month's return.
Speaker #1
Using those 25 factors, but not just the latest values, they looked at the factors from the last five time points.
Speaker #0
Five points. How far apart?
Speaker #1
Three-month intervals. So looking back over what, about 15 months of factor history?
Speaker #0
Okay, so capturing some trend information potentially, and they pre-processed the data.
Speaker #1
They did. They rescaled both the inputs, the factors, and the outputs, the returns, by ranking them. Yeah, ranking across all the stocks at each point in time. Kind of normalizes things, helps the models focus on relative performance rather than absolute values. Got it.
Speaker #0
Okay, let's get to the good stuff. The trading rules and the back tests. They looked at different neural network setups.
Speaker #1
They did. They compared deep neural networks, DNNs, with eight layers, DNN8. and five layers DNN5.
Speaker #0
Against shallower ones.
Speaker #1
Yeah, exactly. Against a three-layer network, NN3. And they tested that NN3 both with and without something called dropout.
Speaker #0
Ah, dropout. That's for preventing overfitting, right?
Speaker #1
Precisely. It randomly ignores some connections during training, makes the network a bit more robust, hopefully less likely to just memorize the training data.
Speaker #0
So what did they find when they compared these deep versus shallow networks? How did they perform?
Speaker #1
Well, looking at their tables four and five, the general trend was Deeper seemed better.
Speaker #0
More layers, better predictions.
Speaker #1
Generally, yes. Performance measured by rank correlation, they called it ERR and directional accuracy, tended to improve as the networks got deeper.
Speaker #0
And COERA is?
Speaker #1
Rank correlation. Basically, how well the model's predicted ranking of stocks matches the actual ranking based on future returns. Higher is better.
Speaker #0
And directional accuracy?
Speaker #1
Simply, the percentage of times the model correctly guessed if a stock would go up or down relative to the median. Okay. And interestingly, even shallow networks with a similar number of parameters didn't quite match up to the DNNs.
Speaker #0
So it wasn't just about the number of knobs to turn, but the structure?
Speaker #1
It seems that way. The depth itself appeared to add value. Right. And the directional accuracy was statistically significant for all the network types, which is encouraging.
Speaker #0
Suggests they were definitely picking up something. Right.
Speaker #1
And the mean squared error, the loss function, was generally lower for the deeper networks too.
Speaker #0
Which... Specific models did best. I see DNN83 mentioned for CR.
Speaker #1
Yeah, DNN83 hit the highest CR at 0.0591.
Speaker #0
Now, 0.0591, that sounds quite small.
Speaker #1
It does, but in this field, forecasting cross-sectional returns, even a small consistent edge like that can be meaningful. It's notoriously difficult.
Speaker #0
Okay, fair enough. And the directional accuracy for that model?
Speaker #1
For DNN83, it was about 52.6% for tertiles, splitting stocks into three groups. And a bit better, 53.4% roughly for quintiles, so five groups.
Speaker #0
Still only slightly better than a coin flip, but statistically significant, you said.
Speaker #1
Exactly. Better than chance, consistently.
Speaker #0
Okay, so DNNs show promise, but they compared them to other ML methods too.
Speaker #1
They did, yeah. Crucial comparison point. They used support vector regression, SVR, and random forests RF as benchmarks. Pretty standard powerful ML techniques.
Speaker #0
And did they tune those models properly?
Speaker #1
Seems so. They mentioned testing 24 different setups for SVR and 37 for random forests to try and find the best configuration for each.
Speaker #0
So what were the best results for SVR and RF?
Speaker #1
For SVR, the best TOR was around 0.0569 directional accuracy, similar to the DNN, so maybe slightly lower, around 52.5% and 53.3%.
Speaker #0
And random forests?
Speaker #1
RF did a bit better on TOR, 0.0576. Directional accuracy was quite close to the best DNNs, actually, around 52.6% and 53.4%.
Speaker #0
Okay, so RF looks pretty competitive there. How did the DNN stack up overall against these?
Speaker #1
This is where it gets interesting. Several of the top DNN patterns, like DNN 81, 83, 84, and 51, they generally outperformed SVR on most metrics. And against random forests, it was closer. But some DNNs showed slight advantages, especially in TRR. Specifically, DNN84 actually beat the best RF model on all the metrics in their comparison table, table six.
Speaker #0
Really? So a specific deep learning setup actually pulled ahead of a well-tuned random forest.
Speaker #1
In their tests, yes. It suggests maybe the deep networks were better at capturing some complex, nonlinear stuff in the data that even RF couldn't quite get.
Speaker #0
Fascinating. OK, prediction accuracy is one thing, but what about actual trading? Did they simulate a strategy?
Speaker #1
They did. A classic long-short portfolio.
Speaker #0
How did that work?
Speaker #1
Simple idea. Use the model's predictions to rank the stocks. Buy the top-ranked ones, sell the bottom-ranked ones. Equal weighting.
Speaker #0
OK. And they looked at turtle and quintile portfolios again?
Speaker #1
That's right. Splitting the stocks into three groups or five groups based on the predictions. One important caveat, though.
Speaker #0
Let me guess. Transaction costs.
Speaker #1
You got it. They didn't include transaction costs in this simulation, so real-world results would likely be lower.
Speaker #0
Understood. But still, what did the simulated performance show? The return versus risk?
Speaker #1
That's the key metric, right? The return-risk ratio. Looking at Table 8, the highest RR ratios, both for Turtle and Quintile portfolios, came from the DNN models.
Speaker #0
Ah, okay. Which ones?
Speaker #1
DNN83 had the best RR for the turtle portfolio at 1.24, and DNNN51 topped the quintile portfolio with 1.29.
Speaker #0
So the deep learning models deliver the best risk-adjusted returns in simulation.
Speaker #1
That's what their results suggest. And what's interesting is, remember, random forests sometimes had slightly higher directional accuracy. Yeah. Well, despite that, some DNNs had better risk-adjusted returns. It implies maybe the DNNs were better at predicting the magnitude of the returns for the winners and losers, not just the direction.
Speaker #0
That makes a difference for portfolio performance.
Speaker #1
Absolutely.
Speaker #0
Did they try combining models and an ensemble?
Speaker #1
Yes, briefly. They combined the best SVR, RF, and their top DNN, DNN83, with equal weights.
Speaker #0
Ensembles often boost performance, right?
Speaker #1
Often, yeah. The idea is different models capture different things. In this case, the ensemble did get the highest COR.0604, so slightly better rank prediction.
Speaker #0
But what about the trading simulation, the RR?
Speaker #1
That's the interesting part. According to Table 8, the improvement in directional accuracy and, crucially, the return risk ratio was pretty limited compared to the best individual DNNs.
Speaker #0
So the ensemble didn't quite deliver the knockout punch in terms of risk-adjusted returns here.
Speaker #1
Not in this specific setup, no. The best individual DNNs still held the top spots for RR.
Speaker #0
Okay, so let's try to summarize the takeaways on the trading rules and backtests. It seems like deeper networks generally did show an advantage here.
Speaker #1
Yeah, that seems to be the main thrust. Deeper networks generally perform better than shallower ones for predicting Japanese stock returns in this cross-sectional study.
Speaker #0
And they held their own or even slightly outperformed standard ML techniques like SVR and random forests.
Speaker #1
Right. Particularly when looking at the simulated long-short trading performance. Yeah. The DNN-based strategies yielded the highest risk-adjusted returns.
Speaker #0
Even though the ensemble improved correlations slightly. It didn't beat the best individual DNNs on that key RR metric.
Speaker #1
Correct. It suggests that maybe just averaging those specific models wasn't the optimal way to combine them. Or perhaps the individual DNNs were already capturing most of the predictable signal effectively.
Speaker #0
Now, the authors did mention limitations, right? Like focusing on standard feedforward networks.
Speaker #1
They did. They pointed out that things like recurrent neural networks, RNNs, might be interesting to explore because they're specifically designed for time series data.
Speaker #0
Which stock factors often are, even if used cross-sectionally here.
Speaker #1
Exactly. And they also suggested just exploring a wider range of deep learning architectures in general. There's a lot more out there now than there was even in 2018.
Speaker #0
So there's definitely room for more research, maybe exploring how these models handle different market regimes or incorporating transaction costs more explicitly.
Speaker #1
Absolutely. And I think a key question going forward is always, how much of this improved performance is genuinely capturing economic signal? versus potentially overfitting complex patterns that might not persist, it's the perennial challenge.
Speaker #0
A very important point. It leaves us wondering how robust these deep learning advantages truly are over the very long term.
Speaker #1
Definitely food for thought.
Speaker #0
Okay, well that wraps up our deep dive for today. Thank you for tuning in to Papers with Backtest podcast. We hope today's episode gave you useful insights. Join us next time as we break down more research. And for more papers and backtests, find us at https.paperswithbacktest.com. Happy trading.

Chapters

Introduction to the Podcast and Paper Overview

0sec

Understanding the Dataset and Factors Used

3sec

Exploring Neural Network Architectures and Backtest Results

52sec

Comparing DNNs with Traditional Machine Learning Methods

2min

Simulating Trading Strategies and Performance Analysis

4min

Key Takeaways and Future Research Directions

8min