I'll help you with your thesis on predicting PSEi prices using news sentiment with a biLSTM model. To provide more targeted advice, I'd like to clarify a few details:

1. What specific news sentiment data are you using (e.g., financial news headlines, social media, specific news sources)? This would help determine appropriate lag structures that match how quickly the PSEi market typically reacts to different types of news.

2. Have you already conducted any preliminary analysis on the relationship between your news sentiment data and PSEi price movements? If so, what patterns have you observed?

3. Are you focusing on predicting specific aspects of PSEi price movements (e.g., directional movement, volatility, exact price points) or a particular investment timeframe (short-term trading vs. longer-term investing)?

Consultancy Note: PSEi Price Prediction Using News Sentiment with BiLSTM Model

Task Overview

Provide advice on:

1. Suitable lag values for PSEi price prediction using news sentiment with a BiLSTM model
2. Methods for lag validation
3. Temporal analysis approaches for the retail-driven PSEi market

Detailed Recommendations

Suitable Lag Values

For Daily Data:

• Consider lag values of 1, 2, 3, and 5 days to capture:
  • Immediate market reactions (1-day lag)
  • Delayed information processing (2-3 day lags)
  • Weekly patterns and extended effects (5-day lag)
• Recommend testing multiple lag combinations as feature inputs to the BiLSTM

For Intraday Data:

• Test lag values of 1, 3, 6, and 12 hours to capture:
  • High-frequency trading patterns
  • Intraday sentiment shifts
  • Market microstructure effects
• Consider market trading hours (9:30 AM - 3:30 PM for PSEi) when designing intraday lags

Lag Validation Methods

Statistical Approaches:

• Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) analysis
• Granger Causality tests between news sentiment and price movements
• Information criteria (AIC, BIC) for model comparison with different lag structures
• Cross-correlation analysis between sentiment signals and price movements

Machine Learning Validation:

• Feature importance analysis to identify most predictive lag periods
• Cross-validation with time-series splits to prevent data leakage
• Ablation studies removing specific lags to measure their contribution

Temporal Methods for PSEi's Retail-Driven Market

Retail Investor Behavior Analysis:

• Incorporate sentiment persistence metrics (how long sentiment effects last)
• Consider day-of-week effects (retail investors more active on certain days)
• Account for retail trading hours vs. institutional trading patterns

PSEi-Specific Considerations:

• Analyze market liquidity patterns to identify periods when sentiment has stronger price impact
• Consider local market holidays and events that affect retail participation
• Incorporate trading volume as a weighting factor for sentiment signals

Advanced Temporal Techniques:

• Time-varying parameter models to capture changing market conditions
• Attention mechanisms within BiLSTM to focus on most relevant time periods
• Wavelet analysis to decompose time-series into different frequency components

Implementation Recommendations:

• Start with simpler models using 1-3 day lags before adding complexity
• Benchmark against simpler models (e.g., ARIMA with sentiment as exogenous variable)
• Consider ensemble approaches combining predictions from different lag structures

Consultancy Note: Optimal Lag Selection for PSEi Price Prediction Using News Sentiment with BiLSTM Model

1. Executive Summary

This consultancy note provides detailed recommendations on selecting optimal lag values for your thesis on predicting PSEi (Philippine Stock Exchange Index) prices using news sentiment with a BiLSTM model. The recommendations are tailored specifically for the retail-driven characteristics of the Philippine stock market, covering both daily and intraday prediction scenarios, along with validation methods and temporal analysis techniques.

2. Recommended Lag Values

2.1 Daily Data Lag Recommendations

• Short-term Lags (1-2 days):

  • Implement a 1-day lag as your baseline model, as news sentiment often has the strongest immediate effect on the following trading day researchgate.net
  • Use a 2-day lag to capture delayed information processing by retail investors, who typically take longer to react to news compared to institutional investors

• Medium-term Lags (3-5 days):

  • Include a 3-day lag to capture weekend effects (news released on Friday affecting Monday trading)
  • Research shows that a 4-day lag often reveals significant negative impact of news emotion on stock returns
  • Implement a 5-day lag (one trading week) to capture weekly patterns and extended sentiment effects, which is particularly important in retail-dominated markets

• Multiple Lag Combinations:

  • Consider using a combination of lags (e.g., 1, 3, and 5 days) rather than a single lag value
  • BiLSTM's bidirectional nature makes it well-suited to process multiple lag inputs simultaneously researchgate.net

2.2 Intraday Data Lag Recommendations

• Ultra-short-term Lags (1-3 hours):

  • Use a 1-hour lag for capturing immediate market reactions to breaking news
  • Implement a 3-hour lag to account for information dissemination and processing time among retail investors

• Medium-term Intraday Lags (6-12 hours):

  • A 6-hour lag can capture half-day trading patterns
  • Research on similar markets found that sentiment classifiers performed best with alignment lags of 5-6 hours sciencedirect.com
  • The 12-hour lag is valuable for overnight news effects on morning trading

• Market-specific Considerations:

  • Align lag values with PSEi trading hours (9:30 AM - 3:30 PM)
  • Consider non-trading hours when news sentiment accumulates but cannot immediately affect prices

3. Lag Validation Methods

3.1 Statistical Validation Techniques

• Autocorrelation Analysis:

  • Apply Autocorrelation Function (ACF) to identify significant lags in the PSEi price series
  • Use Partial Autocorrelation Function (PACF) to identify direct relationships between observations at different lags
  • Analyze cross-correlation between news sentiment and PSEi returns to identify optimal lag relationships frontiersin.org

• Granger Causality Testing:

  • Implement Granger causality tests to determine if lagged news sentiment values have statistically significant predictive power for PSEi prices
  • Test multiple lag specifications (from 1 to 10 days) to identify the optimal lag structure

• Information Criteria:

  • Apply Akaike Information Criterion (AIC) and Schwarz Information Criterion (SIC) to select optimal lag structure
  • Consider using modified information criteria that account for both lag order and rank structure in the model
  • For shorter lag models, AIC may be more appropriate, while for longer lag structures, SIC often performs better arxiv.org

3.2 Machine Learning Validation Approaches

• Cross-validation with Time Series Split:

  • Implement time series cross-validation to prevent data leakage
  • Use expanding window validation to simulate real-world forecasting scenarios

• Feature Importance Analysis:

  • Analyze the contribution of different lagged features to model performance
  • Consider genetic algorithm approaches to optimize lag selection for BiLSTM models mdpi.com

• Ablation Studies:

  • Systematically remove different lag values to measure their contribution to prediction accuracy
  • Compare performance metrics (RMSE, MAE, MAPE) across different lag configurations

4. PSEi Retail-Driven Market Considerations

4.1 Retail Investor Behavior Analysis

• Information Processing Patterns:

  • Retail investors typically take longer to process and act on news compared to institutional investors
  • Consider implementing longer lags (2-5 days) to account for this delayed reaction pattern
  • Analyze social media sentiment alongside traditional news sources to capture retail investor sentiment more comprehensively arxiv.org

• Day-of-Week Effects:

  • Incorporate day-of-week features in your model to account for varying trading patterns
  • Monday and Friday often show distinct patterns in retail-dominated markets
  • Consider separate models or features for pre-holiday and post-holiday trading sessions

• Market Microstructure:

  • Account for PSEi's trading hours and non-trading periods when designing lag structures
  • Consider the impact of overnight news on opening prices versus intraday news on closing prices

4.2 Advanced Temporal Techniques for PSEi

• Attention Mechanisms:

  • Implement attention mechanisms within your BiLSTM architecture to dynamically weight the importance of different time lags
  • This allows the model to focus on the most relevant historical periods for each prediction

• Time-varying Parameter Models:

  • Consider models that allow the relationship between news sentiment and prices to change over time
  • Particularly useful for capturing evolving market conditions in the PSEi

• Sentiment Persistence Modeling:

  • Develop metrics to quantify how long sentiment effects persist in the PSEi
  • Use exponentially weighted moving averages of sentiment to model decay effects

• Hybrid Models:

  • Combine BiLSTM with other architectures like 1D-CNN for feature extraction
  • Consider transformer-based models for capturing complex temporal dependencies sciencedirect.com

4.3 Implementation Recommendations

• Phased Approach:

  • Start with simpler models using 1-3 day lags before adding complexity
  • Benchmark against baseline models (e.g., ARIMA with sentiment as exogenous variable)
  • Gradually incorporate more sophisticated lag structures based on validation results

• Data Preprocessing:

  • Ensure proper alignment of news sentiment data with price data, accounting for time zones and market hours
  • Normalize both sentiment and price data to improve model convergence
  • Consider using wavelet decomposition to handle noise in the time series

• Evaluation Framework:

  • Develop a comprehensive evaluation framework that goes beyond accuracy metrics
  • Include directional accuracy and trading simulation to assess practical utility
  • Compare performance across different market conditions (bull/bear markets, high/low volatility periods)

5. Conclusion and Next Steps

The optimal lag structure for your PSEi price prediction model using news sentiment and BiLSTM will likely involve a combination of multiple lags rather than a single value. For daily data, consider implementing 1, 3, and 5-day lags as your starting point, while for intraday data, 1, 3, and 6-hour lags may be most effective.

Validation through both statistical methods (ACF/PACF, Granger causality) and machine learning approaches (cross-validation, feature importance) will help refine your lag selection. Remember to account for the unique characteristics of the PSEi's retail-driven market, particularly the delayed information processing patterns and day-of-week effects.

Your BiLSTM model architecture should be designed to effectively leverage these multiple lag inputs, potentially incorporating attention mechanisms to dynamically weight different historical periods. A phased implementation approach, starting with simpler models before adding complexity, will help establish robust baselines and demonstrate the value added by more sophisticated techniques.