rl_directional_straddle: Teaching an AI Trader to Obey the Market

The Market Says No First

Most RL trading projects start with the model. This one starts with the market’s refusal. The agent is not allowed to wander through a fantasy exchange where leverage is free and every trade is legal. It has to operate inside margin rules, position limits, and option-specific constraints that make the environment feel closer to a desk than a toy simulator.

That matters because options are not just another asset class. A directional call on NIFTY50 can become a position in volatility, theta decay, and margin exposure in the same minute. If the environment ignores those frictions, the policy learns a story that cannot survive contact with the tape.

Why Options RL Is Harder Than Stock RL

A stock trader can often get away with a simpler loop: observe price, choose buy or sell, learn from PnL. A straddle or strangle is more demanding. The agent has to manage multi-leg exposure, the effect of implied volatility, and the capital cost of holding or adding to a position. One bad assumption about margin can turn a clever policy into a backtest artifact.

That is why the repo leans on a Gymnasium-style environment rather than a standard supervised pipeline. The environment is the market model. It defines what can be seen, what can be done, and what each choice costs.

Inside the Environment: 69 Features, 8 Actions, 1 Capital Constraint

The core environment lives in envs/intraday_option_env_maskPPO_v3.py. It ingests 1-minute OHLC data plus India VIX, then expands that stream into a 69-feature observation vector through the feature calculator. That makes the state much richer than a simple price series. It includes volatility estimates and option-specific signals that matter when the asset itself is a derivative of another asset.

The action space is discrete and compact: hold, enter, exit, add, and reduce across call and put legs. That sounds small, but in options it is enough to express a useful trading grammar. The environment also models SPAN-like margin behavior, so a short position is not just a line item. It is a live constraint that affects whether the next action is even possible.

# Conceptual shape of the environment
observation = 69  # features from price, volatility, Greeks, and market context
action_space = 8   # hold, enter, exit, add, reduce across CE / PE legs

# The environment checks capital and position state before execution
if not enough_margin or position_limit_hit:
    action = mask_invalid_action(action)

reward = blended_reward(pnl_1m, pnl_30m, pnl_120m)

This is the important shift. The repo is not asking an agent to predict a label. It is asking a policy to navigate a simulated trading desk where every action has a balance sheet consequence.

The Trick: The Policy Never Sees Invalid Moves

That difference sounds technical, but it changes the learning dynamics. If the model can never choose an invalid move, it stops wasting updates on nonsense. The agent spends its capacity learning when to wait, when to enter, and when not to scale a position further.

Reward Shaping Is How the Agent Learns Patience

The repo uses blended reward horizons. Instead of only scoring the next minute, it mixes 1-step, 30-step, and 120-step outcomes. That creates a stronger signal for timing. A policy that grabs tiny gains every minute can look busy while doing poorly. A policy that never moves can look safe while missing the entire opportunity set.

Multi-horizon reward shaping fights both failure modes. The short horizon catches immediate execution quality. The medium and longer horizons push the policy to care about follow-through, drawdown, and whether a position survives the next stretch of noise. In practice, that is how you teach a trader to hold through turbulence without turning into a perpetual holder.

This is one of the more subtle parts of the design. The system is not rewarding bravery. It is rewarding disciplined persistence under uncertainty.

Why the Dashboard Matters

The explainability layer is not decoration. In a domain with Greeks, implied volatility, and hidden margin interactions, a policy that only prints PnL is not enough. The repo’s dashboard and SHAP analysis try to answer a narrower, better question: which features were actually driving the decision at this moment?

That matters especially in out-of-sample testing. A strategy can appear sharp in a backtest because it learned a quirk in one market regime. Feature-level attribution and walk-forward reporting are the first steps toward noticing when the model is seeing risk, not just noise.

What This Project Is Really Building Toward

This is a research-grade architecture, not a polished trading product. But it is pointed in an interesting direction. The repo suggests a template for constraint-aware financial RL systems: encode legality in the action space, encode patience in the reward, and encode trust through explainability.

Compared with broad frameworks like FinRL or TradeMaster, this project is narrower and more opinionated. That is a feature. It is not trying to cover every asset class or every agent. It is trying to make one difficult market problem honest enough that an RL policy can actually learn from it.