RECORD_ID: prj_06 STATUS: COMPLETED
[SUBJECT_MATTER]
reinforcement learning — gridworld navigation
Implementation of Q-Learning and SARSA algorithms for autonomous agent navigation in a configurable GridWorld environment, with comparative analysis of learning dynamics.
TIMEFRAME 2024
TECHNICAL_STACK
Python NumPy Matplotlib Reinforcement Learning Q-Learning SARSA AI
system_log // rl_gridworld
This project explores classical Reinforcement Learning techniques through autonomous agent navigation in a customizable GridWorld environment. Two temporal-difference algorithms are implemented and compared in depth.
algorithms
q_learning
- Off-policy TD control — learns the optimal action-value function Q*(s,a) regardless of behavior policy
- Epsilon-greedy exploration with configurable decay
- Update rule:
Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') - Q(s,a)]
sarsa
- On-policy TD control — learns the value of the current policy
- More conservative exploration, less prone to risky shortcuts near negative rewards
- Update rule:
Q(s,a) ← Q(s,a) + α[r + γ·Q(s',a') - Q(s,a)]
environment
- Configurable N×N grid with static obstacles
- Positive rewards at goal states, negative at traps
- Action space:
{Up, Down, Left, Right} - Compatible with OpenAI Gym interface
comparative_findings
| Metric | Q-Learning | SARSA |
|---|---|---|
| Convergence speed | Faster | Slower but smoother |
| Training stability | More variance | More stable |
| Risk behavior | Optimistic (learns optimal path) | Conservative (avoids risky shortcuts) |
| Sample efficiency | Better for simple grids | Better for complex obstacle layouts |
Both algorithms converge to working policies with proper hyperparameter tuning. Q-Learning finds optimal paths faster; SARSA produces more robust policies in environments with negative rewards near the optimal path.
learning_outcomes
- Practical understanding of on-policy vs off-policy learning
- Exploration-exploitation trade-off management through epsilon scheduling
- Hyperparameter sensitivity analysis (learning rate α, discount factor γ)
- Policy visualization with Matplotlib heatmaps
Repository: [Coming soon]