From action selection to sequential credit assignment
How to study this lesson
Read one section, trace the worked example, and try the knowledge check. Bookmark saves a shortcut; notes appear in My notebook. Mark lesson complete is your own assessment and does not mark its lab passed.
By the end, you can
- Distinguish contextual bandits from sequential reinforcement learning.
- Calculate an importance-weighted policy estimate.
- Recognize support, variance, and confounding limits in logged feedback.
Choose the smallest learning problem that fits
A contextual bandit observes a context, selects an action, and receives feedback for that action. It does not model how the choice changes a sequence of future states. This can fit a router selecting one of several answer-generation configurations for a request when the outcome is scored after that request. Sequential reinforcement learning becomes relevant when actions alter what happens next, such as choosing which diagnostic test to run before deciding whether to escalate.
Write down the state, available actions, reward, and episode boundary before selecting an algorithm. A user click may be a convenient signal but a poor objective if the actual goal is a resolved issue. Also ask whether the context contains information available before the decision. Features computed after an outcome leak the answer into learning and can make an offline policy look deceptively capable.
Exploration creates both information and cost
A greedy router always selects the action with the highest current estimate. If an untried action starts with a low estimate, it may never be chosen, so the system never learns whether it is better. An epsilon-greedy policy occasionally explores. For two actions and epsilon equal to 0.2, a unique greedy action receives probability 0.9 and the other receives 0.1 when exploration is uniform over both actions.
Exploration should occur only within the set of actions already permitted for that context. A permissions boundary is not something a reward optimizer may cross to gather data. Log the exact probability assigned to the selected action at decision time. Reconstructing it later from a changed policy is unreliable. Contextual methods, including the linked news-recommendation research, adapt decisions to observed features; an average reward per action alone cannot represent a case where different contexts need different actions.
Counterfactual feedback is missing
A log tells you what happened under the chosen action, not what would have happened under every alternative. Comparing raw average outcomes for two actions can be misleading if one was assigned easier cases. Inverse propensity scoring addresses the selection mechanism when its assumptions hold: multiply the observed reward by the target policy probability divided by the logged action probability, then average across logged events.
Imagine two logged events. The first has reward one, logging probability 0.5, and target probability one. The second has reward zero, logging probability 0.5, and target probability zero. Contributions are two and zero, so the estimated target value is one. The zero-weight event still belongs in the ordinary estimator's denominator. Dropping it changes the estimator. This arithmetic is a toy demonstration, not evidence that a real routing policy achieves perfect success.
Support and variance determine credibility
The logging policy must assign positive probability wherever the target policy might act. If a target action was never possible under logging, its value is not identifiable from those logs without additional modeling assumptions. Very small logging probabilities produce large weights. A single reward-one event with propensity 0.01 and target probability one contributes 100 before averaging; a small dataset can therefore yield an estimate outside the reward's original range.
Weight clipping reduces variance by capping influence, but introduces bias. Self-normalization divides weighted rewards by the sum of weights rather than the number of events; it is also generally biased in finite samples. Report the estimator, clipping rule, sample size, and weight distribution. Effective sample size, computed as squared total weight divided by total squared weight, is a useful diagnostic of concentrated weights, though it does not fix missing support or hidden confounding.
Sequential tasks need a longer accounting horizon
For an episode, return can include discounted future rewards: G_t = r_t + gamma r_(t+1) + gamma squared r_(t+2), and so on. With rewards -1, -1, and 10 and gamma 0.9, the initial return is -1 - 0.9 + 8.1 = 6.2. Immediate-reward optimization would dislike the diagnostic steps even though they enable the final result. The state representation must preserve information relevant to such consequences.
Agent trajectories introduce delayed outcomes, interrupted tasks, tool failures, and expensive exploration. Start by instrumenting a fixed policy and defining episode-level success before training an adaptive one. Compare against a strong deterministic router when possible. A bandit may be sufficient for choosing among safe existing workflows; a full sequential method brings additional data and stability requirements. Use the problem's feedback structure to justify the complexity, then evaluate it under the distribution where it will operate.
Work through the code
Each tuple is observed reward, logging probability, and target probability for the logged action. This fixture illustrates estimator disagreement and concentrated weights. It omits confidence intervals and does not claim that either estimate is the true value of a deployed policy.
import math
def summarize(events):
weights = [target / logged for _, logged, target in events]
weighted = [reward * weight
for (reward, _, _), weight in zip(events, weights)]
total_weight = sum(weights)
ips = sum(weighted) / len(events)
snips = sum(weighted) / total_weight if total_weight else None
squared = sum(weight * weight for weight in weights)
ess = total_weight ** 2 / squared if squared else 0.0
return ips, snips, ess
events = [(1.0, 0.5, 1.0), (0.0, 0.5, 0.0),
(1.0, 0.25, 0.5), (0.0, 0.75, 0.5)]
assert all(0 < logged <= 1 and 0 <= target <= 1
for _, logged, target in events)
ips, snips, ess = summarize(events)
print(f"IPS: {ips:.3f}")
print(f"self-normalized: {snips:.3f}")
print(f"effective sample size: {ess:.3f}")
IPS: 1.000 self-normalized: 0.857 effective sample size: 2.579
Run Python snippets locally with the prerequisites named above. The module coding lab runs directly in your browser.
Pause and reason
A log contains 1,000 events, but the proposed target action had zero logging probability for all high-risk contexts. Can importance weighting evaluate that policy on high-risk cases? Propose a defensible next step.
Check your understanding
A router performs best on average, but each action was assigned a different difficulty mix. Which comparison is most defensible?
Your notes
Explain the mechanism in your own words. Add a failure you want to test.