Compute an exact paired binary comparison

Advanced65 min

Implement paired_binary(old, new). Return n, wins, losses, ties, delta, and p_value for aligned binary task outcomes. Wins means new succeeds while old fails.

Your task

Materialize both input iterables. Require the same nonzero length and values whose exact type is bool or int with value zero or one. Raise ValueError otherwise.
Count wins, losses, and ties using paired task positions, and calculate delta = (wins - losses) / n.
When there are no discordant pairs, return p_value 1.0.
Otherwise calculate the exact two-sided p-value for a fair-binomial null: min(1, 2 * sum(comb(d, k), k from 0 through min(wins, losses)) / 2**d), where d is wins + losses.
Use the standard library only. Return the statistics without labeling a model the winner or making an automatic deployment decision.
Document that independent task pairs are assumed and that repeated runs of one task do not automatically satisfy that assumption.

Examples

EXAMPLE 1

Inputpaired_binary([0,0,0,0,0], [1,1,1,1,1])

Output{"n":5,"wins":5,"losses":0,"ties":0,"delta":1.0,"p_value":0.0625}

A large observed effect from five cases still has substantial sampling uncertainty.

def paired_binary(old, new):
    raise NotImplementedError