Exercise: Why ES can't be backtested with pinball loss

A scoring function

S(\hat{s}, x)

takes a forecast

\hat{s}

and an observation

x

and outputs a loss.

S

is strictly consistent for a statistic

s(\cdot)

of a distribution

F

\mathbb{E}_{X \sim F}[S(\hat{s}, X)] > \mathbb{E}_{X \sim F}[S(s(F), X)]

for all

\hat{s} \ne s(F)

. A statistic is elicitable if a strictly consistent scoring function exists.

Fact. The

\alpha

-quantile (so VaR) is elicitable; the pinball loss

S(\hat{q}, x) = (\alpha - \mathbf{1}\{x < \hat{q}\})(\hat{q} - x)

is strictly consistent.

Counter-fact. Expected Shortfall is not elicitable: no strictly consistent scoring function for ES alone exists (Gneiting 2011).

Tasks

Verify that the pinball loss is strictly consistent for the $\alpha$ -quantile. Compute $\partial \mathbb{E}[S(\hat{q}, X)]/\partial \hat{q}$ and show it vanishes iff $\mathbb{P}(X < \hat{q}) = \alpha$ .
Gneiting's argument for ES non-elicitability rests on the fact that elicitable statistics have convex level sets: $\{F : s(F) = c\}$ must be convex in the space of distributions. Sketch why the $\alpha$ -quantile has convex level sets but $\text{ES}_\alpha$ does not. Give a two-point-mixture example: distributions $F_1, F_2$ with $\text{ES}_\alpha(F_1) = \text{ES}_\alpha(F_2)$ but $\text{ES}_\alpha(\tfrac12 F_1 + \tfrac12 F_2) \ne$ that common value.
Fissler & Ziegel (2016) proved that the pair (VaR, ES) is jointly elicitable. What is the practical consequence for backtesting ES?
Why does the Acerbi-Székely $Z_2$ test sidestep the non-elicitability obstacle? It's not a scoring-function test — what kind of test is it?

Hint

For (2), mixtures of distributions average their densities but non-linearly average their quantiles, and ES is a conditional expectation which doesn't average linearly under mixture.