The uncertainty in future market dynamics is an important consideration when developing strategies for hedging derivatives, particularly data driven strategies such as deep hedging. Deep market generators can produce higher fidelity training data than classical models, but, like those, typically require frequent recalibration to new market data. The resulting strategies are thus susceptible to underperformance if there is a mismatch (distributional shift) between training data and live data. We present a framework to train a modified deep hedger which displays a form of ambiguity aversion, henceforth termed an Ambiguity-Averse Deep Hedger (AADH). The modeller has full control over exactly which aspects of distributional shifts the AADH is to be robust to, through selection of features relevant to the trading strategy which are used to cluster the training data, allowing for the evaluation of a loss function motivated by the theory of smooth ambiguity aversion. We demonstrate that for forward start options with transaction costs, our AADH with feature clustering always outperforms the standard deep hedger in out-of-distribution tests, emphasising the need to prepare for future uncertainty, and the effectiveness of ambiguity aversion in achieving this. For plain vanilla options with transaction costs, our AADH also outperforms the standard deep hedger in the overwhelming majority of out-of-distribution scenarios. Our findings are supported by numerical experiments involving (i) synthetic data with standard stochastic models, as well as (ii) real data experiments on equity indices over extensive periods of time, as well as (iii) a period of significant distributional shifts, the COVID period.