When machine learning models are deployed into the world, they inevitably encounter scenarios that differ from their training data, either novel contexts where the appropriate answers may differ or scenarios with new out-of-distribution inputs. Unfortunately, in such situations, deep neural network models make up answers or misunderstand the context, making the models unreliable. Even if a model makes useful predictions for many examples, such unreliability poses considerable risks when these models are interacting with real people and ultimately precludes models from being useful in safety-critical applications. In this talk, I’ll discuss some ways that we might cope with and address the unreliability of neural network models. As an initial coping strategy, I will first discuss a technique for detecting whether some content was generated by a machine learning model, leveraging the probability distribution that the model assigns to different content. Next, I will describe an approach for enabling neural network models to better estimate what they don’t know, such that they can abstain from making predictions on such inputs (i.e. selective classification). I will lastly describe methods for adapting models with small amounts of data to improve their accuracy under distribution shift.