In this work we study the inverse problem related to the emission of Hawking radiation. We first show how the knowledge of greybody factors of different angular contributions / can be used to constrain the width of the corresponding black hole perturbation potentials. Afterwards we provide a framework to recover the greybody factors from the actual energy emission spectrum, which has to be treated as the sum over all multipole numbers. The underlying method for the reconstruction of the potential widths is based on the inversion of the Gamow formula, a parabolic expansion and the Poschl-Teller potential. We define a "normalized" energy emission spectrum that turns out to be very beneficial for the numerical fitting process, as well as for an improved qualitative understanding of how much information of the black hole potentials are actually imprinted in the spectrum. The connection to recent studies on the inverse problem using the quasinormal spectra of ultracompact stars and exotic compact objects is discussed as well. In the Appendix we show that the spectrum can be approximated surprisingly well and simply with a parabolic expansion of the peak of the classical black hole scattering potentials.