Viewing notes: Adam's connection was lost at around the 33 min mark, so you should skip from there to roughly the 38 min mark. Also, I am aware that I fluffed Hume's example about the destruction of the earth and the scratching of one's finger. I realised it at the time, but hopefully the basic gist the idea got through. I also didn't quite do justice to normative theories of rationality and how they feed into criticisms of the orthogonality thesis.
If you want to read more about these topics, my conversation with Adam was based on the following blog posts and papers:
For all my other writings on intelligence explosions and related concerns, see here.