I finally figured out how to break on thrown exceptions in Jetbrains Rider! I found it via this StackOverflow answer.
The way to do it is to go to “Run | View Breakpoints” and then stop on all .NET Exceptions. Enabled and Suspended are what we need… as with Visual Studio, you’ll want to mute others as they come up, but that’s easy enough.