All of Jake Thackray's recorded music, more or less, is available on Spotify, but not in a way that makes it very easy to get at. EMI's 4-CD box set contains all four of his studio albums, plus an entire acoustic alternative of one of them, plus a load of extras, but as far as Spotify is concerned it's just 4 CDs. So I've put together some playlists to make it easier to pick one of the albums and listen to it, and here are the links. His first album was Last Will and Testament; its orchestrations don't at all reflect his live performance style so you may prefer to listen to the acoustic version, although the track listing is not exactly the same. His second album was Jake's Progress and his third Bantam Cock, with decreasing amounts of accompaniment, and in his last album, On Again On Again, he finally got down to the sound of a man and a guitar, much the most natural way to hear him. There's also a live album, released as Live Performance, containing 29 songs. I've collected the miscellaneous songs from the box set into another playlist for completeness - these seem to be a mixture of demos, singles, alternate takes, and songs that were rejected for the released albums.
There are two collections also on Spotify, Lah Di Dah and The Very Best Of Jake Thackray. These each contain assorted tracks from the individual albums.
The links in this post are all to the http URLs that Spotify generates for you, not to their "spotify:" pseudo-URLs. I don't know whether that's better for you or not. Please let me know if you have a preference.