The simple answer: Yes it is. No leeway, ifs or buts.
To use any animal for our amusement is exploitation, and hence, not vegan.
Would you enjoy carrying a person on your back at various speeds whenever the person felt like it? Would you like to be periodically locked up? Would you like to be "broken in" during the early stages? Most likely not.
And whether the horse is treated "fairly", "kindly", or "companionably" ("happy horse riding!") whilst being ridden is utterly irrelevant as he or she is still used. Period. That is, the defining characteristic of veganism is that animals should not be considered property, and therefore we respect their needs and wants that demand that we do not exploit them.
In a perfect world, horses would be free to pursue their own lives without human interference. Horses are herd animals, who naturally associate with large numbers of other horses, graze in meadows, trot great distances, play, and court. Domestication (like many other animals) has certainly not benefited them, as they've long been exploited as beasts of burden and are kept constantly constrained and only allowed to move about at the will of others.
Exploitation is not just about being cruel; exploitation also means using animals for any kind of work or enjoyment.
For more information on horseback riding, check out this article: