Leave plenty of time!
Plan to get to Hay-on-Wye as early as possible so you don't feel rushed. If you plan it right, you can get out before the last bus and then have time to do something else in the evening.
The buses leave from Hereford, in UK as early as 8:30 a.m. and it takes about an hour to get to Hay-on-Wye. And coming out, it is 5 p.m. for the last bus. And I am not sure about Sundays.