For however long you wish. You will know if it is getting stressed out and needs a rest. Just make the holding sessions not hours long and you'll be fine.
My Leos love to be handled. Often when I take one out others try to scale the walls of their tubs, seemingly to be next in line. I don't know anything about their memory span, but it could be that they associate being held with clean cages or food, since that is mostly when I hold them. They love the warmth too, last night one fell asleep in my hand.