I'd driven down from Bari airport with the sort of misplaced confidence that only comes from having watched one YouTube video about Italian roundabouts. The hire car was too small, the air con was too feeble, and my partner had that particular set to her jaw that suggested I might have taken the scenic route once too often. Three hours, two arguments about whether olives counted as lunch, and one spectacularly wrong turn later, we found ourselves on a dirt track between two dry-stone walls, staring at a goat who seemed personally offended by our presence.

The thing is, Ostuni doesn't announce itself gently. One moment you're winding through dusty countryside wondering if your sat-nav has developed a vendetta, the next you crest a hill and there it is: the White City, blinding in the afternoon sun, piled up like somebody's dropped a sack of sugar cubes on a limestone ridge. We were staying in a trullo just outside the old town, one of those conical-roofed stone houses that look like they've been there since before anyone thought to write things down. Ours had a small pool, a terrace of pale stone, and an outdoor kitchen that immediately made me feel like the sort of person who buys fresh orecchiette from a nonna down the road.

I am not that person. I am the person who forgot to pack swimming trunks and spent the first afternoon doing a very dignified breaststroke in boxer shorts, hoping the neighbouring trulli weren't occupied by anyone with binoculars and a sense of humour.

But here's what I hadn't properly understood about Puglia, despite all the brochures and the well-meaning advice from friends who'd been before: it genuinely doesn't matter. The whole place is constructed around the radical notion that doing very little is, in fact, doing something rather important. By day two I'd stopped checking my phone. By day three I'd read two-thirds of a novel I'd been carrying since Christmas. By day four I was having earnest conversations about the quality of the tomatoes at the small grocer in the old town, as if this were a perfectly normal way for a grown man to behave.

Mornings became a ritual of slow awakening. The light in a trullo is strange and lovely, filtering through thick stone walls, so you surface gradually rather than jolting upright. We'd wander up to Ostuni's centro storico before the heat built properly, meandering through whitewashed alleys that seem designed to make you lose your bearings on purpose. There's a cathedral, of course, and various bars where elderly men conduct serious business over tiny coffees, but mostly there's the pleasure of simply being somewhere that hasn't hurried to meet you.

Back at the house, the afternoons dissolved into a routine so simple it felt almost illicit. A swim, perhaps, if I could locate my trunks. A book on the terrace. The occasional olive, which I will maintain absolutely counts as sustenance if you're horizontal enough. My partner took to sketching, though she claims everything came out looking like vegetables, and I developed an entirely unearned expertise in the local Primitivo wine.

We did venture to a nearby beach one morning, but found ourselves back by mid-afternoon, slightly sun-drunk and relieved to return to our quiet corner. The real discovery, I think, was that I'd spent years confusing activity with experience, collecting sights like some sort of anxious tourist merit badge. Ostuni offers something else entirely: the profound, slightly embarrassing relief of realising you don't need to be anywhere else, doing anything else, being anyone else.

On our final evening, I cooked badly on the outdoor grill while the sun dropped behind the olive groves. The smoke got in my eyes, the lamb was arguably overdone, and I knocked over a glass of that Primitivo I'd grown so fond of. Nobody minded. We sat there as the swallows began their evening circuits, the White City glowing pink across the valley, and I understood something about holidays I'd been too busy to learn before. Sometimes the best thing you can do is absolutely nothing at all, and simply let a place work its quiet magic on you.