Dan Davies has a lot to get off his chest. I ask one question and the words come flooding out. But then, there are so many words to come out. And he's been ruminating over them for so long. Because, for nearly a decade, Davies has lived with Jimmy Savile. Or at least lived with thoughts of Jimmy Savile, the idea of Jimmy Savile, questions about Jimmy Savile – the most basic being who exactly is Jimmy Savile? He's spent the past decade trying – and, until now, failing – to write a book about him, though his interest in him dates even further back than that.

"It was 1980, and I was nine years old," he says. His mum, for a treat, took him to a recording of Jim'll Fix It. "It was a big deal. It was still the age of three television stations; he was one of the biggest stars in the country. And I just had a really weird, visceral reaction against him. I couldn't really put it into words. But I was creeped out by him." In his teens, he stumbled across a copy of his autobiography, "which was just so dark and weird and odd and posed so many questions about him" that his fascination grew. And, from then on, he found himself collating a "dossier". He just kept on noticing him cropping up. In Andrew Morton's biography of Princess Diana. In the Sun, describing himself as the "Godfather", who "fixed" things. Telling Lynn Barber in an interview for the Independent that getting a knighthood was "a relief", that it got him "off the hook".

Is this not just hindsight talking? "Friends of mine would tell you they were bored of me going on about Jimmy Savile. They would all vouch for that." He became a journalist, and eventually an editor "got fed up with me banging on about him and told me to go and interview him".

That was in 2004. "And that's when I first had the idea for the book. This interview that was meant to last an hour in his house lasted, I don't know, God, it was about seven hours, something like that. It just went on and on and on." He interviewed him again for another profile in 2006. And another profile again in 2008. Shortly after which he – bizarrely – ended up going on the QE2's farewell Mediterranean cruise with him and began researching his biography in earnest. "I saw myself going up the river of his life and hopefully finding out everything on the way and then having a climactic final confrontation with him. I was going to call it Apocalypse Now Then. The implicit awareness was that it was going to be dark, because even in that first meeting there was a real, dark, underlying subtle menace to him."

And then he died?

"Yes. And then he died. I was just… I was really angry. I just felt like I'd wasted so much time. I felt so stupid. I just hadn't got far enough. I hadn't really got to the crux of who he was."

He went to the funeral but the book languished. Publishers told him no one would be interested to read about the "darkness" at the heart of a national hero, "but I decided I had to keep on going anyway even if I self-published". And then, finally, it all came out. "After Newsnight's investigation was axed, I met with Meirion Jones [Newsnight's investigations producer], who shared with me what was already in the public domain but only if you knew where to look…"

The publication of the book this week marks what Davies hopes is the end of this "painful process", the culmination of a decade of obsession. He's weary rather than jubilant. It is a relief, finally, to get it out there, though even this he's doing cautiously. He didn't want to sell an extract from it, which is what would usually happen, because "I really didn't want to sensationalise it. I don't want it to be seen as a sensationalist book."

But it isn't. In Plain Sight: the Life and Lies of Jimmy Savile is the first thoroughgoing attempt to try and make sense of Savile's life – his crimes, his place in the nation's culture, the roots of his pathology, his network of influence at the very top of British life, and the still unbelievable breadth and extent of his offending over the course of more than six decades. Except sense isn't the right word. There is no sense here. This is not a story that has narrative coherence. It's a riveting read; pacy, incredible, gobsmacking, though Savile eludes to the very end. He is impenetrable, unknowable, the black hole at the heart of the book.

Davies, however, documents it all: his Depression-era boyhood; his devotion to his mother, "the Duchess"; his wartime experiences as a Bevin Boy down the mines; his almost single-handed invention of disc jockeying in Mecca ballrooms up and down the land; his place at the epicentre of the 60s music scene; his reinvention as one of the biggest stars of 70s TV; his acclaim as a charity campaigner; his knighthoods – one from the Queen, one from the pope – his friendship with Margaret Thatcher; his "mentorship" of Prince Charles; his role as confidant to Lady Di, and, finally, the three-day quasi-state funeral featured on all front pages, on all news bulletins, to which David Cameron, Nick Clegg and Ed Miliband, as well as Prince Charles, sent tributes.

Davies begins with the dismantling of Savile's grave, a six foot-wide, four foot-high triple headstone that had taken the stonemasons eight months to complete. In the wake of the first outpouring of revelations, Scarborough council decided to take it down, a mechanical digger ripping apart the epitaph carved across the bottom: "It was good while it lasted."

"I do think he knew it would all come out," he says. "And I think in a way he was trying to groom me to write a very positive biography about him. Or at least he tried to groom me. He did offer me access, show me kindness. He was charming, he was charismatic, in a very weird, odd way. He was fascinating and the fact that he kept not indulging me, but offering me this access even when I wasn't writing anything about him, even when I kept coming back to him saying: 'I want to write this book about you.' He'd say: 'No, no, we're not writing a book.' But he still let me interview him for three days."

You were supposed to be his Boswell?

"Maybe. It didn't turn out that way, though."

It didn't. Because In Plain Sight, all 567 carefully footnoted pages of it, is a devastating portrait of "Sir Jimmy", the hero of Stoke Mandeville, the nation's favourite eccentric, Yorkshire's most famous son. There is just so much material: harrowing accounts of 12-year-olds who'd gone to have their tonsils out and ended up being raped in the hospital TV room; the testimony of friends and producers and former secretaries; the transcript of the police interview when he was told, under caution, about allegations of sexual assault made against him by former pupils at Duncroft approved school, in Staines; the cataloguing of his high-ranking friends in West Yorkshire and North Yorkshire police; details of how Margaret Thatcher lobbied for his knighthood. Perhaps most bizarre and compelling of all, though, is Savile's own testimony against himself: the columns he wrote for the Sun, which detailed how he picked up teenage girls on Scarborough seafront; the accounts in his autobiography of how he'd rough up troublemakers in his days as a dancehall manager; the "3,000 birds" he'd slept with. Or what Davies calls his habit of dispensing "unorthodox high-wire alibis".

In God'll Fix It, a slim volume of his thoughts on religious affairs, he stated how he believed that life consisted of credits and debits.

"The thing I keep coming to is that he always had these figures he quoted. 'I've run 221 marathons.' 'I've had 107 wrestling bouts.' 'I've raised £45m for charity.' So if you balance on one side, £45m-worth of charity, and that is the counterweight to what's on the other side… well, there must have been an awful lot on the other side. And that was his words in 1978, which is at the end of the peak period, if the statistics from the Metropolitan police are anything to go by."

It is an incredible read; preposterous, unbelievable, utterly damning about the institutions that were taken in and duped by Savile: the NHS, the BBC, the royal family, the government, the civil service, the entire nation. And a cast of characters that spans everyone from Elvis Presley to the Beatles, from the Rolling Stones to the Queen to the pope. It could never have been fiction. Everything about the Savile story strains credibility. The man who dressed like a paedophile was a paedophile. Though this is only one part of it – just as paedophilia was only one of his many crimes. It was the kaleidoscopic aspect of Savile's life that gave Davies the idea to write the book in the first place. He didn't just want to write about Savile but had the idea that the history of the nation's popular culture could be told through him.

"He occupied such a big part in successive generations' lives. From our parents' generation, he was someone who sat at the epicentre of the nascent British pop scene when it exploded and took over the world. He described it as being like the cork on the breaking wave. He was right there. He pioneered playing records in dance halls. He was hosting, arguably, the biggest show on radio for pop music. And then he got Top of the Pops.

"Then, in the 70s, he became ubiquitous on TV: on Top of the Pops, on Jim'll Fix It, on the "clunk click" campaign, in the British Rail ads. For all those different generations, you've got such different things. And it was such a very long career he had. From the late 50s really right through to his death. It's a hell of a long career."

A hell of a long career as a celebrity, an even longer one as a sex offender. Because what Davies's book makes clear is the relationship between Savile, Britain's most prolific ever sex offender, and Savile, the national celebrity. How the one accounted for the other, or at least how they were intimately entwined. His fame was the bedrock of it all. It enabled access everywhere, afforded him protection and gave him an instant in with his victims. Unbelievably, in the 70s, he was the face of a children's book, Stranger Danger. But then he was the perfect face for it because he wasn't a stranger. We all knew who Jimmy Savile was.

What becomes clear is that it's not just Savile who's in the frame here. It's celebrity. Our attitude towards it. The way we treat it. The adulation we afford it.

"Especially since he's one of the first celebrities, as we know them now. He was somebody who said: 'I can't sing, I can't dance, I don't tell jokes – what do I do?' He was famous for being famous. But he came out of an era in which this sort of fame was a new phenomenon. That morphed into the 70s and 80s, when celebrity culture solidified and cemented to the point that when people might have been thinking, 'Hold a second, that's a bit dodgy', he was untouchable."

But this has gone largely uninterrogated, though it's Savile's untouchability that comes across so strongly in the book. One of the most enraging, bleakest moments of all is the transcript of the interview two police officers conducted with him in 2007. Victims had come forward – former pupils from Duncroft approved school. Their allegations were taken seriously. But even getting an interview with Savile had been an obstacle-strewn course. Files vanished. The CPS delayed for months. When the interview did, finally, take place Savile ensured it was conducted on his home turf.

"'I have a policy,' he told the officers. They were calling him Jimmy. 'Can I call you Jimmy?' they said. 'What do you mean by a policy?' 'Well, I have a policy that I will pass this to my people and my people are big important people and we will be able to put time in the Old Bailey and you will be in the Old Bailey and these people will be in the Old Bailey.'" It worked. The investigation was dropped.

"His life reveals itself as a landscape of red warning flags," says Davies. "It's just flag after flag after flag. People did try and report him but a lot of the reckoning that we are going through now is an understanding that the victims have a real issue with authority too."

Has there really been a reckoning, though? Have we actually processed the fact that a serial sex offender was at the heart of our culture, our national institutions, for 50 years? Or have we managed to simply write him off as a weirdo?

"No, I don't think we have. I think it just feels like inquiry after inquiry. We need to get to the bottom of why it happened. Why it was allowed to happen. Why people didn't feel that they could speak up, why they weren't believed when they did speak up, why they weren't listened to. If something good is to come out of this then, hopefully, that will be what it is."

In fact, it seems increasingly clear that we haven't truly reckoned with the past. There have been inquiries into the police investigation of Savile. There have been, or are ongoing, NHS inquiries into 28 hospitals he had a connection with. A new BBC inquiry led by Dame Janet Smith has yet to be published but it's believed it will show there were 1,000 victims there. It rolls on and on. And yet.

Savile is dead. There was no trial. No verdict. He got away with it. None of the victims have seen justice done. More and more information about what he did and how he did it is coming to light, but it doesn't change this one simple, irrevocable, unsurpassable fact. He got away with it. And it's this, more than anything else, that perhaps makes Savile so difficult to process. We are all affected by it. He was part of the wallpaper of our lives. And none of us have had our day in court.

"The only person unaffected by it is him," says Davies. "He got away with it. He got to die on his bed at home, which is what he wanted to do, and he got to have his big funeral. I think he would have seen that as the ultimate win." People weren't believed. The police investigation was dropped. Celebrity won.

In November 2012, in an article for the London Review of Books that quoted Davies at length, Andrew O'Hagan wrote about Savile and his place in the "creepy" culture of British light entertainment. He made the point that we get the celebrities we deserve. That our culture is paedophiliac. That Savile was our creature. We created him.

And the deference that we accord celebrities, the privileges they enjoy, the thrall in which they hold us, remains unexamined. Savile's victims ranged in age from five to 75, came from every social class and both sexes; his offending was as prolific and indiscriminate as his charm, his magical aura. The effect of the book, and the most recent NHS inquiries – which contained the most shocking revelations so far but which didn't make it into the book – is that everything in Savile's life was structured around committing these crimes.

"Somebody said something along those lines, that it was every minute of every day. I was thinking, God, is that actually possible? But then you read those NHS reports – particularly the Leeds one – and he is literally taking opportunities as they arise. And not just taking them but engineering them… He targeted people at the top and bottom of institutions. He was very, very clever. So he would make a beeline for the house governor, the top person at the hospital, and then the humble porters. So he'd have both ends of the spectrum covered."

He had the chief porter at Leeds infirmary on his payroll – he made him secretary of his company, in 1981, when he earned £91,500. And he had Mrs Thatcher in his debt. He took the financial embarrassment that was Stoke Mandeville hospital off the government's hands (he raised £12m for a new spinal injuries unit) and he never let her forget it.

And you, Dan? When you found out about his crimes, did you ever feel that he'd also made you complicit?

"No, because if I'd ever found anything out, I would of course have printed it."

But Davies admits that he did "throw me off balance. He did suck me into this kaleidoscopic world of anecdote and well-polished stories that he'd told many times. But then he talked about other stuff as well that was just intriguing. But he never completely won me over."

It's a strange coincidence to be published in a week like this one, I say, when historical sex abuse has dominated the headlines again. When two new inquiries have been launched. What do you make of those?

"I think it's deeply worrying that 114 files have gone missing or have been destroyed. I don't see how the establishment will ever allow a true understanding of what has happened to emerge."

You think another cover-up is inevitable?

"I fear another cover-up is inevitable. I am astonished – well, I'm not astonished, I'm a Hillsborough survivor so I'm never particularly astonished by cover-ups. One hundred and fourteen files go missing implicating supposedly some prominent figures within the political establishment. The circles moving out from that will be considerable."

In Savile's case, there are numerous incidents of clumsy cover-ups. Davies is scathing about West Yorkshire police's inquiry into its conduct regarding Savile. He notes how a telephone conversation between Margaret Thatcher and Savile has "gone back into the vault". How Surrey police redacted the name of Princess Alexandra from a report into Duncroft approved school.

But then, it's all about power. Perpetrating abuse. Covering up abuse. Everything about Savile's story is the story of power. The power he had over his victims. The power he acquired through his establishment contacts. The power of his fame. Who Savile was and why he did what he did remains frustratingly opaque, though plenty of suppositions are put forward. His intense relationship with his mother and his unwillingness to ever talk about his father. His first sexual experience, which he didn't see as abuse but which we probably would today. As a young teenager, a woman thrust her hands down his trousers and coldly "took what she wanted", a pattern eerily close to his behaviour towards his own victims. The fact that Davies discovered Savile's older brother, Johnnie, had lost his job as the entertainment officer at south London's Springfield psychiatric hospital for sexually abusing a patient. A family of seven, he notes, produced two sons who were sex offenders.

Savile's preoccupation with death is convincingly ascribed to the fact that he grew up opposite an old people's home, where the nuns encouraged him to spend time with their bodies – "to say goodbye". And the five days he spent next to his mother's open casket, he told Lynn Barber, were "the best five days of my life".

Davies confesses in the book that, for a long time – before the truth about Savile came out, while he was still obsessed with his dark side but didn't know what it was – he wondered if Savile had murdered someone. But then, given the nature of his transgressions, including the latest from the NHS inquiry at Leeds infirmary, which found allegations of necrophilia, and the instances of violence cited in the book, it's hard to know what he would stop at.

If you could ask him anything, I say, what would it be? Davies thinks about it. "Did you feel remorse for what you did? Did it plague you when you were alone in your flat at night? Did you sleep easy?"

It is the end of a long road for him.

"My instinct was right. I was right. It's what I keep coming back to. But I do feel angry that I've spent so much time of my life on him. It's just such a dark and horrible story. It could have been somebody who'd done something really, really truly heroic and amazing."

So why do you think that was?

He shrugs. "I haven't really come to that." Did I choose him, he asks at the end of the book. Or did he choose me?

DAN DAVIES ON HIS FIRST ENCOUNTER WITH SAVILE

The taxi turned into West Avenue in Leeds and continued a few hundred yards up a gentle incline before dropping me off outside Lake View Court. I got out and pressed the intercom button marked "Penthouse" and after a short pause, a voice: "Morning." The sound of the Yorkshire Dalek was unmistakable. The door buzzed, I pushed against it and took a seat in a small lobby that smelled of potpourri.

Two or three minutes later, I heard voices coming from the lift shaft. The wooden doors slid open, releasing a cloud of smoke and two large, unsmiling men in their 50s. "Frisk him," barked Jimmy Savile, who had stepped out of the lift behind them and was wearing a blue shell suit with chevrons of red and white on the shoulders.

I was pinned to the wall and searched before Savile finally called the men off. He chuckled and extended his hand, introducing them as Mick Starkey, a West Yorkshire police inspector, and Jim "The Pill" Cardus, a retired pharmacist. "Meet the Friday Morning Club," Savile trumpeted before ushering the men out of the front door to the flats.

It was March 2004, our very first meeting. Jimmy Savile had thrown me off balance. The rules of engagement had been established: he was on home turf and he was in charge.

The lift opened directly into a wood-panelled vestibule adorned with plaques from Royal Navy ships and army regiments. A cascade of woolly hats and thick coats of the type more commonly worn by football managers clung to a row of hooks; beneath them, a row of running shoes.

This cramped entrance hall opened right into a long, rectangular living room with electric blue shagpile carpet and floor-to-ceiling windows on two sides.

The clutter of the room's time-warp interior was in stark contrast to the panoramic views of Roundhay Park and the hills beyond. An ancient-looking exercise bike, a low sideboard with two This Is Your Life books lying open on the top and a glass-fronted cabinet stuffed with what looked like cups, medals, plaques and various awards from his career in entertainment dominated the first half of the room.

At the far end was a black swivel reclining chair, an L-shaped white leatherette sofa and a coffee table cluttered with ashtrays, pens, brochures and assorted pieces of paper. Perched on the top of a modest black stack hi-fi system was a framed photograph of Jimmy Savile in a field. He was wearing a kilt and struggling under the weight of a 15-foot caber. Above it was another framed shot: Jimmy Savile standing with Prince Charles and Princess Diana on the steps of the National Spinal Injuries Centre in 1983.

He showed me through to his kitchen. It was decorated in tiles of pink and brown, or as he put it, "the colour of sex". He asked me what was missing but I already knew, having read scores of newspaper and magazine interviews over the previous 20 years: it didn't contain a cooker. He liked to boast that none of his many homes had one. "It would give women the wrong idea and that would only lead to brain damage."

Coffee made, he shuffled into the front room and sank into his black recliner that allowed him to lean back at a 45-degree angle. Lighting one of the three giant Bolivar cigars I had bought him – one of his conditions for granting the interview – he announced that we could now begin.

That first interview, which was scheduled to last an hour, went on for the entire afternoon and into the early evening. It set the template for the series of lengthy meetings that followed, staged in the same flat in Leeds or in the seafront apartment he had bought for his beloved mother in Scarborough. In time, these summits began to stretch over days and nights.

The routine became familiar; the reclining chair, a fresh Cuban cigar being lit as a symbolic cutting of the ribbon on the talks, although talks is accurate in so much that I would be expected to initiate a line of questioning before listening in silence as he laid the brick of one anecdote on top of another, gradually disappearing behind the wall of his own mythology. He selected stories as I imagined he picked records in his dancehalls during the 50s and early 60s.

After hours spent enveloped in the fug of his cigar smoke, we'd break to walk down to the front in Scarborough, to have tea in a local cafe or to eat in his favourite pizza restaurant in Leeds. I once slept in the room he kept as a shrine to his mother, "the Duchess", who died in 1972, and on another occasion, in Leeds, was forced to take evasive action from an avalanche of platform shoes and Top of the Pops outfits that spilled out of a cupboard in his spare room.

Occasionally, between the retelling of those same, well-polished tales, a crack would appear that offered a glimpse into what lay beyond. This usually happened late at night when he'd had a drink, or when there was something on his mind. It was a matter of remaining vigilant.

In the days and nights I spent with him, he succeeded in drawing me in and knocking some of the sharp edges off my doubts about him. But he was never less than evasive, and narcissistic to the point of only talking about himself in terms that fed his myth. And while I asked him about the rumours that dogged him on every occasion we talked, in hindsight I wish I'd pressed him more on sex, his lack of empathy and why he insisted on remaining so concealed.

The reality is, though, he was never going to give up that which he'd guarded so jealously and for so long; the things he must have known would lead to his certain fall.

• This article was amended on 16 July 2014 to make clear that Surrey police, not Buckingham Palace, redacted the name of Princess Alexandra from a report on Jimmy Savile's conduct at Duncroft approved school.