Gilbert & George are excited. And proud. And smell so good that I’m tempted to ask what colognes they’re wearing. We’ve all just arrived at the Hayward Gallery on London’s South Bank, where their forthcoming show, 21st Century Pictures, is mid-hang. As soon as we’re through the doors, Gilbert darts about like a puppy let off the leash, urgently assessing the installation. “White Bastards!” George says to me, pointing to the 2004 piece as though he’s just spotted an old friend.

The artists may be in their 80s, but all the work here was made in the last 25 years, and the gallery vibrates with rudeness, colour, contrarianism and cheek, their often vast creations luxuriating, as always, in warts-and-all humanity.

Perhaps the most contrarian thing about this show is that there’s going to be an accompanying orchestral extravaganza at the neighbouring Royal Festival Hall. Musical pieces will be performed by the Philarmonia, picked in order to explore Gilbert & George’s themes of sex, money, race and religion (collectively, the title of a massive four-part artwork). Their art will be projected on to screens during the performance.

This all goes against much of what Gilbert & George stand for. Just last year George dismissed an immersive Van Gogh experience as “some populist nonsense”. But most of all, the event flies in the face of their longtime motto: “Music is the enemy.” I want to talk to you about music, I say as provocatively as possible, but they decline to rise to the bait. “I was a choirboy, after all,” says Gilbert.

“That’s an entirely different matter,” quips George, to guffaws around the room. But Gilbert continues: “I was even in a Bach choir, once. I must have been 18 or 19 because it was when I went to the Art Academy in Munich.”

Their big break as artists was musical, too, singing Underneath the Arches – the 1932 ode to homelessness – together. So are you a good singer as well, I ask George. “Of course,” says Gilbert. “But it was not important, the singing.”

“Coming from darkest Devon and from Italy,” says George, the Devonian, “we suddenly realised there were so many people who were in trouble in cities, and we could see in every corner there were tramps and young people in trouble.”

Gilbert takes over: “We lived in the East End of London, so they were all in front of churches, sleeping rough.” The pair befriended many of them, from survivors of the two world wars, says George, to men who, “had been in trouble with sex, and the vast majority had been caught cottaging in a public lavatory and sent to prison for three months, could never tell their parents or their employers.”

The song, which they started performing in 1969, says Gilbert, “made us famous overnight. We made it into a living sculpture. Normally, sculptures are quite dead. We used to do these metallised heads, full of colours. We put on Vaseline and then powders; extraordinary. And we did the Hyde Park walk. That was the day of the Rolling Stones concert.”

“We were just surrounded by people,” says George. “Young people asking us questions, imagining we had the answers to all the difficulties in life or something. Extraordinary.” Gilbert: “And then it became overnight, big success, galleries, New York, Amsterdam.” They would sing for around eight hours every day, and then were “badly behaved in the evenings,” says Gilbert. “We were entertained by all the famous artists at the time. The pop artists like Rauschenberg. Warhol took a portrait of us.”

“New York was extraordinary,” adds George. “It was pre-Aids. Life was wonderful, pre-Aids.”

So far, so rock’n’roll, and further clues to extracurricular musical enjoyment come when Gilbert reminisces about clubbing in 80s London. “We used to go to the Blitz,” he says of the club where the New Romantics used to wear their most outrageous outfits. “We were a part of the beginning, when they started it all.” George adds that the address was the icing on the cake: “Great Queen Street.”

Yet at some unspecified point in the distant past, they decided “to eliminate music from our art,” says Gilbert. “We felt that the streets of London were filled with galleries and theatres and everything, but that …” George finishes his sentence “… art was on the sideline a bit, because art with music didn’t feel very serious. Art without music is always better.”

Music, George adds, is “too soothing. We’re allowed to be against the middle-classes for liking music too much … Couple of violins and it’s all fine. Some pictures by us, ‘Woah, I’m not sure about that.’”

Do they enjoy music privately? “We don’t have the machine for doing that in our house,” says George . There’s a radio, but that’s for the news, “to check on the enemy.” And they watch films.

They seem to have little idea how or why this Philarmonia musical extravaganza came about, nor did they choose most of the repertoire, including Duke Ellington’s Black, from Black, Brown and Beige and Elgar’s Land of Hope and Glory. Although Underneath the Arches is on there, along with a medley of songs picked by them, not that they know what made the cut.

“We did make these little suggestions,” says George, rummaging in his pocket and retrieving a stack of hand-written notes which includes a list: “As Time Goes By, You Made Me Do It, Pack Up Your Troubles, I’ll Be Seeing You, We’ll Meet Again, There’ll Be Joy and Laughter and Peace Ever After.” They don’t sound very 21st century. “They’re just songs that – we’re war babies remember – the first music I could hear was connected with the war.”

What do they do to relax? “There’s a question,” says George. I’ve seen them having their nightly dinners at the Dalston Turkish restaurant Mangal II over the years. “That’s quite good,” says Gilbert. “Probably relaxing, yes.” Although it turns out they’ve since switched to Mangal I around the corner because Mangal II, says Gilbert, “started to have music every night and we didn’t like it.”

“Enough is enough,” confirms George.

The marketing blurb for the Philarmonia event asks: “Can the orchestra convince them that music is NOT the enemy?” “Whatever,” says George. “Just see what happens.” I get the impression they’re embracing the cross-pollination arts institutions go in for these days largely because they’re so chuffed with their retrospective, and are happy to promote and celebrate it in all the ways required of them. (Plus, it’s amusingly contrarian.)

And celebrating they are.

“It’s very thrilling to have a big show in a public space like this in the centre of London,” says George. Anything else they want to say about the new show?

“Make them come,” says Gilbert. “In many different ways.”

• Gilbert & George: Sex, Money, Race, Religion is at the Royal Festival Hall, London, on Sunday 5 October. Gilbert & George: 21st Century Pictures is at the Hayward Gallery, London, 7 October to 11 January