Dirty Projectors' mastermind David Longstreth has talked up the Brooklyn collective's Swing Lo Magellan as "an album of songs". As a USP, that seems fairly underwhelming. You could say every album on this week's rock and pop release schedule, from technical death metal titans Nile's At The Gates of Sethu to the expanded reissue of Showaddywaddy's 1979 opus Crepes and Drapes, is an album of songs.

But the comment makes more sense in context of the Dirty Projectors' back catalogue and the polarised response it's received. To their fans, some famous (Bjork and David Byrne are past collaborators), Longstreth is a polymath genius, flitting between 20th-century orchestration, medieval vocal polyphony and what Vampire Weekend's Ezra Koenig – a former Dirty Projector himself – admiringly called "a fucked-up version of American music". He is so brimming with original ideas, so superior to his peers, that he bears comparison to the late Frank Zappa.

Then there are people for whom that comparison is the problem in a nutshell: like Zappa, he deals in smug, arid intellectual exercises, albums that are – to borrow a phrase from the former culture minister Kim Howells – cold, mechanical, conceptual bullshit. It's an intriguing schism, summed up in the fact that people who love Longstreth have an unerring knack of describing his albums in a manner apparently designed to instill in anyone who hasn't heard them a burning desire never to do so as long as they live: "So good at what they do they can be hard to like", "the execution is as important as the music itself" and, indeed, "the listener can picture a West that has been smithereened into archipelagodom, where the human survivors place their faith in po-mo calypso's power to impress the warlord carnival judges".

Listening to this sixth album, you do wonder if Longstreth might have noted some of the dissenting voices and conceded they have a point. Certainly, there's a huge emotional gulf between, say, 2007's Rise Above – on which Longstreth "reimagined" the music of hardcore punks Black Flag and drained every last drop of feeling from it in the process – and this album's Impregnable Question or See What She Seeing, both plaintive love songs, the former boasting a lovely McCartney-esque melody.

There are moments here when old habits appear to die hard: the album opens with a burst of weird, comically over-emphatic humming, as if to suggest that what follows should be taken in inverted commas, while Unto Ceasar features a lot of breaking-the-fourth-wall stuff involving the backing vocalists asking "when should we burst into harmony?" before singing. Whether you find that kind of thing brilliantly witty or profoundly irritating, it's vastly outweighed by songs that seem designed to be taken at face value. Gun Has No Trigger has a rather Zappaesque topic – how easily conformity overwhelms dissent – but rather than sneering about it, it sounds anguished; there's a fantastic moment at the end of the chorus when Longstreth's voice cracks as it sings the song's title and the backing vocals switch from softly cooing to an emphatic wail. It's genuinely moving, not a phrase you would apply to much of their past work. So is the closing Irresponsible Tune, of all things, a heartfelt acoustic ballad about the power of music: "Without our songs … life is pointless, harsh and long."

But not all of Swing Lo Magellan is that straightforward. Disparate styles still crash together, rhythms are fidgety, time signatures are tricksy, and at times Longstreth appears to be singing a different song to the one the band are playing. But what's striking is how easy it is to listen to, partly because the melodies are unfailingly great, partly because the emotions strike home, but mostly because the tricksiness feels like it's there in service to the song, rather than for the sake of it. When Dance For You's skittering drums and trebly guitar is suddenly submerged beneath a luscious interlude of orchestra and church organ, it fits with the song's lyrical evocation of music's mysterious power: "I want to feel the breath of a force I can't explain."

You could argue that making your music more accessible without sacrificing any of its originality is an infinitely more impressive feat than all the brainiac stuff that's preceded it in the Dirty Projectors' oeuvre. There's always a chance that it is all intended in inverted commas, which would be a bit depressing, but perhaps that doesn't matter. The best moments of Swing Lo Magellan transcend whatever intentions their author may have had, as amazing songs are wont to do.