John Singer Sargent is the sort of artist who aggravates the critics. In the eyes of Critic A he offends by virtue of having been a salon painter, producing portraits to commission, demonstrating little original artistic vision and embodying sickly late Victoriana. Critic B, on the other hand, is anxious about the way in which Sargent’s works, which largely depict wealthy white sitters, are bound up with privilege.

For Critic B, Tate Britain’s decision to present this exhibition through the lens of fashion alleviates some of her concerns, because fashion exhibitions are cool, and it’s great that the Tate is foregrounding the overlooked history of material culture. Critic A remains unimpressed: this curatorial decision puts Sargent beyond the pale, merely reinforcing the triviality, even the ‘femininity’ of his oeuvre. And what of the ordinary art enthusiast who likes Sargent’s work and comes to the exhibition with neither prejudice nor anxiety? Well, for them, this exhibition offers much to admire, but the best attitude to take towards the fashion theme may be largely to ignore it.

There is no denying that Sargent was extremely accomplished at capturing the texture of fabrics – how is it possible to render in oils the texture of velvet (Mrs Hugh Hammersley) or the precise sheen of cream satin (Mrs Joshua Montgomery Sears)? And he certainly depicted his subjects wearing some notably daring outfits: Madame X’s risqué black dress, or Dr Pozzi in his crimson dressing gown. Yet the exhibition makes no attempt to trace any chronological development in fashion history and, in the case of many of the paintings, the only tangible connection to ‘fashion’ appears to be that the sitter is wearing clothes.

Often the thing that strikes us most about Sargent’s portraits is not a sitter’s attire so much as their personality. Sargent is adept at capturing expressions – no ‘default’ faces here, as in the works of Waterhouse or Burne-Jones – and does interesting things with pose and composition that bring his sitters to life. In room two, themed around the colour black (black clothes, black walls), we know not to get on the wrong side of the authoritative, no-nonsense Jane Evans, house mistress at Eton, and we feel we could have a good gossip with the cheeky Mrs Edward Darley Boit. Perhaps most eye-catching is Mrs Edward L. Davis, with her assertive pose, wrist on hip: the embodiment of modern American wealth. Her eyes sparkle with a captivating brightness and confidence; her young son’s glaze over slightly as the boredom of the pose sets in. Sargent is brilliant at eyes, incidentally. He gets the fact that nobody has an identical left and right eye, and that from such asymmetry lifelikeness springs.

Subsequent rooms are themed around ideas rather than shades. Some of the themes are rather vague (‘Sitting for Sargent’, ‘Cutting a Figure’), others laboured (Vernon Lee works well in the room themed around gender ambiguity; Henry Lee Higginson, a military man in conventional male daywear, does not). ‘Performance’ is a fitting theme for the Spanish dancer la Carmencita, whose portrait is illuminated by being set in proximity to the actual dress worn and a film clip of her dancing in it. It is a struggle, however, to see how Lord Ribblesdale, top-hatted in hunting attire, fits into this room. Does this painting really present, as all the pictures in this room are said to, ‘a different, often complicated story about identity’ or is the man simply on the lookout for the next fox?

The exhibition takes us through a succession of vividly coloured galleries: deep green, rich plum and lavender blue. Perplexingly, paintings are not always shown off to best effect against these hues. Mrs Charles E. Inches wears a sumptuous deep red ballgown, and the real item is displayed in a cabinet beside it, though neither looks as striking as it might when placed against a paler scarlet wall with undertones of tomato. Elsewhere, orangey salmon-pink walls do not flatter Aline de Rothschild, who looks vaguely queasy, and the blue silk backdrop to ‘Lady Agnew of Lochnaw’ clashes with the laurel-green wall against which this most stunning of portraits has been hung.

Gripes about curatorial choices aside, however, this is a gorgeous exhibition, spacious, abundant and featuring almost no duds. Most of the big showstoppers you could wish for are present, including the perennial favourite, ‘Carnation, Lily, Lily, Rose’ (whose young sitters were almost certainly not interested in fashion). There are also many impressive paintings you may not have seen before. Charles Stewart, Sixth Marquess of Londonderry dressed up to the nines for Edward VII’s coronation, carrying what we must now refer to as Penny Mordaunt’s sword. The astonishingly tall, slender Winifred, Duchess of Portland, whose steady, direct gaze seems almost shockingly sexually inviting. The sumptuous painting of the flush-cheeked Mrs Fiske Warren and her daughter, in gorgeous cream and golden-hued pink,  is made all the more charming for being displayed next to photographs of the pair in the artist’s studio. Here they are ‘off-duty’ and smiling, grateful to be able to abandon their formal pose, and our jaws drop to see how closely the real people resembled their embodiment in oils.

With a different curatorial approach, there are themes in the exhibition that could have been brought out more clearly, to illuminate who these people were, their interests, enthusiasms and interconnected relationships. Not least, scattered around the exhibition are many sitters who had an interest in music. Chief among these is the pretty Madame Ramón Subercaseaux, who leans back casually from the piano (its keyboard an impressionistic single line of white) to gaze straight at the viewer. There are also paintings of music patrons and enthusiasts, such as Mrs Robert Harrison or opera lover Mrs Carl Meyer, both good friends of Sargent’s. Outside the exhibition, there is on the wall a list of works by Fauré, Chabrier, Albéniz et al,  which were important to the painter and a link to a Spotify playlist. Visitors are invited to walk around the exhibition immersed in Sargent’s sound world, but no headphones are provided, so almost nobody does.

As the exhibition moves to a close, it adopts a chronological focus not evident in the earlier rooms. By the late 1900s, Sargent was moving away from society portraits. Many of his later paintings were of outdoor scenes and his style had taken a rather impressionistic turn. Here, interestingly, there seems to be a closer focus on fabrics – coarser fabrics, indeed, than the delicate silks and satins of the drawing room. Facial features and expressions are hinted at rather than captured with forensic precision and so here, at last, in paintings such as ‘Nonchaloir’ or ‘Cashmere’, it is the textures of cloth that seize our attention first.

Fashion sells, as the V&A has shown time and again with its blockbuster shows, and Tate Britain probably hopes to draw in a crowd who wouldn’t turn up for an exhibition marketed simply on the Sargent name. To be honest, the fashion conceit supposedly underpinning this exhibition is often strained to its limit. Do go to this exhibition though, and then go back again, each time drawing your own connecting threads between the sitters and others in the room, and you will find it an absolute delight.

Sargent and Fashion is at the Tate Britain until 7 July.