It’s one of the great enigmas of architectural history. On Christmas Day 1699 John Vanbrugh wrote a long, gossipy letter to the Duke of Manchester. He described how he had spent the summer touring country houses, including ‘four or five days’ at Chatsworth, where he showed the Duke of Devonshire his designs for a house. The house was being built for the Earl of Carlisle, who wanted a new seat on his estate in North Yorkshire. Vanbrugh had proposed a grand design, with ‘low wings’ that stretched out on either side of the main house ‘being adorn’d with those Ornaments or Pilasters and Urns’. It would be called Castle Howard.

Vanbrugh was 35 years old. He had been a wine merchant in London and a trader in Gujarat; an officer in the Earl of Huntingdon’s foot regiment and a sailor in a naval attack against the French; a political prisoner in the Bastille and a celebrated writer of racy plays. He had little formal education, no training as an architect and had never before designed a building. In fact, nothing about his biography suggested that here was a brilliant pioneer of the English Baroque. Yet the Earl of Carlisle asked him to make plans for what became one of Britain’s most impressive stately homes. Why?

The traditional answer is that both men were members of the Kit-Cat club. This was a collection of authors and aristocrats with a shared commitment to Whig politics and witty conversation. As well as the playwright William Congreve, the philosopher John Locke and the journalist Joseph Addison, the club also included statesmen like the future prime ministers Robert Walpole and Thomas Pelham-Holles, Duke of Newcastle. Originally, the Earl of Carlisle tried to hire William Talman, architect of Chatsworth and Comptroller of the Royal Works, to build his house, but the two men fell out over fees. Then Carlisle turned towards a fellow member of the Kit-Cat club, John Vanbrugh, who did not charge anything, instead treating the commission as a labour of love.

Even if that story explains how Vanbrugh got the job, it does not explain how he designed a building of such splendour. This is the real mystery, and one Charles Saumarez Smith attempts to solve in his new biography of the architect. In John Vanbrugh: the Drama of Architecture, Saumarez Smith shows how Vanbrugh’s wayward early life provided a valuable preparation for his architectural career. Published to coincide with the 300th anniversary of Vanbrugh’s death this year, the biography argues that the originality and daring of the architect’s work was due in part to his lack of formal training, as Jonathan Swift (a staunch Tory and member of the rival Scriblerus club) suggested in this teasing couplet:

Van’s Genius, without Thought or Lecture

Is hugely turn’d to Architecture.

The politics of the period mattered. Members of the Kit-Cat club were committed to Protestant succession, a strong Parliament and hostility to the French. At the time, a strong Parliament meant a strong aristocracy: less an organ of democratic representation than a counterweight to royal absolutism. When James II was replaced by William and Mary, the monarchy was bound to a constitutional role, giving newfound power to noble families. Members of the peerage began expanding their country houses, or even building new seats as symbols of their growing influence. For example, the Duke of Somerset rebuilt Petworth House between 1688 and 1702 on a grandiose scale inspired by Versailles.

In this context, Vanbrugh’s biography was an advantage. He was descended on his father’s side from Dutch merchants, while his mother’s ancestors linked him to several gentry and noble families. He had inherited his family’s commitment to parliamentary democracy and the Protestant cause, and even carried secret messages to William of Orange in the years before the Glorious Revolution. This was the reason for his imprisonment in France: arrested in Calais and charged with spying, Vanbrugh spent the second half of his twenties in captivity, and freed in 1692 during an exchange of political prisoners. After his release, he spent several months in Paris, which not only introduced him to French comic dramatists, but also the magnificent buildings constructed during the reign of Louis XIV.

At the same time, Vanbrugh’s lack of architectural experience was not the disadvantage it might be today. Over the course of the 17th century, as classical buildings became popular among the upper classes, architecture was increasingly viewed as a liberal art. Discussing buildings and exchanging drawings became another of those clubbable aristocratic activities, such as collecting sculptures or gambling on horses. Training required travel to study the palazzos of the Italian Renaissance, as well as scholarship to learn the rules of Vitruvian proportions or the layout of Roman villas. The master mason was demoted, while the gentleman amateur with the leisure and learning to conjure classical designs was promoted.

Of the great architects of the English Baroque, only Hawksmoor learnt his trade the traditional way. Christopher Wren was professor of Astronomy at Oxford, Inigo Jones a designer of costumes and scenery for the stage, and Thomas Archer a squire and Grand Tourist. Their aristocratic clients shared the same cultivated interests, such as the great patron of Palladian architecture Richard Boyle, the Earl of Burlington, who was responsible for buildings such as Chiswick House and Burlington House. As a result, the architects of the late 17th and early 18th centuries resembled the well-spoken art consultants of today, helping a billionaire find the right pictures to fill his walls. Vanbrugh moved easily through this world, well-served by his mix of charm, creativity and self-confidence, as well as his relentless networking and opportunistic streak.

This talent for social climbing has since harmed Vanbrugh’s status. Similarly, his colourful career has earned him the reputation of a dilettante. For some critics, Vanbrugh’s background as a playwright and theatrical impresario proves the superficial flamboyance of his buildings. Ever since Kerry Downes’ books in the 1970s and 1980s, the orthodox view was that Vanbrugh’s assistant Hawksmoor did the serious work on both Castle Howard and Blenheim Palace. However, as Saumarez Smith argues in his new biography, this ignores the quality and creativity of Vanbrugh’s designs.

The author has also curated an exhibition of those drawings at Sir John Soane’s Museum in London. They seek to overturn Vanbrugh’s reputation as an architectural showman by proving beyond doubt that he was a brilliant draughtsman. The sketches reveal his ease at freehand composition, his ability to translate ideas into images, and his relaxed approach to architectural precedents. Though the influence of Wren’s churches and Chatsworth House can be seen in the ornament of Castle Howard, the layout of the building is much closer to the 17th-century palaces constructed on the continent.

What these sketches illustrate most obviously is Vanbrugh’s talent for staging. Baroque architecture has a fondness for spectacle, including optical illusions and structural sleight of hand. This can be seen in the trompe l’oeil painting that turns a flat expanse of ceiling into a seven-storey heaven, or the hidden windows that pierce the dim side chapels of a church with beams of divine light. And, as Vanbrugh’s designs for Castle Howard make clear, he used force perspective to give the building a dramatic sense of scale.

The house was approached through a range of shallow hills, delaying the moment when the façade was revealed. However, the addition of a dome to the corps de logis – popular in churches, but unused for domestic buildings at the time – means the outline can be glimpsed from much farther away. Furthermore, the house was designed so that most of the rooms were visible from the outside, adding to its impression of size. Coming closer, the tapered wings and U-shaped courtyard funnel the eye towards the central block, which is set upon a podium and enclosed in giant orders, increasing the overall proportions.

Though the composition of Castle Howard has been called picturesque, this does not capture Vanbrugh’s careful choreography of visual effects. Furthermore, to dismiss the house as theatrical ignores the effort its architect made in managing perspective. Vanbrugh thought about buildings in terms of abstract shape and visual effect, which is evident in the movement he created through advanced and recessed wings, as well as his sculptural use of masonry. It is also evident in his smaller projects: the temples, belvederes and pyramids that decorate the parkland, as well as the two houses the architect designed for himself.

Sketches for these houses also form part of the exhibition. One of them, Goose-Pie House, was constructed from brick and stone rescued from the remains of Whitehall Palace. Constructed from an odd assortment of features – rusticated bays and quoins, projecting classical pavilions – it resembles a piece of scenery. Another was Vanbrugh Castle, a mock-medieval seat on the eastern edge of Greenwich Park, which included battlements, a keep and flanking towers, and predated Strawberry Hill (often called the first Gothic Revival house in the country) by 30 years. As Saumarez Smith explains: ‘He wanted to make a big impact on a small scale. This is the essence of theatre.’

The exhibition is part of a programme of events marking the tercentenary of Vanbrugh’s death. Over the course of the year, there are lectures and shows taking place at half a dozen houses associated with the architect: Blenheim Palace and Castle Howard, Grimsthorpe and Kimbolton castles, Seaton Delaval Hall and Stowe House. Organised by the Georgian Group, the events not only attempt to inform the public about this remarkable architect, but also – judging by the colourful website accompanying the anniversary – try to turn him into a national treasure. Perhaps all this effort contains the unspoken admission that Vanbrugh’s monumental designs are admired without being loved.

This is certainly true for Blenheim Palace. The great Oxfordshire mansion that was a gift from a grateful nation, after the Duke of Marlborough defeated the armies of Louis XIV. As a result, Vanbrugh’s design combined the scale of a ducal palace with the solidity of a triumphal monument. The house boasts almost twice the number of rooms as Castle Howard (300 according to some estimates), with internal courtyards resulting in a double-depth corps de logis. Furthermore, to celebrate the Duke’s military might, Vanbrugh massed stonework along the base and roof-like fortifications, while giving the central block four square-based corner towers to suggest the layout of a castle. He even decorated the building with stone grenades, upended fleur-de-lis and a 30-tonne bust of the vanquished Louis XIV, looted from Tournai during the duke’s campaigns in the Low Countries.

During the mid-17th century, the building of great seats had all but ceased in Britain, as the disruption of the Civil Wars brought an end to an impressive run of Tudor prodigy houses and Jacobean mansions. Although work resumed with the Restoration, it was not until the end of the century that country seats began to rival the grand chateaux and schlosses of Europe. Blenheim is the most colossal of these constructions, with interiors designed for ceremonial occasions. Its state rooms were arranged along enfilades, creating extended internal vistas for a processional approach to the painted saloon, where the duke would sit enthroned, overlooking a park filled with triumphal arches and victory columns.

The Duchess of Marlborough never liked the design, blaming Vanbrugh for the building’s ballooning costs. After the duke’s death in 1722, she took over the project, soon forcing Vanbrugh to abandon the building site, and then employing Hawksmoor to complete the work. The duchess was not Vanbrugh’s only detractor, and, once finished, Blenheim’s size and spectacle meant it was widely ridiculed. In one letter, Alexander Pope wrote: ‘I never saw a great thing with so much littleness in it’, also quoting the Duke of Shrewsbury, who described it as ‘a great Quarry of Stones above ground’.

Several generations of the Churchill family have complained about living at Blenheim. As well as the enormous cost of running the house, every comfort and convenience has been sacrificed to magnificence. This is even more true of the architect’s final commission: Seaton Delaval Hall in Northumberland, which was constructed for Admiral George Delaval, who made a fortune capturing prize ships during the War of Spanish Succession. By this point, Vanbrugh had stopped working with Hawksmoor, and the forbidding majesty of this building is perhaps the purest expression of his style. But in 1822 the central block was badly damaged by fire, and the fact that no member of the Delaval family has since tried to occupy the main house shows how inhospitable the original is.

Saumarez Smith calls the hall ‘a classical fortress far removed from the prissy conventions of contemporary Palladian design’. However, by this point the English Baroque was falling out of fashion, associated with the absolutist power and Catholic excess rejected by the Whig ascendancy. Now, aristocratic landowners favoured those prissy Palladian conventions: Wanstead House was commissioned in 1715, Houghton Hall in 1722, and Chiswick House in 1726. Meanwhile, Colen Campbell’s Vitruvius Britannicus – the textbook of graceful neoclassicism – was published in 1715.

Vanbrugh did not live long enough to see Seaton Delaval finished, dying in 1726 at the age of 62. English Baroque, the style he pioneered, had lasted little more than half a century. But its influence would endure for much longer: Wren’s domestic architecture was revived in the Neo-Georgian public buildings of the 1930s and 1950s, while Hawksmoor’s churches were praised by the mid-century Modernists for their monumental stonework and geometric simplicity.

In contrast, though Vanbrugh had many admirers, his imitators are much harder to find. The neoclassical architects Robert and James Adam praised his ‘novelty and ingenuity’, the painter Joshua Reynolds admired his ‘originality of invention’, and John Soane encouraged his pupils to study ‘the picturesque effects of his works’. Yet the architecture of the Georgian era, even at its grandest scale, had a politeness that was absent from Vanbrugh’s designs. In fact, there is more evidence of his legacy in the eclectic excess of Victorian Revivalism or the Edwardian Grand Manner.

By the 19th century, Charles Barry was praising the varied outlines of Vanbrugh’s buildings. Come the 20th century, Edwin Lutyens was complimenting his control of scale and mass. Both these architects designed buildings on the largest scale for ceremonial occasions, whether Barry’s Houses of Parliament or Lutyens’ New Delhi. These are Vanbrugh’s true disciples: not the architects of country houses, but the creators of public spaces such as museums, cathedrals, and centres of political power.

For Saumarez Smith, Vanbrugh’s real legacy was in the playful postmodern architects of the late 20th^t century. Robert Venturi, who with his wife Denise Scott Brown designed the Sainsbury Wing of the National Gallery, has celebrated Vanbrugh’s ‘messy vitality over obvious unity’. Unlike the puritanism of Modernist architecture, Vanbrugh’s buildings communicate the full range of emotions, from brooding melancholy to triumphant celebration. At the same time, Vanbrugh remains sui generis, his designs both rooted in history and full of reckless experiments, both carefully constructed and visually extravagant. As the new biography concludes, they prove that ‘buildings can be theatre’.