Online sales put in their worst monthly performance in nearly five years last month as consumers watched the Olympics on television rather than shop on the internet.
Non-store retailing sales volumes, which also includes mail order, fell by 6.7% between July and August, the biggest drop since December 2007, the Office for National Statistics (ONS) said.
But the London 2012 Games had a negligible overall effect, the ONS added, as total retail sales volumes dipped 0.2% in August, following a 0.3% rise in July.
The figures echo survey evidence from the British Retail Consortium and CBI earlier this month, which also pointed to a minimal impact from the Olympics.
Samuel Tombs, UK economist at Capital Economics, said the figures "add to the evidence that the Olympics did not provide the hoped-for boost to spending on the high street".
But Mr Tombs said that even after the drop in August, retail sales volumes are still 0.6% higher than three months ago and are likely to contribute to a brief return of GDP growth in the third quarter.
He added: "With consumer confidence still very weak and inflation set to outpace earnings growth for another six months or so, we would not be surprised to see further falls in retail sales in the coming months."
Suit specialist Moss Bros is among retailers that have blamed Britain's summer of major events for slow business. The retailer said it had missed out on nearly £2 million of hire business due to events such as the Queen's Diamond Jubilee and the Olympic Games as more weddings were deferred.
There was some positive impact from the London 2012 Games at sporting goods stores, which helped sales volumes in the "other stores" category jump 1.5% month-on-month. But this was offset by a 0.7% fall in department stores and a 2.7% fell in household good stores between July and August.
Food stores saw a slight 0.2% rise in sales volumes in the period, which some retailers put down to improved weather and the Olympics.