Wales scored three tries of their own through centre James Hook, Tom Prydie and Alun Wyn Jones but South Africa stormed to victory - much to Gatland's annoyance.
"We were outstanding for 25 minutes and controlling it at 16-3 up, then we just handed them the game," said Gatland.
"Ill-discipline let them back into it and we gave them the game on a plate. South Africa just don't make mistakes.
"It is just about being accurate, not making mistakes and doing the simple things. We just went through a poor period and of course the other side will come back into the game.
"We lost three line-outs in a row, we had a couple of turnovers at the breakdown and at 16-3 we gave away a stupid penalty through Jonathan Thomas that gave them a bit of momentum.
"We also missed a couple of tackles and it was disappointing."