Mike Phillips scored the game's only try but four missed kicks proved costly and Gatland insisted his side deserved better.
He said: "We just feel like the destiny of the result was taken out of our hands with the red card.
"He's lifted him, that's a yellow card, but he's not driven him into the ground. Does that mean every time there's a tackle where you lift someone off the ground it's a red card?
"Why spoil the semi-final with a red card? He's not a dirty player. I'm just gutted. We were down to 14 but we showed great character and I'm proud of our efforts. But we can't go to just one tournament and do well, we've got to kick on from here.
"I feel let down, I thought this team were good enough to go on and take the final but it was not to be. We just feel that ultimately the result wasn't in our control."
Wales had edged the opening quarter of the game with James Hook knocking over an early penalty before Warburton lifted Clerc before dropping him to the floor.
Rolland produced the red card and Morgan Parra rubbed salt into the wounds by drawing France level with the resulting penalty.
Gatland's side continued to threaten but France took the lead after another Parra penalty from a Dan Lydiate infringement seven minutes before the break.
The French took control early in the second half with Parra adding another three points and Julien Bonnaire causing havoc in the Welsh lineout.
But Wales roared back into it through Mike Phillips who slipped a Pascal Papé tackle to dart over on the hour mark.
Stephen Jones' conversion attempt shaved the outside of the post and when Nicolas Mas was penalised for coming in from the side four minutes from time, Leigh Halfpenny's effort from the halfway line dropped agonisingly under the posts.
Wales continued to press but after an interminable passage of play France eventually turned the ball over for Dimitri Yachvili to kick the ball out and end the game.
Winger Shane Williams who is set to bow out of the international game after next Friday's third place play-off said: "Any game you play for your country, you give it your all and want to play your best. We'd love to go home with a third place.
"Coming to the end of my career it'll probably be an emotional time.
"There's no bitterness, the side who played the best today won. We knew the France team turning up today would be a good France team - Parra at 10 had a great game, he bossed things. We wish them all the best in the final.
"We're absolutely devastated, we worked hard getting through the group and it was a great performance last week."