And Palmer, who plays his club rugby in Paris with Stade Francais, is expecting a backlash from Les Bleus when they meet at Twickenham next Saturday.
"Everyone hates the English and everyone raises their game when we come up against them. It is the same when we go to Cardiff, Dublin, Edinburgh, even the Australians and Kiwis don't like us," said Palmer.
"That's just the way it is - it's all part of our colonial history and our empire. England also have the biggest union in the world and it's always good to beat the big boys, isn't it?"
France fell to a record 59-16 defeat to Australia in the autumn despite leading after half time.
And after a season and half in the Top 14 this French inconsistency comes as no surprise to Palmer.
"It can be quite strange at times to see players play so well one day and so poorly the next," he said.
"They are much more up for their home games than away games, but I'm not suggesting that's going to be an issue next week. I'm not sure why it happens, but I think it might be a cultural thing.
"Prestige plays a big part in French rugby, I feel. There's a lot of money spent on things like the after-match functions, which are so much better than the functions after English games.
"That is all related much more to French club rugby, though, and we saw with the international side last weekend that they gave a good performance against the Irish in a difficult place to play.
"We know we will have a tough game against them. They are the Six Nations champions and they won a Grand Slam last year."