They headed to New Zealand having won the RBS 6 Nations earlier in the year and all their group games in the Land of the Long White Cloud but came unstuck against Les Bleus.
Johnson's contract as head coach is due to expire in November but RFU operations director Andrew admitted it will take nearly two months to settle on what to do next.
"It is very disappointing that it has happened. The key thing is that we take the next four to six weeks with Martin and review what has gone on," said Andrew.
"Martin will reflect on what has happened and why we have gone out in the quarter-final of a World Cup. The important thing now is to take stock.
"We said right from the start that we would take this right through to this World Cup with Martin in charge. We have done that and now we will sit down with Martin and look at what has happened.
"It is my job to look at the review and take it forward and at some point in the next few weeks reflect and then make a decision. We will do that as quickly and as openly possible as we can."