A former Conservative MP who lost his seat at last year’s general election has been accused of still "parading around as an MP" by the leader of the House of Commons.