Thousands of people were fired from the IRS this week in the thick of tax season.