NEW YORK (CNNMoney) -- The IRS identified roughly two million tax returns that were potentially fraudulent last year, a sharp increase that has the agency struggling to keep up.
The agency's fraud detection system identified more than one million returns as being potentially fraudulent -- up 72% from the previous year -- according to a mid-year report to Congress from the Taxpayer Advocate Service, the taxpayer watchdog arm of the IRS.
And almost another one million returns were identified as being part of a scheme dubbed "Operation Mass Mail," where tax cheats submitted fraudulent returns that were automatically voided before they even made it to the processing stage.
The increase in tax fraud can be attributed to a number of factors, including more electronic filing, higher dollar amounts of refundable credits and a growing number of people using debit cards to receive refunds, said TAS spokesman Kenneth Drexler. However, at the same time, he said many of these factors also serve an important purpose for taxpayers.
The IRS has also warned of tax preparers who file improper forms and claimed false refunds for their clients. And identity theft, where someone steals the Social Security number of another person to get an undeserved refund, was a big part of the overall increase in fraud cases -- with the IRS reporting more than 450,000 identity theft cases.
One ongoing cause of identity theft is that the Social Security Administration's Death Master File, a database of deceased individuals that includes personally identifying information like full names, Social Security numbers and addresses, is publicly available for identity thieves to steal information from, the TAS said. The watchdog therefore recommended that the Social Security Administration restrict access to the database.
'Unprecedented backlogs': As a result of the sharp increase in fraud and identity theft last year, the agency was simply unable to keep up with the demands in the latest tax season.
"The IRS is experiencing unprecedented backlogs in return processing because of identity theft and other refund fraud, is instituting 'hard freezes' on questionable returns because it cannot timely address them and is unable to answer anywhere from an average of about 30 percent [of calls] overall in fiscal year 2012 to a low of 65 percent of calls [for one customer service department]," TAS said in its report.
That meant that at busy times during the filing season, the IRS was only answering one out of every nine calls it received from taxpayers whose refunds were being held. And people only got through to a representative after waiting an average of more than an hour.
In addition, the IRS not only put a hold on returns that ended up being fraudulent, but also on "tens of thousands" of legitimate returns, which often led taxpayers to spend "many months" going back and forth with the IRS before receiving their refunds. Some of those legitimate returns likely belonged to identity theft victims, since the IRS often puts a victim's, as well as the criminal's, returns in limbo as it investigates the fraud.
"As the IRS develops [its] filters, it must also create procedures that would allow honest taxpayers with legitimate refund claims to receive their money without unnecessary delay," TAS said in its report.
But TAS acknowledged that the IRS is under competing pressures. While it needs to get refunds issued to qualifying taxpayers as quickly as possible, it is also required by Congress to stop payments on potentially fraudulent refund claims.
"[T]he IRS must somehow prevent refunds from going out on over two million bogus claims -- and at the same time quickly process returns and issue refunds to the over 145 million individual taxpayers who file legitimate claims," TAS said in its report.
More delays to come?: TAS said the agency's shrinking resources and growing responsibilities are making it next to impossible to prevent these delays and backlogs.
While the IRS had pushed last year to increase its 2012 budget by $1 billion, the 2012 budget that was ultimately passed by Congress cut the IRS's budget by about 3%, to $11.8 billion.
The report also said that last-minute changes to tax laws only added to the delays of millions of taxpayers returns last year and led many taxpayers to underclaim deductions and credits because the laws were uncertain when they filed their returns. Others ended up owing taxes or penalties because late-year changes increased the amount they owed the IRS and they didn't pay a sufficient amount.
The TAS said it is concerned that the same uncertainty this year will impact the upcoming filing season as well, especially if Congress doesn't extend many tax provisions that are scheduled to expire at the end of the year.
Another area TAS raised concerns about for the upcoming tax season is the IRS's shift to automated electronic services, which do things like automatically identify suspicious refund claims, instead of requiring a representative to do it manually.
The watchdog said most taxpayers "will never receive a call from the IRS before the IRS proposes an assessment of tax or levies on their assets or incomes." Instead, correspondence audits, which involve a request for information in writing through the mail, are the main method of communicating with taxpayers.
In the coming year, TAS also plans to examine what kind of impact the IRS's Offshore Voluntary Disclosure Programs -- which lure in tax evaders with the promise of reduced penalties and no jail time -- are having on tax collection and it will look into the impact of the agency's "fresh start" initiative, which relaxes filing rules for taxpayers struggling financially.