Rubin, 70, will continue to serve as a director until his term expires at the next annual meeting in the spring, Citigroup said.
Citigroup has been one of the hardest hit banks by the housing market downturn due to its heavy investments in mortgages and other types of loans. Analysts believe Citigroup will report a fifth straight quarterly loss when it releases fourth-quarter results later this month.
Rubin had been slowly paring back his role Citigroup for months, after chairman of the bank for about a month after it ousted former chairman and CEO Charles Prince in November 2007. Win Bischoff became Citi's chairman in December 2007, and investment banking head Vikram Pandit became CEO.
"This is not a decision that I have come to lightly," Rubin said in a letter released by the bank. "But as I enter my 70s and with all that is now in place at Citi, I believe the time has come for me to make these changes."
He also wrote: "My great regret is that I and so many of us who have been involved in this industry for so long did not recognize the serious possibility of the extreme circumstances that the financial system faces today."
In August 2008, Rubin gave up his title as head of the board's executive committee, and became a "senior counselor" instead.
Citigroup shares fell 38 cents, or 5.3 percent, to $6.78 in late afternoon trading.