(CNN)-- In Hollywood movies, they're often portrayed as danger-dodging men with dark glasses, smoothly working behind the scenes to protect the president at any cost.
But a group of Secret Service members drew worldwide attention over the weekend for a different reason -- accusations of misconduct involving prostitutes.
The incident -- which allegedly occurred when agents and officers brought prostitutes to a hotel in Colombia -- violates strict behavior rules in an agency that aims to stay out of the spotlight, and usually succeeds, experts and officials said.
"The main problem here for the Secret Service agents is not so much a criminal violation, but really it is a dereliction of duty, not doing their job," said U.S. Rep. Peter King, who chairs the House Homeland Security Committee.
Security clearances yanked
While soliciting prostitution is legal in some parts of Colombia, it's a breach of the agency's conduct code, U.S. government sources said. High-level officials in the Secret Service and the Department of Homeland Security were outraged over the incident, the sources said, noting that the investigation indicated the prostitutes were brought to a hotel that had been secured for the Summit of the Americas.
Dark side of secret service scandal
Colombia scandal plagues Secret Service
Secret Service agents relieved of duty
Obama on Secret Service allegations
But officials and analysts disagreed about how great a threat the alleged misconduct posed -- and how common it could be.
"This is really the biggest scandal in the history of the Secret Service," said Ron Kessler, a former Washington Post reporter who has written a book about the agency.
Dan Emmett, a former Secret Service agent, said such allegations have never surfaced before, arguing that this is "a very isolated type of incident."
"The president, since the beginning of the presidential protective division (of the Secret Service) in 1902, has been on hundreds and hundreds of overseas trips. Now, this is the first time anything like this has come up. ... In my 21-year career, this was certainly not the norm, and I think this is certainly out of the ordinary," Emmett said.
President demands 'rigorous' investigation
Authorities have released few details about the allegations.
There was a dispute over payment between at least one Secret Service member and a woman brought to his hotel, U.S. government sources familiar with the investigation have said. At least one of the women brought to the hotel talked with police, and complaints were filed with the U.S. Embassy, the sources said.
U.S. government sources said the agents and officers under investigation were not part of the president's personal protective detail, and stressed that there was no threat to the president.
Such details indicate that the Secret Service agents allegedly involved would not have been "in close proximity to the president," Emmett said.
"A presidential advance team that goes overseas is literally a cast of hundreds," he said. "You have the people who are going to be doing the various sites, the motorcade routes and so on. Most of these people are not in direct access to the president."
Emmett dismissed speculation that the Secret Service agents and officers allegedly involved could have been easy targets for blackmail.
"In the world of espionage and foreign intelligence services, anything is possible. However, I just don't see it in this case," he said. "These particular individuals, I don't believe they would have been targeted. ... They just simply don't have the information that would be valuable."
But King, who was briefed on the investigation, said there were still security concerns.
"Their job is to protect the president of the United States. They can't put themselves in a compromising position where they could be blackmailed or threatened," he said. "Nor should they bring prostitutes into a security zone 48 hours before the president of the United States is arriving."
Lawmakers say they're troubled by scandal
Kessler, who first broke the story to the Washington Post but did not reveal his sources, said some of the Secret Service agents involved are married and could have been easy blackmail targets.
"As a result, they could have let terrorists into secure areas and that could have resulted in assassination," he said.
U.S. Rep. Darrell Issa told CBS's "Face the Nation" that he wasn't satisfied with repeated reassurances that the president wasn't in danger.
"The investigation ... will be about how did this happen and how often has this happened before. Things like this don't happen once if they didn't happen before," said the California congressman, who chairs the House Oversight Committee.
Secret Service members sent home after incident
The investigation into the allegations has stretched beyond the Secret Service.
The U.S. military has said five U.S. troops who were working with the Secret Service are under investigation for missing curfew and alleged misconduct at the Colombian hotel where Secret Service agents are said to have brought prostitutes. On Monday, Pentagon spokesman George Little said the number of troops involved "may be more than five."
Speaking to reporters in Colombia Sunday, U.S. President Barack Obama said he would wait for the investigation's results before passing judgment.
"I expect that investigation to be thorough and I expect it to be rigorous. If it turns out that some of the allegations that have been made in the press are confirmed, then of course I'll be angry," he said.