Diddy has one less lawyer fighting for him heading into his federal trial, because his legal team couldn’t get on the same page regarding the best defense for him … TMZ has learned.

Sources with direct knowledge of Anthony Ricco‘s exit from Diddy’s camp tell us there had been a power struggle going on between lead attorney Marc Agnifilo and him for several months, and it finally came to a head.

As we reported, Ricco shook up Diddy’s team Friday morning by informing the judge he could no longer “effectively” defend his client — but his filing lacked detail about why he was jumping ship.

For his part, Agnifilo has repeatedly denied the trafficking charge, and instead says the so-called freak offs always involved consensual sex between Diddy, a girlfriend and a third party. As we reported, Agnifilo filed a motion to dismiss count 3 of the indictment, which is all about the Mann Act … the transportation of individuals to “engage in prostitution.”

Our sources say Ricco grew frustrated when his efforts to speak directly to Diddy about legal strategies were thwarted by co-counsel, and that’s why he felt he couldn’t properly defend the hip hop mogul.

We’re also told Ricco strongly believed they wouldn’t be ready to go to trial by May 5 — the scheduled start date — and thought they should ask the judge for more time. His co-counsel strongly disagreed, push came to shove and Ricco peace’d out.

Our sources connected to Diddy told us Ricco’s departure is much ado about nothing, saying lead defense attorneys Agnifilo and Teny Geragos have been rock solid for almost a year now.

We reached out to Diddy’s reps for comment, but no word back yet.