Judge Mathis is weighing in on Kim Kardashian and Kris Jenner‘s defamation lawsuit against Ray J, saying the sex tape — and all the money they made off it — makes it tough to sue him now.

We caught up with Judge Mathis in Wednesday in NYC, and he thinks Kim’s gonna have a tough time building a case against the guy who — as the Judge sees it — helped her launch her career.

You gotta see the full clip … he claims KK would be better off trying to get a gag order or a cease-and-desist — but he doesn’t see a defamation case.

As we reported, Kim and Kris’ lawsuit — filed by Alex Spiro — slams Ray J for falsely claiming they’re the subject of a federal criminal RICO investigation … saying Ray is “unable to accept the end of his fleeting relationship” with Kim over 2 decades later.

While Ray has yet to say why the feds would have any business going after the Kardashians … we know he’s beefed with them in the past over public discussions about the sex tape deal — and Mathis clearly thinks that’s what’s at the heart of their latest dispute.