Kate and Oliver Hudson Don't Watch Each Other's Work: 'You've Seen One … You've Seen 'Em All'

Kate, 42, and Oliver, 45, stopped by The Late Show with Stephen Colbert Thursday, where they admitted they don't watch each other's film

Oliver — who stars in FOX's The Cleaning Lady as FBI agent Garrett Miller

immediately replied that his sister does not watch his new show, and told Kate, "Just say no."

"I did," she said. "I watched the first one … I haven't seen the second episode

The Cleaning Lady premiered Jan. 3 and has aired four episodes so far. The series also stars Elodie Yung and Adan Canto.

Oliver had no hard feelings about his sister's viewing habits, telling Colbert, "I'm gonna let her off the hook here