"Ashley said, 'I have not been in front of a camera since I was 17, and I don't feel comfortable acting,' " executive producer Bob Boyett tells PEOPLE. "Mary-Kate said, It would have to be me because Ash doesn't want to do it. But the timing is so bad for us."

Production conflicted with the 29-year-old twins' schedule as designers of The Row and Elizabeth and James, Boyett adds. The show does play upon their real-life career, explaining that Michelle is absent from the first episode's family reunion because she's busy running her fashion empire in New York.

Despite some initial disappointment, the cast tells PEOPLE they understand the twins' reluctance to join the show. (PEOPLE's requests for comment from the Olsens were not returned.)

"Lori [Loughlin] was talking to me about the twins and said, 'They just won a CFDA award.' And I'm like, 'Yeah, that's like winning an Oscar.' It was them and Tom Ford. I get where they're at," John Stamos tells PEOPLE.

There's also geography (the Los Angeles-based cast has stayed very close over the years; the Olsens are in New York City much more often) and the age difference.

"They're family members," Dave Coulier tells PEOPLE. "But I think they have a different perspective. When we get together and we reminisce, they were babies [at the time]. So they don't share those same memories. I can't really speak for them but I would understand why they don't have the same sentiments that we all do."