“If people don’t like it, that’s fine, certainly my tastes are not for everyone,” Sarah Daugherty tells PEOPLE ...