Baby and Grandpa had a great time together, they played there was no crying, Baby was fine.
So, maybe Baby has just decided he doesn't like Grandma. Then I realize, no, it is not Grandma, it is just that Baby has separation anxiety about Husband. The day Grandma baby-sat, Husband & I went out together, this was the first time since Baby was born. The entire week after Grandma baby-sat, Baby was super-clingy with Husband in the morning when he left for work and at night when he got home.
When Grandpa was here, Husband had left for work hours earlier, so it was just me leaving and there were no problems. Now, I have been away from Baby previously, but Baby was always with Husband when I wasn't there (except for the once when my Brother watched him and I think that bad experience was because Baby was nursing every 10 minutes at that time and we hadn't found a bottle that he would take yet). The day after Grandpa baby-sat, I went out and Baby was home with Husband - No Problems.
So, I have concluded that either Grandpa is the best or the Baby only cares about Husband.