Even though Diddy was just spotted last week taking a family vacation with Kim Porter and their three children together, it seems he still has a soft spot