A few months ago, rumors had been flying that an affair between The Game and Gloria Govan is what initially broke her and fiance L.A. Lakers Matt Barnes