Color-kinematics duality has been shown to be present in QCD scattering amplitudes with massive quarks. More generally, the duality was extended to gauge theories with flavored massive matter in any Lie-algebra representation, with the resulting double copy for gravitational amplitudes. In this talk, I will discuss the details of the extension, including a new decomposition of QCD amplitudes into independent color-ordered primitives, and how the duality imposes new relations between the primitives. Furthermore, I will discuss how the extension of the duality can be used to compute gravitational amplitudes with matter and how this can be used to obtain amplitudes in pure gravity theories.