Disordered solids that have undergone a plastic deformation in one direction, "remember" that direction also when relaxed back to the state of zero-stress. This mechanical memory effect is well visible in the so-called Bauschinger effect: materials that have experienced a plastic strain exhibit a softer stress response under reverse loading as compared with reloading in the same direction. The origin of the Bauschinger effect in solid-like polymers remains elusive and it is not clear what is the order parameter that is responsible for this memory. We perform large-scale molecular dynamic simulations of glassy and semicrystalline polymers of a coarse-grained model to shed light into the structural origin of the mechanical memory. We find that the degree of chain orientation and stretch are correlated to the cumulative strain provide a good measure of mechanical memory in solid-like polymers.