You are right that there must be a change in dipole moment for the stretch to be IR active. To determine whether or not there will be a change in dipole moment, examine the bond that's stretching and look at the two fragments that it's connected to. If these two fragments are equivalent, there will be no net change in dipole moment for the symmetrical stretch and it will be IR-inactive.
For example, the carbon-carbon triple bond in acetylene does NOT have an IR active stretch, just as the nitrogen-nitrogen triple bond in free dinitrogen does NOT have an an IR active stretch.
Another example (with sample spectra) is 2,3-dimethyl-2-butene. Both sp2 carbons have the exact same substituents and the C=C stretch produces no change in dipole moment. The IR spectrum of this molecule has no C=C stretch in the 1600 cm-1 region. AIST:RIO-DB Spectral Database for Organic Compounds,SDBS
However, 2,3-dimethyl-1-butene does not have identical sp2 carbons. This stretch does produce a change in dipole moment and there is a sharp absorption at 1646 cm-1. AIST:RIO-DB Spectral Database for Organic Compounds,SDBS
I hope this answers your question!