Recent Chandra studies of low-mass X-ray binaries (LMXBs) within early-type galaxies have found that LMXBs are commonly located within globular clusters of the galaxies. However, whether all LMXBs are formed within globular clusters has remained an open question. If all LMXBs formed within globular clusters, the summed X-ray luminosity of the LMXBs in a galaxy should be directly proportional to the number of globular clusters in the galaxy regardless of where the LMXBs currently reside. We have compared these two quantities over the same angular area for a sample of 12 elliptical and S0 galaxies observed with Chandra and found that the correlation between the two quantities is weaker than expected if all LMXBs formed within globular clusters. This indicates that a significant number of the LMXBs were formed in the field and naturally accounts for the spread in field-tocluster fractions of LMXBs from galaxy to galaxy. We also find that the ‘‘pollution’’ of globular cluster LMXBs into the field has been minimal within elliptical galaxies, but there is evidence that roughly half of the LMXBs originally in the globular clusters of S0 galaxies in our sample have escaped into the field. This is likely due to higher globular cluster disruption rates in S0s, resulting from stronger gravitational shocks caused by the passage of globular clusters through the disks of S0 galaxies that are absent in elliptical galaxies.