The isospin violating dark matter (IVDM) scenario offers an interesting possibility to reconcile conflicting results among direct dark matter search experiments for a mass range around 10 GeV. We consider two simple renormalizable IVDM models with a complex scalar dark matter and a Dirac fermion dark matter, respectively, whose stability is ensured by the conservation of "dark matter number." Although both models successfully work as the IVDM scenario with destructive interference between effective couplings to proton and neutron, the dark matter annihilation cross section is found to exceed the cosmological/astrophysical upper bounds. Then, we propose a simple scenario to reconcile the IVDM scenario with the cosmological/astrophysical bounds, namely, the IVDM being asymmetric. We also discuss collider experimental constraints on the models and an implication to Higgs boson physics.