NbO2 and VO2 both exhibit similar electrostructural phase transitions: a structural phase transition from higher symmetry rutile phase into lower symmetry distorted rutile phase, and metal insulator transition. Because of this, they have various potential applications in optoelectronic devices, temperature sensors, smart windows, and memory devices. However, the mechanism of the transition in each material is not fully understood, and previous attempts to create a unified pictured of both materials has led to surprises. For example, the electronic transition is suppressed when the materials are alloyed (into compounds of the formula NbxV1-xO2), which is unusual in analogous systems. This body of research aims to unravel the unexplained aspects of the structural instability in such systems with the help of single crystal x-ray diffraction, diffuse scattering, and three-dimensional difference pair distribution function. T across the NbxV1-xO2 phase diagram disagree with the currently accepted model, where the transfer electrons. The observations are instead more consistent with the geometric frustration of displacements model recently used to explain the suppression of long-range structural order in the V1-xMoxO2 phase diagram. Two separate short-range ordered phases are observed in the NbxV1-xO2 phase diagram, one is 2D-M2 phase in lower Nb composition, and another is unknown 2D phase in higher Nb composition whose type of local ordering is not known at this moment. The structural instability due to oxygen defects is also studied for substoichiometric NbO2-§. The role of oxygen vacancies in both the stabilization of the β-phase and its crystallographic intergrowth with the α-phase in NbO2-δ single crystals is investigated using total x-ray scattering, electron diffraction, and transmission electron microscopy. We have found the existence of β-phase and mixed phase of α-domain and β-domain based on the nominal oxygen composition. In the proposed model, the α and β phases are both compromised structures against the competition of forces from the geometric frustration. Despite their structural similarities and close energies, intergrowth between the two is energetically unfavored by mismatches in atom displacements. The appearance of 2D nets of diffuse scatterings in these crystals is interpreted in terms of two-dimensional defects of stacking fault created by crystallographic shear in rutile (110) direction. Additionally, the correct space group symmetry for β-phase is shown to be I41/a instead of previously reported I41. The model proposed in this work is a step toward explaining the origin of the unique structure of NbO2 in both its alpha and beta forms, which has never been fully explained.