The Chandra Deep Field South (CDF-S) 1 Ms exposure produced a catalog of 346 X-ray sources, of which 59 were not visible on the Very Large Telescope ( VLT)/FORS1 and the ESO-MPI/WFI deep R-band images to a limit of R-vega=26.1-26.7 mag. Using the first release of the ESO VLT/ISAAC JHK(s) data on the CDF-S, we identified six of the 12 such objects that were within the coverage of these IR observations. The VLT/FORS1 I-band data further confirm that five of these six objects are undetected in the optical. The photometric properties of these six counterparts are compared against those of the optically brighter counterparts of Chandra sources in the same field. We found that the location of these optically brighter Chandra sources in the near-IR color space was bifurcated, with the color of one branch being consistent with that of E/S0 galaxies at 0less than or equal tozless than or equal to1.5, and the other branch being consistent with that of unreddened active galactic nuclei/quasi-stellar objects (AGNs/QSOs) at 0less than or equal tozless than or equal to3.5. The six counterparts that we identified seemed to lie on the E/S0 branch and its extension, suggesting that these X-ray source hosts are mostly luminous E/S0 galaxies (M-V similar to -20 mag in the AB system) at 1less than or equal tozless than or equal to2.5. On the other hand, some of them can also be explained by AGNs/QSOs over a wide redshift range (0less than or equal tozless than or equal to5), if a range of internal extinction (A(V)=0-1 mag) is allowed. However, the later interpretation requires fine-tuning extinction together with redshift for these objects individually. If they are indeed AGNs/QSOs, the most luminous of them is just barely qualified for being a QSO. Finally, we point out that neither high-redshift (z>5) star-forming galaxies nor irregular galaxies at lower redshift can be a viable explanation to the nature of these six counterparts.