Both the direction and the strength of the exchange bias field Heb of an FeNi/FeMn double layer are manipulated locally by He ion irradiation in an applied magnetic field. The magnitude of Heb is enhanced over its initial value in the low ion dose regime. Above a threshold irradiation dose a reduction and eventually a suppression of Heb is found. The direction of Heb is initialized by the field direction during irradiation. These observations are discussed considering the structural modifications caused by the ion bombardment. The data are well described by a model, which is based on the competition between the enhancement of the exchange bias field strength due to defect creation in the antiferromagnetic layer and the decrease induced by intermixing at the ferromagnet/antiferromagnet interface.