The present research assessed if 1,096 psychologist participants sufficiently updated their beliefs in psychological effects when presented the results of multi-lab replication studies. In Phase I, participants read summaries of results from studies scheduled for replication attempts. For each study, participants made estimates of the population effect size and probability that the population effect was greater than d = .1 (i.e., non-trivial). During Phase I, participants were randomly assigned to a control or prediction condition with the only substantial difference being that those in the prediction condition were informed of the methodology for replication studies (not the results) of the original effects they evaluated and asked to predict how their confidence in the effect would change given various hypothetical replication study results. Approximately 1 to 1.5 years later, participants completed Phase II—the questions were the same for participants in the control and prediction conditions—in which they read summaries of replication results and provided revised effect size estimates and revised probability estimates. Participants’ prior beliefs in original effects and replication evidence were quantified with Bayesian models which allowed us to model how a perfectly rational Bayesian agent would update their beliefs in original effects after incorporating replication evidence. While participants did update their beliefs substantially in the direction consistent with the replication evidence, as predicted, participants’ confidence updates were insufficient for the weight of new evidence regardless of if confidence in psychological effects should have increased or decreased. Results suggest an impediment to scientific self-correction as it seems that psychologists underutilize replication evidence when updating their beliefs.