Anomaly rituals cancelled because target pawn is "out of position"

For some reason the rituals started to fail with a message "Chronophagy ritual has been cancelled because [target pawn] is out of position".

If target is a prisoner, then this message appears right away and the pawns don't even try to fetch the victim. If I put the ritual spot in the same room as the prisoner, invoker puts prisoner in the circle but targets starts to wander around and colonists just stand there until I cancel the ritual.

If the target is a colonist, they gather as normal but a second after the ritual actually starts it is cancelled.

Any ideas on why this is happening? Game is modded, but I don't think I added mods that would affect AI or Anomaly content.