'But if the manslayer at any time goes beyond the border of his city of refuge to which he may flee,