driven into the wilderness
Were they not, rather, exterminated? So it is signified in the Testament of Egderus, though no independent verification of this genocide has ever been made (or likely ever can be made, given the fragmentary condition of the evidence).
But is it not suggested here — however tenuously — that the Remnant may have once been partially assimilated into the larger society ruled by the (or a) Golias, and then, under some unspecified political or social pressure, purged from the general population?