A Self-Training Approach to Domain Document-Level Relation Extraction with Large Language Models

• 1、the School of Cyberspace Security, Beijing University of Posts and Telecommunications, Beijing 100876

Abstract：For document-level information extraction in low-resource domains, dispersed cross-sentence evidence in long documents, noisy candidate entity pairs, and scarce annotations often jointly constrain recall. This paper proposes an LLM self-training algorithm based on data augmentation. It first performs supervised fine-tuning with a small amount of labeled data to align task definitions and structured outputs; then, guided by probabilistic re-sampling and rule-based verification, it uses an LLM to generate and filter high-confidence pseudo-labeled data to enrich long-tail relation patterns; finally, it introduces reward-based preference alignment, and suppresses stochastic fluctuations via a relation-similarity-set reward, enabling stable decision-making under noise and long contexts. Experiments on DocRED and Re-DocRED validate the effectiveness of the proposed method, and results on a cybersecurity AZERG dataset show higher recall and more robust structured extraction, indicating that this paradigm is well suited for information-dense yet annotation-scarce domain document extraction scenarios.

Keywords： document-level relation extraction few-shot learning self-augmented data generation preference alignment STIX threat intelligence