Q-REAL: Towards Naturalness and Distortion Evaluation for AI-Generated Content

Quality assessment of AI-generated content is vital for modeloptimization, yet most existing evaluation datasets and models rely oncoarse-grained single scores, failing to provide targeted guidance. Tobridge this gap, we introduce Q-Real, a fine-grained quality assessmentdataset specifically designed for AI-generated images. This dataset com-prises 10K images generated by multiple models, annotated along twocritical dimensions, naturalness and distortion which are widely re-garded as the most significant aspects of AI-generated image quality.For each image, we localize major entities and provide a set of judgmentquestions and attribution descriptions along these dimensions to facili-tate comprehensive evaluation. Based on this dataset, we establish theQ-Real Bench to rigorously evaluate models on the challenging tasks ofjudgment and grounding with reasoning. And to tackle these challenges,we further propose a fine-grained training pipeline for Multimodal LargeLanguage Models (MLLMs), empowering them to judge, localize prob-lematic entities with detailed analysis, and predict quality score. Ex-perimental results demonstrate the high quality and significance of ourdataset, as well as the effectiveness of our proposed training pipeline.The dataset is available at https://huggingface.co/datasets/AGI-Eval/Q-Real.