ET-SAM: Efficient Point Prompt Prediction in SAM for Unified Scene Text Detection and Layout Analysis

Previous works based on Segment Anything Model (SAM)have achieved promising performance in unified scene text detection andlayout analysis. However, the typical reliance on pixel-level text segmen-tation for sampling thousands of foreground points as prompts leadsto unsatisfied inference latency and limited data utilization. To addressabove issues, we propose ET-SAM, an Efficient framework with twodecoders for unified scene Text detection and layout analysis based onSAM. Technically, we customize a lightweight point decoder that pro-duces word heatmaps for achieving a few foreground points, thereby elim-inating excessive point prompts and accelerating inference. Without thedependence on pixel-level segmentation, we further design a joint train-ing strategy to leverage existing data with heterogeneous text-level an-notations. Specifically, the datasets with multi-level, word-level only, andline-level only annotations are combined in parallel as a unified trainingset. For these datasets, we introduce three corresponding sets of learn-able task prompts in both the point decoder and hierarchical mask de-coder to mitigate discrepancies across datasets. Extensive experimentsdemonstrate that, compared to the previous SAM-based architecture,ET-SAM achieves about 3× inference acceleration while obtaining com-petitive performance on HierText, and improves an average of 11.0% F-score on Total-Text, CTW1500, and ICDAR2015. The code and modelsare available at ET-SAM.