The line is a common feature in the man-made environment, inherently encoding spatial structural information. We propose an accurate and robust bundle adjustment (BA) solution that estimates the 6-DoF pose with an independent RS model of the camera and the geometry of the environment based on the line among the images captured by rolling shutter (RS) camera. However, the nonlinear nature of camera motion curves the projection of each 3D straight line on rolling shutter images, exacerbating the complexity of the original problem. To tackle the challenges, we first establish the RS line projection theory using the Plücker line parameterization and derive a series of line-to-line errors that avoid time-consuming curve-to-line (c2l) errors and exhibit the degeneration-resistant ability. We also provide complete degeneration-resistant proof of our method. Extensive synthetic and real data experiments demonstrate that our method achieves comparable efficiency with higher accuracy over existing point-based RSBA solutions.