Clemens will take over for Charlie Whitehurst, who became an unrestricted free agent.
Clemens spent the last two seasons with the St. Louis Rams as a backup to starter Sam Bradford. Clemens started the last nine games after Bradford went down with a knee injury, completing 142 of 242 passes for 1,673 yards and eight touchdowns, with seven interceptions.
Clemens was selected by the New York Jets in the second round of the 2006 draft. He signed with Washington in 2011, was released at the end of training camp and he later signed with Houston. He was claimed by the Rams after being waived that December.
Conner is the second former Colts player to sign with the Chargers in the last three days, joining running back Donald Brown, who signed with the Chargers on Tuesday.
Chargers general manager Tom Telesco was with the Colts' front office when Conner was drafted in 2010.
Conner started 36 of 52 games played in four seasons with the Colts. He had 236 tackles, a sack and two forced fumbles, including a 104-tackle season in 2011 when he played in all 16 games, with 15 starts.