Ramsay joined Kilmarnock during the 1919-20 season and was a member of the side that won the Scottish Cup. In February 1924, Leslie Knighton, the manager of Woolwich Arsenal, persuaded Ramsay to join the club. He made his debut against Liverpool on 1st March 1924. He held his place in the side for the rest of the season.
Henry Norris sacked Leslie Knighton at the end of the 1924-25 season. In the summer of 1925 Herbert Chapman, the highly successful manager of Huddersfield Town, was persuaded to join Arsenal. In the 1925-26 season Arsenal finished in second-place to Chapman's old club, Huddersfield Town. Top scorer was Jimmy Brain who established a new club record with 33 goals. Ramsay played in 26 games that season until being replaced by Billy Blyth.
Ramsay regained his place at the start of the 1926-27 season before rejoining Kilmarnock.
James Ramsay died in 1969.