The second Pictou Academy was built when the original school on Church Street became too small. The new school, located on Patterson Street, overlooked the town.