Jim attended Nortre Dame for two and one-half years: 1939-40, 1940-41, and fall of 1941. When he wasn't playing football, he was studying mechanical engineering, which remained an avocation throughout his life. Like his two team mates in high school, Gunnar Eide and Herb Johnson, Jim played end for Notre Dame. Freshmen were restricted to the freshman team, but he made varsity when a sophmore. As a young member of the team he didn't see much action, but he was affiliated with one of the greatest college football dynasties of all time. The head coach during the 1939-1940 season was Elmer Layden. Elmer had been one of the famous Four-Horsemen playing fullback under the legendary coach Knute Rockne and went on to become Commissioner of the NFL. 1941 began a new dynasty of success at Nortre Dame with Frank Leahy, also a protege (tackle) of Knute Rockne. Rockne left an unparalleled legacy of winning and training future winners.

A spirit of nationalistic urgency was sweeping the country after the bombing of Pearl Harbor in December of 1941, and Jim attempted to join the Army. To his surprise, the medical examiners found some problem in his eyes and rejected him. The problem was only temporary since he was soon to find himself in flight school training to fly P-38 twin engine fighters and ultimately back on the football field playing in the LA Coliseum for the 4th Air Force.