In 1957 he returned to Moscow to take charge of relations between the Soviet Communist Party and other Communist countries, including the European satellites, the East Asian Communist states, and later Cuba. He held this post for 10 years, acquiring considerable experience in international relations during that time.

In 1967, his political responsibilities increased greatly. That year he was appointed chairman of the Soviet secret police (KGB, acronym for the Committee for State Security). He was chosen by the Soviet leaders in the Politburo for two major reasons. First, he was not a Stalinist; they could rely on him to maintain party control over the secret police. Second, he was not a close supporter of Brezhnev and could be counted on not to let the KGB fall under the control of the new party leader. One of the principal tasks which confronted Andropov was the restoration of the prestige of the secret police, whose reputation had suffered severely in previous years when public denunciation of Stalin's crimes had revealed its terrible abuses of power in carrying out Stalin's terror. At the same time, he had to silence such Soviet "dissenters" as the physicist Andrei Sakharov and the novelist Alexander Solzhenitsyn, who were demanding further destalinization and publicly protesting violations of human rights in the Soviet Union. Their activities were reported and their writings published in the West.

Andropov remained chairman of the KGB for 15 years, longer than any other secret police chief since Stalin's death. He owed his lengthy term of service to his success at the job. During those years the KGB became one of the most efficient secret police organizations in the world. He organized a public campaign to raise the prestige of the KGB among the Soviet population. He appears to have prevented KGB officers from abusing their power for the sake of personal profit, as other party and police officials were doing. By the early 1980s Andropov had accumulated material from KGB investigations to prove widespread bribery and corruption within the Soviet bureaucracy. He appointed loyal party officials to high positions within the KGB and established his own reputation for efficiency and incorruptibility. His years of secret police leadership made him a major contender to become the next leader of the Soviet Union.

Meanwhile he was able to eliminate public dissent within the Soviet Union. He used several methods of repression. The KGB arrested dissenters for violating laws banning "anti-Soviet propaganda." They were sentenced to years of hard labor in prison camps. Other were sent without trial to psychiatric hospitals for the criminally insane, where they were treated with mind-altering drugs. The most prominent dissenters, protected from harsh punishment by their international fame, had to accept permanent exile abroad. By the end of the 1970s the KGB had virtually wiped out all groups defending human rights and individual liberties in the Soviet Union and had enforced public silence on Stalin's crimes.

Andropov was rewarded for his success. In 1973 he became a member of the ruling party committee, the Politburo. He was its youngest member at that time. In mid-1982, his colleagues on the committee designated him Brezhnev's successor, making him a member of the Secretariat and permitting him to resign his post as chairman of the secret police. Within two days of Brezhnev's death on November 10, 1982, he received the formal party appointment of general secretary.

Andropov had only a brief time to be leader of the Soviet Union. He began in those months to rejuvenate the party leadership and to implement new policies. He appointed to the Politburo younger Communist officials, including a young expert on agriculture named Mikhail Gorbachev. He launched a campaign against corruption, making use of the secret police to hunt out and punish culprits within the state and party apparatus. He tried to improve industrial production by introducing measures punishing absenteeism and rewarding productivity. Finally, he launched a "peace offensive" intended to limit the introduction of new U.S. nuclear missiles in Europe. When in early September 1983 a Soviet fighter plane shot down a South Korean airliner flying over Soviet air space, he defended the hasty action of his frontier forces. The international protest over that incident seriously worsened Soviet relations with Western countries.

In late 1983 Andropov fell seriously ill. Suffering from an incurable kidney disease, he sought the agreement of his colleagues in the Politburo to the appointment of Mikhail Gorbachev as his successor. However, an older Politburo member, Konstantin Chernenko (whom Brezhnev had originally favored), was able to prevent this move and claimed the succession for himself. Andropov died in February 1984.