After Peter and Paul went to Rome and were martyred, the early continued to recognize the See of Rome as possessing a special significance.  As early as AD 80, around the time John the Evangelist was writing his Gospel, we have a letter from Clement, bishop of Rome, to the Corinthians, in which Clement speaks with the Apostolic authority of Peter and Paul to settle a dispute that had arisen among the faithful in Corinth.

The letter is well worth referencing for the glimpse it gives us into the first century of the Church, and I've linked it below.  All throughout, we see implicitly the authority of Rome recognized and exercised.

At the outset, Clement writes, "Owing, dear brethren, to the sudden and successive calamitous events which have happened to ourselves, we feel that we have been somewhat tardy in turning our attention to the points respecting which *you consulted us*; and especially to that shameful and detestable sedition, utterly abhorrent to the elect of God" (emphasis mine).  So it is clear that the church in Corinth appealed to the church of Rome to settle a conflict.

Later, we see Clement say "If, however, any shall disobey the words spoken by Him through us, let them know that they will involve themselves in transgression and serious danger," which indicates his own solemn knowledge of his authority over the Church.

Yet later, the letter concludes with the following: 

"Right is it, therefore, to approach examples so good and so many, and submit the neck and fulfill the part of obedience, in order that, undisturbed by vain sedition, we may attain unto the goal set before us in truth wholly free from blame. Joy and gladness will you afford us, if you become obedient to the words written by us and through the Holy Spirit root out the lawless wrath of your jealousy according to the intercession which we have made for peace and unity in this letter."

It is clear, then, that even in the earliest days after the death of Peter, his successor in Rome was aware of his authority as the chief steward, and the holder of the keys, and the other churches recognized and submitted to this authority.

Again, this letter was written while the Apostle John was still alive.  Surely, if the church in Rome had overstepped, the Apostles directly appointed by Christ would have had ample opportunity to set the record straight.