God can be said to have a number of perfections - infinite love, providence, justice, etc. Infinite mercy would seem to be one of many perfections, but in fact, as Fr. Julian Chrosciechowski, M.I.C., explained in his book, God's Infinite Mercy, "All [of God's] perfections are radiant with His mercy" (page 25).
"...We may conclude that God's Goodness is only Mercy endowing; His Generosity is Mercy unusually free with its gifts; His Providence is Mercy that is ever watchful; His Justice is Mercy that rewards over and above merit and lessens the due punishment. Finally, Love is the Mercy of God, that, full of compassion, draws us to Him" (Chrosciechowski, 26).
One could counter with the existence of Hell, but perhaps Hell would be even worse than it is were it not for God's justice being tempered with mercy.
The parable of the prodigal son helps explain how God's justice is tempered with His mercy, and like some other parables, shows that God does not punish those who turn back to Him as justice would demand. His justice is indeed greatly tempered with mercy for those who will accept His mercy.
A number of passages in the Diary of St. Faustina point to this. On one occasion, Jesus said to St. Faustina, "Write this: Everything that exists is enclosed in the bowels of My mercy, more deeply than an infant in its mother's womb" (Diary, 1076).