What is the “sin that leads to death” (16-17)? To answer this, we must consider the context of the entire letter.
God has given His people resurrection life, by faith in Jesus, which overcomes the world (5:4, 10-12). John writes so that authentic Christians may possess confidence in this resurrection life (5:13).
For this confidence, he gave a three-part test, summarized in 5:1-5: do we a) love the brotherhood (2a), b) obey His commandments (2b), and c) believe rightly about Jesus, that he is the Son of God (5)? Again, John’s goal is our confidence in God - even when we sin (1:9).
Confidence leads to loving, faith-filled, obedient action: prayer (5:14-15), for our brother (16). If we observe our brother sinning, and we pray for him, we can have confidence - just as we do for ourselves (1:9) - that God will answer our prayer, and bring that resurrection life to pass in that brother, despite that sin, overcoming that sin, maybe even through that sin.
But some had defected from faith in Jesus, love to His people, and obedience to His word. Thus they were never authentic (2:19) and therefore do not possess resurrection life. This defection, therefore, is the “sin that leads to death.” As long as they remain in that defection, don’t pray for them as you would yourself or your brother. Pray instead that they would be truly “born of God” (5:18).