Using a vegetable peeler, peel the yellow part of the lemons. Be careful not to get too much of the white pith, as it is bitter. Reserve remaining peeled lemons.
Place water, sugar, and lemon peels into a small pot and bring to a boil.
Remove from heat. Place rosemary and tea bags into the pot and cover with a lid. Allow to steep for 5 minutes.
Remove lid and strain the tea mixture, removing tea bags, lemon peels, and rosemary. Do not squeeze the tea bags, as this can make it bitter.
Juice the remaining peeled lemons and add the lemon juice to the tea mixture.
Stir and chill. Serve over ice.