Preheat oven to 350 degrees F. Prep your 9x5” loaf pan with a non-stick spray or line with parchment paper.
Mix 1/3 cup brown sugar and 2 tsp cinnamon in a small bowl, set aside.
In a stand mixer or with a hand mixer, beat 1/2 cup softened butter and 2/3 cup granulated sugar together until fluffy, approx. 2 minutes. Add eggs and vanilla, continue to mix.
Gently mix flour, baking powder and salt in separate bowl, then slowly add to creamed butter mixture until fully combined. Add 1/2 cup milk into this mix until smooth.
Pour half the batter into the lined loaf pan, and add half of the peaches on top of the batter in the pan. Sprinkle half of the brown sugar + cinnamon mixture on top of the apple layer.
Repeat and add the remaining batter overtop and top with remaining peaches and brown sugar mix. Use a knife or toothpick to gently swirl the apple mixture into the batter, ensuring the peaches on top layer or patted into the loaf batter.
Bake until toothpick inserted into middle of loaf comes out clean, about 1 hour. Allow loaf to cool on wire rack after baking.