Preheat your oven to 325°F (160°C) and grease and flour a 9x5 inch loaf pan. In a large bowl, cream together the softened butter and granulated sugar until light and fluffy, then beat in the eggs one at a time, followed by the vanilla extract.

In a separate bowl, whisk together the flour and baking powder. Gradually add the dry ingredients to the wet ingredients, alternating with the milk, mixing until just combined. Gently fold in the diced fresh peaches.

Pour the batter into the prepared loaf pan and bake for 60-70 minutes, or until a wooden skewer inserted into the center comes out clean. Let the cake cool in the pan for 10 minutes before transferring it to a wire rack to cool completely.