"I would think they would affect the flavor."
It's supposed to! Well, maybe not the stems, but the caps....
"The first step then is to separate the stems from the chilies, while keeping that star-like green cap intact for it adds a very subtle complex perfume-like flavor to the sauce.."
https://nourishedkitchen.com/ferment...-sauce-recipe/
"Trim the stems from the peppers. If they don’t pull off easily, I like to leave the little green caps on the peppers (but still remove the long stems). The caps impart a subtle, earthy perfume to the final sauce.
Read more at https://www.gardenbetty.com/fermented-hot-chile-sauce/#rXu7uxXkSIDWXI1G.99"
https://www.gardenbetty.com/fermented-hot-chile-sauce/