Preheat oven to 350 F. Line a 12 cup muffin pan with cupcake liners. In a large bowl, whisk flour, cocoa powder, sugar, salt, and baking powder together. Add in milk, oil, eggs, and vanilla extract. Whisk until smooth.
Evenly divide batter between cupcake liners. Bake for 18-20 minutes. Let cool completely. In a large bowl, beat butter with cocoa powder, sugar, milk, and vanilla extract until smooth. Place crushed Oreo cookies into a shallow dish.
Frost cupcakes, dip the tops in the crushed Oreos. Top with gummy worms.