Line an 8 inch square pan with aluminum foil or parchment paper.
Heat the chocolate and the sweetened condensed milk over medium low heat until the chocolate is melted. Be sure to keep stirring so it melts evenly.
Add in the vanilla and the other spices. If you are concerned about the spiciness, add only 1/8 teaspoon of the cayenne pepper and taste it, adding more slowly.
After everything is thoroughly combined, immediately pour the fudge into the pan. Allow it at least 4 hours to set in the refrigerator.
Remove from the pan, cut, and enjoy. Store in an air-tight container outside of the refrigerator for up to 2 weeks.