By default, the EdgeAnnotations uses the font defined in the document options, under Annotations --> Notes.

If you do not wish to use this option, you can uncheck it and select another font format. The size of complementary geometry elements in the annotation is proportional to the font size.