Yes, if you file Schedule C for your referee business, and if you have a home office for that business, you can deduct mileage to get to and from a game.
The rules are "miles from your business to the worksite", so you would need to have a home office to consider your house as your business location.