Check Java:
Java 8 is not supported with FileMaker Server 12.
To resolve this issue, contact us to upgrade to FileMaker Server 13 or remove Java 8 and reinstall Java 7.
Java Runtime Environment 7, update 71 will work with FileMaker Server 12v5.
Uninstall Java 8 on the machine first.
Mac uninstall Java Instructions: http://www.java.com/en/download/help/mac_uninstall_java.xml
PC uninstall Java instructions: http://www.java.com/en/download/help/uninstall_java.xml
Download Java 7 (Mac or PC)
Update Java security preferences on the server to Medium.
If the Admin Console doesn’t open with Java update 71.
Try changing the Java security preference to Medium on the Java Control panel in the Mac System Preferences.
Make sure you click - Apply.