Java – Fix Invalid Keystore Format Error

I had exactly the same issue. Indeed, the keystore file was invalid and not related to the JDK//JRE version. The problem in my case was caused by Maven. I was using the following option in my pom file:

<resources>
        <resource>
            <directory>src/main/resources</directory>
            <filtering>true</filtering>
        </resource>
    </resources>

The "true" value in the filtering was messing with the key file. Therefore, the keyfile that was available in my classpath when Spring run was not exactly the same I had under my directory "src/main/resources" and that caused the Invalid Keystore Format exception. When I tested with keytool I was using the one under the "resources" folder so that was misleading the real issue.

Solving the issue: in your pom.xml file, change the value for "filtering" to "false". Another way of solving the issue was to specify explicitly the location of the keystore in the application.properties file. So instead of:

server.ssl.key-store: classpath:keystore.jks

I used

server.ssl.key-store: keystore/keystore.jks

It seems to be a bug in what keytool displays rather than what it does. Consider the following experiments.

(EDIT: bug report filed)

First, my jdk version is 1.8.0_152:

excalibur:~ ronan$ java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

Now, create the keystore as you specified:

excalibur:~ ronan$ keytool -genkey -alias serverprivate -keystore server.private -keyalg rsa -storepass apassword -keypass apassword
What is your first and last name?
  [Unknown]:  Art Vandelay
What is the name of your organizational unit?
  [Unknown]:  Export/Import
What is the name of your organization?
  [Unknown]:  Vandelay Industries
What is the name of your City or Locality?
  [Unknown]:  New York
What is the name of your State or Province?
  [Unknown]:  New York
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=Art Vandelay, OU=Export/Import, O=Vandelay Industries, L=New York, ST=New York, C=US correct?
  [no]:  yes


Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore server.private -destkeystore server.private -deststoretype pkcs12".

Now, following the instructions given:

excalibur:~ ronan$ keytool -importkeystore -srckeystore server.private -destkeystore server.private -deststoretype pkcs12
Enter source keystore password:  
Entry for alias serverprivate successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

Warning:
Migrated "server.private" to Non JKS/JCEKS. The JKS keystore is backed up as "server.private.old".

But when we list it with keytool it still says JKS.

excalibur:~ ronan$ keytool -list -keystore server.private
Enter keystore password:  
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

serverprivate, Dec 4, 2017, PrivateKeyEntry, 
Certificate fingerprint (SHA1): 16:E8:C6:12:7A:F1:7A:B8:64:98:EC:12:C4:07:9E:67:06:BD:DD:BD

However, openssl can parse it as pkcs12 just fine.

excalibur:~ ronan$ openssl pkcs12 -in server.private
Enter Import Password:
MAC verified OK
Bag Attributes
    friendlyName: serverprivate
    localKeyID: 54 69 6D 65 20 31 35 31 32 34 31 33 32 30 38 31 38 32 
Key Attributes: <No Attributes>
Enter PEM pass phrase:
Bag Attributes
    friendlyName: serverprivate
    localKeyID: 54 69 6D 65 20 31 35 31 32 34 31 33 32 30 38 31 38 32 
subject=/C=US/ST=New York/L=New York/O=Vandelay Industries/OU=Export/Import/CN=Art Vandelay
issuer=/C=US/ST=New York/L=New York/O=Vandelay Industries/OU=Export/Import/CN=Art Vandelay
-----BEGIN CERTIFICATE-----
MIIDoTCCAomgAwIBAgIEKSoRITANBgkqhkiG9w0BAQsFADCBgDELMAkGA1UEBhMC
VVMxETAPBgNVBAgTCE5ldyBZb3JrMREwDwYDVQQHEwhOZXcgWW9yazEcMBoGA1UE
ChMTVmFuZGVsYXkgSW5kdXN0cmllczEWMBQGA1UECxMNRXhwb3J0L0ltcG9ydDEV
MBMGA1UEAxMMQXJ0IFZhbmRlbGF5MB4XDTE3MTIwNDE4NDM1NVoXDTE4MDMwNDE4
NDM1NVowgYAxCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhOZXcgWW9yazERMA8GA1UE
BxMITmV3IFlvcmsxHDAaBgNVBAoTE1ZhbmRlbGF5IEluZHVzdHJpZXMxFjAUBgNV
BAsTDUV4cG9ydC9JbXBvcnQxFTATBgNVBAMTDEFydCBWYW5kZWxheTCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAIwVIy24Hws6ffaEQGhYEyaw2JC4UPLH
wtSXpJaZf3lrdtbktbLC+zKJ93fFVfoukX5vpnZKxv+f724duAszkVWN3bBrnAwb
6sh445tXiRhT2eS+z0rHtXwiPWUKOAPOxGUP1Lhfl5ZCPR4bmohyh/daSX1ppT3P
k6//lf1y2RdsS5hZ5Ltx9dr99/txsv4a1D8uuPDHIf8dvWfOqGb7ywIAlrbh5JsA
4IWjMVFMpDQ2WM/C+v48y2W7AdgxGD0J/nNC6t4QfAgwhvVyXbiu9IbAR+ivn59E
VqC83qv4bBkffB3yYk6WSHSl7YloJueWalVAoudf4uHeaNZBLCvbNsMCAwEAAaMh
MB8wHQYDVR0OBBYEFNOc6gJkUFCSgyHCDFGJmZwtPLgIMA0GCSqGSIb3DQEBCwUA
A4IBAQAVyc6j9U+va5U02MxsjDE6LWDPg+0FaqpTAQlU6xhVrZlKWuX+Dup4PF6S
s4oqhInb7XcGOjbLkVTnuEY5mwfHnyGqNhmyyUwJOVrWZJWBZADvhdcZi0w/jclg
jAb7dP1VssR4Ozgsd1XkGxnS7P8e344xpjQd1krPc1Od7qhFSF6FufTDvgEcrmx/
qrHpLpwoYPTyuv+NtD+7KlWINqWq5PT1olydd6ki8O+bTsgDuy8rnvqd/jJ7dsuv
VLNc+IDEEMO26hS8d1WkppTTuLrayE07LCV8df68WHv6KumDSc5Mb0zOUdLqZUDQ
Wmxbt0kEAAG6jgbpACBpGEqtAtTs
-----END CERTIFICATE-----

and the following Java snippet can also parse it.

import java.io.FileInputStream;
import java.security.KeyStore;
import java.util.Collections;

public class Main {
    public static void main(String[] args) throws Exception {
        KeyStore pkcs12 = KeyStore.getInstance("PKCS12");
        pkcs12.load(new FileInputStream("../../../server.private"), "apassword".toCharArray());
        for (String alias : Collections.list(pkcs12.aliases())) {
            System.out.println(alias);
        }
    }
}

and the output is

serverprivate