IllegalStateException: Unable to register SSL bundle after 3.3.8 or 3.4.2

[TazBruce]

Bug report

Spring Boot Version: 3.4.2

After upgrading to Spring Boot 3.4.2, my app is crashing on boot with the following logs:

"@timestamp":"2025-01-27T10:42:49.772470089+13:00","level":"ERROR","thread_name":"main","logger_name":"o.s.b.SpringApplication","m
essage":"Application run failed","throwable_class":"ApplicationContextException","stack_trace":"java.io.IOException: **'/..data/tls.k
ey'** is neither a file nor a directory\n\tat o.s.b.a.s.FileWatcher$WatcherThread.register(FileWatcher.java:150)\n\tat o.s.b.a.ssl.Fi
leWatcher.watch(FileWatcher.java:93)\n\t... 80 common frames omitted\nWrapped by: java.io.UncheckedIOException: Failed to register
paths for watching: [/opt/tls/tls.key, /opt/tls/tls.crt]\n\tat o.s.b.a.ssl.FileWatcher.watch(FileWatcher.java:96)\n\tat o.s.b.a.s.S
slPropertiesBundleRegistrar.watchForUpdates(SslPropertiesBundleRegistrar.java:82)\n\t... 79 common frames omitted\nWrapped by: j.la
ng.IllegalStateException: Unable to watch for reload on update\n\tat o.s.b.a.s.SslPropertiesBundleRegistrar.watchForUpdates(SslProp
ertiesBundleRegistrar.java:85)\n\tat o.s.b.a.s.SslPropertiesBundleRegistrar.lambda$registerBundles$2(SslPropertiesBundleRegistrar.j
ava:70)\n\t... 78 common frames omitted\nWrapped by: j.lang.IllegalStateException: Unable to register SSL bundle 'server'

My application.yaml has the following config to mount a certificate:

spring:
  ssl:
    bundle:
      pem:
        server:
          keystore:
            certificate: file:${TLS_CERT_PATH:}
            private-key: file:${TLS_KEY_PATH:}
          reload-on-update: true
server:
  ssl:
    bundle: server
    enabled: ${TLS_ENABLED:false}
    enabled-protocols: TLSv1.3

My k8s deployment provides the environment variables:

- TLS_ENABLED=true
- TLS_CERT_PATH=/opt/tls/tls.crt
- TLS_KEY_PATH=/opt/tls/tls.key
- KEYSTORE_PATH=/opt/tls/keystore.p12

I'm not sure where /..data/tls.key comes from seeing as there's no config that provides that.

Possibly related to #43586?

Any help is appreciated!

[wilkinsona]

If it's related to #43586 then I suspect that /opt/tls/tls.key is a symlink that's pointing to /..data/tls.key. If this doesn't help, please provide a complete yet minimal example that reproduces the problem.

[TazBruce]

Hey @wilkinsona, thanks for your reply! I'm unsure if ../data is a path we can use. Hopefully the below can help 😄

This article may be a helpful link for context. Kubernetes manages symlinks much differently than a regular operating system.

So, when you start an inotify monitor on “user-visible files”, the default behavior of the system call is to follow the symbolic links recursively and watch the regular file at the end of the symbolic link chain (as that’s the file that will probably get updated, but not on Kubernetes).

This is where the Kubernetes AtomicWriter implementation comes into the picture: If there’s an update to the Secret/ConfigMap, kubelet will create a new timestamped directory, write files to it, update ..data symlink to the new timestamped directory (remember, it’s something you can do atomically, and finally “delete” the old timestamped directory. It’s how the files from a Secret/ConfigMap volume are always complete and consistent with one another.

To replicate this, we'd need to run this within a cluster (perhaps with kind), and have a pod that is consuming a certificate secret by mounting it as a volume with the relevant environment vars to point to it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 1
  revisionHistoryLimit: 3
  template:
    spec:
      containers:
        - name: app
          image: docker.io/library/my-app:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8443
          volumeMounts:
            - name: tls
              readOnly: true
              mountPath: /opt/tls
      volumes:
        - name: tls
          secret:
            secretName: my-cert-secret
---
apiVersion: v1
kind: Secret
metadata:
  name: my-cert-secret
type: kubernetes.io/tls
data:
  # values are base64 encoded
  tls.crt: |
    xxxxx
  tls.key: |
     xxxxx

To validate this further, we've had a go at writing some unit tests to verify

   @Test
    void rbSymlink1(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
        Path realFile = tempDir.resolve("realFile.txt");
        Path symlink = tempDir2.resolve("symlink.txt");
        Path relative = symlink.getParent().relativize(realFile);
        System.out.println(realFile);
        System.out.println(symlink);
        System.out.println(relative);
        Files.createFile(realFile);
        Files.createSymbolicLink(symlink, relative);
        System.out.println(Files.readSymbolicLink(symlink));
        WaitingCallback callback = new WaitingCallback();
        this.fileWatcher.watch(Set.of(symlink), callback);
        Files.writeString(realFile, "Some content");
        callback.expectChanges();
    }
 
    @Test
    void rbSymlink2(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
        Path realFile = tempDir.resolve("realFile.txt");
        Path symlink = tempDir2.resolve("symlink.txt");
        System.out.println(realFile);
        System.out.println(symlink);
        Files.createFile(realFile);
        Files.createSymbolicLink(symlink, realFile);
        System.out.println(Files.readSymbolicLink(symlink));
        WaitingCallback callback = new WaitingCallback();
        this.fileWatcher.watch(Set.of(symlink), callback);
        Files.writeString(realFile, "Some content");
        callback.expectChanges();
    }
 
    @Test
    void rbSymlink3(@TempDir Path tempDir, @TempDir Path tempDir2) throws Exception {
        Path realFile = tempDir.resolve("realFile.txt");
        Path symlink = tempDir2.resolve("symlink.txt");
        Path symlink2 = tempDir.resolve("symlink2.txt");
        Path relative = symlink.getParent().relativize(realFile);
        System.out.println(realFile);
        System.out.println(symlink);
        System.out.println(symlink2);
        System.out.println(relative);
        Files.createFile(realFile);
        Files.createSymbolicLink(symlink, relative);
        Files.createSymbolicLink(symlink2, symlink);
        System.out.print(Files.readSymbolicLink(symlink2));
        System.out.print(" -> ");
        System.out.println(Files.readSymbolicLink(symlink));
        WaitingCallback callback = new WaitingCallback();
        this.fileWatcher.watch(Set.of(symlink2), callback);
        Files.writeString(realFile, "Some content");
        callback.expectChanges();
    }

Interestingly enough, the third test (rbSymlink3) fails with the following:

FileWatcherTests > rbSymlink3(Path, Path) FAILED
java.io.UncheckedIOException: Failed to register paths for watching: [/var/folders/bq/sj25gqrx6vv81mfpxp86x09m0000gq/T/junit-15635141749719039404/symlink2.txt]
at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:96)
at org.springframework.boot.autoconfigure.ssl.FileWatcherTests.rbSymlink3(FileWatcherTests.java:180)

Caused by:
java.io.IOException: '/Users/xxx/src/spring-boot/spring-boot-project/spring-boot-autoconfigure/../junit-15635141749719039404/realFile.txt' is neither a file nor a directory
at org.springframework.boot.autoconfigure.ssl.FileWatcher$WatcherThread.register(FileWatcher.java:150)
at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:93)
... 1 more

Not 100% sure I'm on the right track, but I feel like it's something to do with how java manages working directories that have symlinks which point to relative directories (note how the IO exception leads to a ../ path which is similar to what we faced in the boot crash above)

Any help is much appreciated!

[bclozel]

@TazBruce Thanks for the feedback and the link to this article.

I have replicated the k8s setup and behavior with a test:

	/*
	 * Replicating a k8s configmap folder structure like:
	 *
	 * secret.txt -> ..data/secret.txt
	 * ..data/ -> ..a72e81ff-f0e1-41d8-a19b-068d3d1d4e2f/
	 * ..a72e81ff-f0e1-41d8-a19b-068d3d1d4e2f/secret.txt
	 *
	 * After a secret update, this will look like:
	 *
	 * secret.txt -> ..data/secret.txt
	 * ..data/ -> ..bba2a61f-ce04-4c35-93aa-e455110d4487/
	 * ..bba2a61f-ce04-4c35-93aa-e455110d4487/secret.txt
	 */
	@Test
	void shouldTriggerOnConfigMapUpdates(@TempDir Path tempDir) throws Exception {
		Path configMap1 = createConfigMap(tempDir, "secret.txt");
		Path configMap2 = createConfigMap(tempDir, "secret.txt");
		Path data = tempDir.resolve("..data");
		Files.createSymbolicLink(data, configMap1);
		Path secretFile = tempDir.resolve("secret.txt");
		Files.createSymbolicLink(secretFile, data.resolve("secret.txt"));
		try {
			WaitingCallback callback = new WaitingCallback();
			this.fileWatcher.watch(Set.of(secretFile), callback);
			Files.delete(data);
			Files.createSymbolicLink(data, configMap2);
			FileSystemUtils.deleteRecursively(configMap1);
			callback.expectChanges();
		}
		finally{
			FileSystemUtils.deleteRecursively(configMap2);
			Files.delete(data);
			Files.delete(secretFile);
		}
	}

	Path createConfigMap(Path parentDir, String secretFileName) throws IOException {
		Path configMapFolder = parentDir.resolve(".." + UUID.randomUUID());
		Files.createDirectory(configMapFolder);
		Path secret = configMapFolder.resolve(secretFileName);
		Files.createFile(secret);
		return configMapFolder;
	}

This test is green and I'm not replicating the issue.

Note, the article you're mentioning does not refer to relative folder but to folders with names starting with "..". In your case, it seems it's trying to resolve "/..data/tls.key" which doesn't seem to exist?
I think that the "rbSymlink3" test is invalid as it's trying to watch an link to a relative path that doesn't exist.

Maybe you can show the complete file structure of the container in this case?
Can you think of anything wrong with my test regarding k8s's behavior?

Thanks!

[kasprzakdanielt]

In my usecase, I have a letsencrypt certbot updating my files. Same problem as OP. When I have "reload-on-update" enabled it won't start.
System is Debian 12 with spring-boot running inside a docker container.
Container has a mounted path under "/etc/letsencrypt:/etc/letsencrypt"

ls -l /etc/letsencrypt/live/XXX/fullchain.pem
/etc/letsencrypt/live/XXX/fullchain.pem -> ../../archive/XXX/fullchain32.pem

config:

          reload-on-update: true
          keystore:
            certificate: file:/etc/letsencrypt/live/XXX/fullchain.pem
            private-key: file:/etc/letsencrypt/live/XXX/privkey.pem

Caused by: java.io.UncheckedIOException: Failed to register paths for watching: [/etc/letsencrypt/live/XXX/privkey.pem, /etc/letsencrypt/live/XXX/fullchain.pem]
2025-01-28T09:59:38.023118852Z 	at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:96)
2025-01-28T09:59:38.023127740Z 	at org.springframework.boot.autoconfigure.ssl.SslPropertiesBundleRegistrar.watchForUpdates(SslPropertiesBundleRegistrar.java:82)
2025-01-28T09:59:38.023136635Z 	... 80 common frames omitted
2025-01-28T09:59:38.023145117Z Caused by: java.io.IOException: '/../../archive/XXX/privkey32.pem' is neither a file nor a directory
2025-01-28T09:59:38.023154025Z 	at org.springframework.boot.autoconfigure.ssl.FileWatcher$WatcherThread.register(FileWatcher.java:150)
2025-01-28T09:59:38.023162721Z 	at org.springframework.boot.autoconfigure.ssl.FileWatcher.watch(FileWatcher.java:93)
2025-01-28T09:59:38.023171357Z 	... 81 common frames omitted

[bclozel]

@kasprzakdanielt I'm trying to reproduce this in a test but I'm missing something.
I'm not sure where the "/../../archive/XXX/privkey32.pem" path is coming from. Can you share the result of ls -l /etc/letsencrypt/live/XXX/privkey.pem which seems to be the issue here?

[kasprzakdanielt]

@bclozel ah, sorry, I copied the wrong part of a stacktrace.

 ls -l /etc/letsencrypt/live/XXX/privkey.pem
 /etc/letsencrypt/live/XXX/privkey.pem -> ../../archive/XXX/privkey32.pem

Here is a more detailed file structure:
https://community.letsencrypt.org/t/certbot-file-structure-need-detailed-demo/39687/2

[bclozel]

Thanks @kasprzakdanielt I managed to reproduce the behavior in a test and fixed it for the next maintenance release.

[kasprzakdanielt]

Glad to help. Thanks for the fix @bclozel