Kafka message sending fails with 'class SslBundleSslEngineFactory could not be found'

[kst1980]

Spring Boot Kafka SSL Issue on EKS Deployment using Spring boot 3.4.3(tried 3.3.5 and 3.4.2)
We are facing an issue where the Kafka producer works fine in local environments (IntelliJ, mvn spring-boot:run, and java -jar). However, after deploying to EKS, we encounter the following exception:

Exception occurred: org.apache.kafka.common.config.ConfigException: Invalid value org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory for configuration ssl.engine.factory.class: Class org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory could not be found.. Stack Trace: [org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:778), org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:531), org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:524), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:114), org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:134), org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:643), org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:295), org.springframework.kafka.core.DefaultKafkaProducerFactory.createRawProducer(DefaultKafkaProducerFactory.java:944), org.springframework.kafka.core.DefaultKafkaProducerFactory.createKafkaProducer(DefaultKafkaProducerFactory.java:826), org.springframework.kafka.core.DefaultKafkaProducerFactory.doCreateProducer(DefaultKafkaProducerFactory.java:793), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:768), org.springframework.kafka.core.DefaultKafkaProducerFactory.createProducer(DefaultKafkaProducerFactory.java:762), org.springframework.kafka.core.KafkaTemplate.getTheProducer(KafkaTemplate.java:976), org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:828), org.springframework.kafka.core.KafkaTemplate.observeSend(KafkaTemplate.java:805), org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:608),

Issue Details:
Kafka producer initialization delayed: Instead of initializing at application startup, it only starts when sending a message, leading to the SslBundleSslEngineFactory class not being found.
Temporary workaround: We manually create the producer in a @PostConstruct method inside a KafkaConfig class.

@Slf4j
@Configuration
@RequiredArgsConstructor
public class KafkaConfig {
    private final ProducerFactory<String, String> producerFactory;

    @PostConstruct
    public void initializeProducer() {
        try {
            Producer<String, String> producer = producerFactory.createProducer(); // Workaround to initialize Kafka producer
            log.info("Kafka Producer initialized successfully: {}", producer);
        } catch (Exception e) {
            log.error("Failed to initialize Kafka Producer", e);
        }
    }
}

Spring Boot Configuration (application.yml)

spring:
  application:
    name: test
  ssl:
    bundle:
      pem:
        kafkaCert:
          keystore:
            certificate: classpath:KafkaKeystoreCert.pem
            private-key: classpath:KafkaKeystoreKey.pem
          truststore:
            certificate: classpath:KafkaTruststorePem.pem
  kafka:
    bootstrap-servers: ${KafkaBootstrapServers}
    ssl:
      bundle: kafkaCert
    security:
      protocol: SSL
    client-id: client-id
    producer:
      retries: 3
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

Request for a Proper Solution
Do we have a proper fix for this issue instead of the workaround?

Is there a missing dependency or classpath issue specific to the environment?
Are there configurations that need adjustment to ensure Kafka SSL settings load correctly?
Any best practices for ensuring SslBundleSslEngineFactory is properly available at runtime?

[mhalbritter]

This could be a classloader issue. It does work locally with SSL enabled, too? Or is SSL only enabled in EKS?

Please take the time to provide a complete minimal sample (something that we can unzip or git clone, build, and deploy) that reproduces the problem.

[kst1980]

Yes, it works locally with the same configuration and SSL enabled, but I read that it might be functioning correctly locally due to the JVM.

I have added the following dependency, which provides org.springframework.boot.autoconfigure.kafka.SslBundleSslEngineFactory, and verified its presence in the image after packaging:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-autoconfigure</artifactId>
</dependency>

Since the Spring Kafka producer is lazy-initialized, it throws a ClassNotFoundException after the application startup.

[mhalbritter]

Please provide a sample to reproduce.

[kst1980]

kafka-producer.zip - sample

[mhalbritter]

Hey, do you have any instructions how to reproduce the issue?

I've added a compose file and some SSL config (the certs in your sample are all empty), and it works, both in tests and locally when running with java -jar. Do i need to do something else to see it fail?

sb-44414.zip

Since the Spring Kafka producer is lazy-initialized, it throws a ClassNotFoundException after the application startup.

Also I don't see any spring-boot-autoconfigure, which you mentioned here.

[kst1980]

spring-boot-autoconfigure is included in spring-boot-starter, so I didn't add it explicitly. As I mentioned earlier, everything works fine locally. However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.

[patpatpat123]

I do not want to pollute this thread, apologies.

I am facing the same issue with all the above springboot versions mentioned.

For me, I have a case where it is working in "normal" mode and local, but I always encounter this same issue when running with GraalVM native image

Just wondering if I should create a new issue, targeting specifically native images, or if this is related.

[mhalbritter]

@kst1980

However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.

I still think this is a classloader issue. The classloader associated with the thread when using the @PostConstruct workaround is able to the load the class, so it's not a missing dependency.

Is there something different in the dependencies or are any agents attached when running in EKS?

[mhalbritter]

I do not want to pollute this thread, apologies.

I am facing the same issue with all the above springboot versions mentioned.

For me, I have a case where it is working in "normal" mode and local, but I always encounter this same issue when running with GraalVM native image

Just wondering if I should create a new issue, targeting specifically native images, or if this is related.

Please open a new issue, and please attach a reproducer. Thanks!

// Edit: Nevermind, there's a class hint missing. While it's the same error message as this issue, it's not related. I've opened #44435 for that.

[mhalbritter]

Instead of putting SslBundleSslEngineFactory as a string in the configuration map, we decided to put the class directly in the config map. This hopefully fixes the classloading issue.

[kst1980]

@kst1980

However, we're encountering this issue when deploying to AWS EKS and making a sample request. Additionally, we're using Aiven Kafka.

I still think this is a classloader issue. The classloader associated with the thread when using the @PostConstruct workaround is able to the load the class, so it's not a missing dependency.

Is there something different in the dependencies or are any agents attached when running in EKS?

Using the Datadog Java Agent https://dtdg.co/latest-java-tracer, when can I test this fix and with which version?

[mhalbritter]

Maybe that's the problem. The fix (hopefully) will be released as 3.3.10 next month (https://calendar.spring.io/), or available on Spring Snapshots repo as 3.3.10-SNAPSHOT.