Provide a possibility to extend/override configuration properties binding process

[dmitriytsokolov]

We faced an issue where a property source for 700+- objects (consists of 100+- fields) binds to java objects slowly: around 6 minutes. Our services rely on spring cloud hot refresh and because of slow binding users are struggling. We could optimize this process for our use case, but most of the binding classes are closed for extension.

[philwebb]

There's not really enough information here for us to be able to take any action. Could you describe what optimizations you're trying to apply and what changes you'd need to do them?

A sample application that shows the issue would also be very helpful. Perhaps if we profile it there might be some general optimizations we can apply for everyone.

[dmitriytsokolov]

I'm mostly interested in extending org.springframework.boot.context.properties.bind.Binder class, but almost all classes inside org.springframework.boot.context.properties.bind are package-private.

About optimisation. I'm not sure it would work generally, but the idea is the following:
I have an ConfigurationProperties class which first child field is a hashmap, so my yml files that are bond to ConfigProps looks the following way:

config-props-prefix:
  some-root-field-name:
    map-key-1:
      obj-field1-key: value1
      obj-field2-key: value2
      ...

And I have thousands of such files that are bond to the map. And I don't need a property config-props-prefix.some-root-field-name.${some-map-key}.obj-field1-key to be processed by binder multiple times, so I could cache the result and use it for binding process of other map entries. Does it make any sense to you?

I'm using spring boot 2.7.18, but I've tested on spring boot 3.2.7 and performance is the same.

[wilkinsona]

@dmitriytsokolov thanks for the additional details. Can you please share a sample application that uses Spring Boot 3.2.x or later that demonstrates what you've described above?

[dmitriytsokolov]

@wilkinsona sure. Should I just create an example with couple of config files, or with hundreds to emulate performance issues?

[wilkinsona]

Thanks. Something that replicates the performance issue would be ideal. Perhaps you could use a script to generate a sufficient volume of synthetic test data?

[dmitriytsokolov]

attaching example of spring app with 1000 configs that is started up for 40-50s locally. In my real app I have more fields and more spring placeholders, so app start time/hot refresh takes 5-6 minutes
demo_config_props.zip

One more thing: initially I though that I could cache results of resolved spring placeholders (because I have a lot of them) and get some performance benefits, so I've made the following change in Binder class locally:

	private <T> Object bindProperty(Bindable<T> target, Context context, ConfigurationProperty property) {
		context.setConfigurationProperty(property);
		Object original = property.getValue();
		Object fromCache = this.resolvedPlaceholdersCache.get(original);
		if (fromCache != null) {
			return fromCache;
		}
		Object result = this.placeholdersResolver.resolvePlaceholders(original);
		result = context.getConverter().convert(result, target);
		this.resolvedPlaceholdersCache.put(original, result);
		return result;
	}

but I've got approximately minus 20 seconds out of 5-6 minutes of binding time. It's not much, but still with the ability to override/extend some binding logic I could reduce it even more knowing some specifics of my serivce

Anyway, will be waiting for your input, thanks a lot

[dmitriytsokolov]

The other option I'm thinking of:
As you see I have a simple hashmap on the root level. So if only a part of the binding beans were opened for extension, I could try rebinding config props partially manually (binding of each map's element asynchronously and build the map by myself). This should help, but of course I'm not sure about it. I'm going to install local version of spring with required updates and will try to implement this approach in my app. But I would appreciate your input on this idea.

[Mobe91]

I am experiencing similar issues with a spring-boot application that I am working on. It is a multi-tenant application where configuration for each tenant is provided via environment variables (~6000 keys in total atm). The binding of these properties takes more than 10 minutes. This happens on startup and whenever the configuration is refreshed using Spring Cloud as suggested by @dmitriytsokolov. I also suspect this to be the cause for the issue I commented on here (at least in my case).

I have done some profiling for this and was able to identify the following performance hotspots:

1. Most of the time was spent in org.springframework.boot.context.properties.source.SystemEnvironmentPropertyMapper#isLegacyAncestorOf. I am wondering if this behavior is still needed or if it could be removed.

   It was hard for me to work around this issue as the property mappers are statically constructed and assigned in org.springframework.boot.context.properties.source.SpringConfigurationPropertySource#from. I ended up creating a copy of SpringConfigurationPropertySource and SpringIterableConfigurationPropertySource that use a modified SystemEnvironmentPropertyMapper (stripped the legacy stuff from it).

2. Also org.springframework.core.env.MapPropertySource#getPropertyNames was a hotspot as it seems to re-construct the property names many times during binding.

   I was able to work around this by wrapping all property sources in an immutable wrapper before providing them to the binder. The wrapper evaluates the property names only once at construction time.

3. Another hotspot was org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.Mappings#updateMappings(java.lang.String[]) which was invoked very frequently. Most of the ConfigurationPropertySources were not immutable so the Mappings were updated every time they were accessed.

   I reused the workaround from (2) to solve this by implementing org.springframework.boot.origin.OriginLookup and returning true for org.springframework.boot.origin.OriginLookup#isImmutable. At that point the binding runtime was down to < 10 seconds.

4. The last hotspot was in org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource#containsDescendantOf. Here the descendent relationship cache implemented in org.springframework.boot.context.properties.source.SpringIterableConfigurationPropertySource.Mappings is only used if this.ancestorOfCheck == PropertyMapper.DEFAULT_ANCESTOR_OF_CHECK which is not the case for org.springframework.core.env.SystemEnvironmentPropertySource. So the descendent caching is not in effect for the SystemEnvironmentPropertySource.

   I decided to not create a workaround for this as it would have required further modification to the Spring implementation and the performance was at an acceptable level already. However, I assume that there would be a further substantial reduction in runtime with an improved descendent caching.

Above evaluation was done using Spring Boot 3.1.6. However, I checked the latest Spring Boot sources and there seems to have been no relevant changes to the code sections mentioned above.

[philwebb]

Unfortunately the relaxed binding code continues to be a source of frustration. @Mobe91 have you tried using the ConfigurationPropertyCaching interface to enable caching globally?

[Mobe91]

Thank you for the helpful pointer @philwebb , I did not know about the ConfigurationPropertyCaching interface!

By enabling caching globally using the ConfigurationPropertyCaching interface I was actually able to remove the workarounds for (2) and (3) mentioned in my earlier comment without taking a performance hit.

Unfortunately, the workaround for (1) is still needed to yield acceptable performance due to the lack of descendent caching for SystemEnvironmentPropertySource mentioned in (4).

Some numbers:

• With workaround (1): ~8 seconds
• Without workaround (1): ~55 seconds

[quaff]

Unfortunately the relaxed binding code continues to be a source of frustration.

Could it be disabled by default since next major release?

[dmitriytsokolov]

Hi @Mobe91. Thanks for the comment. Would it be possible for you to somehow provide snippets of your workarounds?

@philwebb is anyone going to take a look at the example I provided? Would you suggest anything on this matter?

[wilkinsona]

@dmitriytsokolov We're a small team with many different priorities. Rest assured, we will look at the sample when we have time to do so. That may take a little while as we work on higher priority issues for 3.4.0-RC1 that releases next week. Thank you for your patience in the meantime.

[dmitriytsokolov]

Of course, thanks a lot @wilkinsona!

[Mobe91]

@dmitriytsokolov

Would it be possible for you to somehow provide snippets of your workarounds?

Sure, this is what it comes down to:

/**
 * Mostly a copy of {@link SpringConfigurationPropertySource} to pass different {@link PropertyMapper}s to the
 * constructed {@link ConfigurationPropertySource}. Sadly, at the time of writing Spring does not provide a way
 * to do this differently.
 */
public class CustomConfigurationPropertySourceFactory {

    private static final PropertyMapper[] DEFAULT_MAPPERS = { DefaultPropertyMapper.INSTANCE };

    private static final PropertyMapper[] SYSTEM_ENVIRONMENT_MAPPERS = {
        NonLegacySystemEnvironmentPropertyMapper.INSTANCE,
        DefaultPropertyMapper.INSTANCE };

    public static ConfigurationPropertySource from(PropertySource<?> source) {
        Assert.notNull(source, "Source must not be null");
        PropertyMapper[] mappers = getPropertyMappers(source);
        if (isFullEnumerable(source)) {
            return new SpringIterableConfigurationPropertySource((EnumerablePropertySource<?>) source,
                mappers);
        }
        return new SpringConfigurationPropertySource(source, mappers);
    }

    // need to copy some more static methods here - skipped for brevity
}

/**
 * Copy of {@link SystemEnvironmentPropertyMapper} modified to remove the handling of legacy property naming
 * <a href="https://github.com/spring-projects/spring-boot/issues/42361#issuecomment-2399941504">for performance
 * reasons</a>.
 */
class NonLegacySystemEnvironmentPropertyMapper implements PropertyMapper {

    // Get rid of isLegacyAncestorOf and all uses of it
}

Use it:

        // Make a copy of the environment's PropertySources to wrap them in ConfigurationPropertySources created
        // using our CustomConfigurationPropertySourceFactory.
        Iterable<ConfigurationPropertySource> configurationPropertySources = env.getPropertySources().stream()
            .filter(Predicate.not(ConfigurationPropertySources::isAttachedConfigurationPropertySource))
            .map(CustomConfigurationPropertySourceFactory::from)
            .toList();
        // Enabling caching for the copied ConfigurationPropertySources at this point is important for the binding
        // performance.
        ConfigurationPropertyCaching.get(configurationPropertySources).enable();
        Binder binder = new Binder(configurationPropertySources, new PropertySourcesPlaceholdersResolver(env));

[wilkinsona]

I have a possible optimisation for this situation that significantly improves the startup time of the provided sample. On my machine, using Spring Boot 3.3.7, the sample starts in around 30 seconds:

2024-12-19T14:57:53.740Z  INFO 16601 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 29.585 seconds (process running for 29.994)

With the optimisation in place it's less than 7 seconds:

2024-12-19T14:54:25.793Z  INFO 15157 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 6.101 seconds (process running for 6.582)

The tests all pass with the optimisation in place so I am hopeful that it doesn't break anything functionally. I am, however, a little wary that it may regress the binder's performance in other scenarios so I'd like to discuss it with the rest of the team before we proceed.

@dmitriytsokolov If you'd like to try things out in your real-world app, it would be very interesting to know the impact that the change has. You should be able to copy the modified Binder source into your app and it'll be preferred to the one in the spring-boot jar. I'd recommend doing so with Boot 3.3.7 to avoid any other incompatibilities.

[dmitriytsokolov]

Hi @philwebb . Thanks a lot for the feedback!

I just tried 3.5.0-M3 and it works the same as 2.7/3.1 versions:

2025-03-24T22:15:34.747+01:00  INFO 45342 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 85.613 seconds (process running for 85.982)

are you sure you haven't deleted/changed app config or file(s)?

[philwebb]

Damn, you're too quick! The SNAPSHOTs for the updates haven't yet been pushed to artifactory. When this CI job passes can you please try again (running mvn -U to force a SNAPSHOT update).

[dmitriytsokolov]

haha, sure, I will try later 🤣
Anyway, it seems that if the fixes work as you mentioned above - there is only one way for us to fix this: bump spring to the latest.
There is no way those can be added to 2.7.x, right?

[philwebb]

I'm afraid the changes are too risky to backport. 2.7 is also out of OSS support so you should really be thinking about upgrading anyway.

[dmitriytsokolov]

@philwebb I just checked with the new SNAPSHOT and still see +- the same results. I'm attaching a screenshot where you can see that I'm using the recent version with your latest commit.

What I'm doing wrong?

I'm using Mac M1

[philwebb]

Is this the same demo application? Can you attach another zip if so. I'll check it again when I'm back at my desk.

[dmitriytsokolov]

Yes, the same (as in this message). Here it is:

demo_config_props_3.zip

[philwebb]

I need to add the following to make things import correctly, but after that I still see the improvement:

  <repositories>
    <repository>
      <id>spring-milestones</id>
      <name>Spring Milestones</name>
      <url>https://repo.spring.io/milestone</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </repository>
    <repository>
      <id>spring-snapshots</id>
      <name>Spring Snapshots</name>
      <url>https://repo.spring.io/snapshot</url>
      <releases>
        <enabled>false</enabled>
      </releases>
    </repository>
  </repositories>
  <pluginRepositories>
    <pluginRepository>
      <id>spring-milestones</id>
      <name>Spring Milestones</name>
      <url>https://repo.spring.io/milestone</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
    </pluginRepository>
    <pluginRepository>
      <id>spring-snapshots</id>
      <name>Spring Snapshots</name>
      <url>https://repo.spring.io/snapshot</url>
      <releases>
        <enabled>false</enabled>
      </releases>
    </pluginRepository>
  </pluginRepositories>

Can you double check that your jar has a org.springframework.boot.context.properties.bind.Binder class that includes a new cache field?

[philwebb]

Is that 67 seconds figure accurate as well? Even before these changes I can run the demo application in about 4-5 seconds. Is there anything specific to your mac that might slow the application down? Antivirus for example?

[philwebb]

Running the original demo zip I also see some improvements, although not as dramatic as Andy's

2025-03-24T16:42:43.072-07:00  INFO 88502 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 13.555 seconds (process running for 13.728)

vs

2025-03-24T16:44:18.182-07:00  INFO 88536 --- [demo_config_props] [           main] c.e.d.DemoConfigPropsApplication         : Started DemoConfigPropsApplication in 20.074 seconds (process running for 20.257)

[dmitriytsokolov]

yes, it has both
private final Map<Object, Object> cache = new ConcurrentReferenceHashMap<>();
and
private ConfigurationPropertyCaching configurationPropertyCaching;

Numbers should be accurate. Also I don't see this slowness in other apps/services. Initially, I provided a slightly different version of the app and Andy Wilkinson got the same results as me (here)

[philwebb]

Oh, it might be me. I don't have Lombok installed and I think it's not binding anything because of that. Let me dig a bit more and get someone else from the team to also try the demo.

[philwebb]

Well that's disappointing. It was Lombok missing from my IDE. Back to the drawing board :(

[dmitriytsokolov]

I can imagine how frustrating that must be 😢

[philwebb]

I've pushed 2 more updates with #44867 and #44868, with those the app (delomboked) starts in 10.285 seconds for me (down from 22.919).

I can't find much more to optimize and I think the sheer volume of entries is probably overwhelming the Binder which wasn't really designed with this in mind.

@dmitriytsokolov Can you let me know if the updates make any difference for you?

[dmitriytsokolov]

Wow! I've got ~2.5 times better performance! Amazing work, @philwebb. Thanks a lot!

Answering to your previous question:

...but I'd be interested to know if using the latest 3.5 SNAPSHOT removes the need for the extension points your looking for?

Ideally, I'd like to have a possibility to extend the binder and/or related beans, so that I can apply Andy's patch, slightly update our code, and get even better performance because this part is important for a couple of our services.
My assumptions might be wrong, but opening these beans would be a step toward spring's philosophy when you can intervene in any part of the spring and change it if necessary, wouldn't it?

[dmitriytsokolov]

Also, I will try to test the new version with the real app to confirm performance improvement, but it will take some time because I will need to bump spring in a couple of starters and services

[philwebb]

but opening these beans would be a step toward spring's philosophy when you can intervene in any part of the spring and change it if necessary, wouldn't it?

It's a bit of a tricky balancing act because we've found that sometimes when we open things up it becomes very hard to make changes to that area of the code. The binder is certainly one of the more complex areas of the codebase and can be a frequent source of bugs.

I have a branch that applied Andy's patch on top of mine, but I didn't find it made a lot of difference with the sample I was running. Let me try again and run the original sample. Perhaps we can merge that one as well.

[philwebb]

Another option if we don't open up the classes is for you to maintain a local copy of the binder and use it directly. You should be able to register or replace the ConfigurationPropertiesBindingPostProcessor bean with one that uses your fork.

You might even be able to do that with a 2.7 application, depending on the version of Java that you run.

[dmitriytsokolov]

@philwebb I see, thanks for the explanation.
Andy's fix works when there are multiple property sources, you can find example of such app config file in the original zip. Other classes and profile config files are identical.

[dmitriytsokolov]

You should be able to register or replace the ConfigurationPropertiesBindingPostProcessor bean with one that uses your fork.

I think I looked into this class back then, but as far as I remember it will be a tricky "fork". As for me it's worth dedicating that effort to updating the spring. But anyway thanks for the hint, I will take a look one more time

[dmitriytsokolov]

@philwebb I've updated Spring to the latest 3.5.0-SNAPSHOT in one of my real services and got a 30-40% performance improvement. It's not as massive as in the test app because there is complex business logic that is executed during hot refresh, but the improvement in the binding process itself is impressive. Thanks a lot for your work!