Auto-configuration for RestClient, WebClient and generated @HttpExchange proxies

[ch4mpy]

Spring Cloud offers some auto-configuration for its @FeignClient which entered maintenance mode in favor of RestClient and WebClient used with HttpServiceProxyFactory for @HttpExchange.

The new solution provides similar declarative REST client features, at the price of quite some Java conf especially when request authorization is involved - which should almost always be the case.

I experienced @HttpExchange proxies auto-configuration using application properties in this starter of mine and I think that the features I implemented are worth integrating into the "official" framework because they greatly improve developers' experience.

Sample

Use case

Let's consider the pool of oauth2ResourceServer microservices from this sample repository.

the 3 different declinations of the MicroserviceChouette*Application call:

• MicroserviceMachinApplication  on behalf of the resource owner at the origin of the requests: the requests are authorized re-using the Bearer token in the security-context (MicroserviceChouetteApplication is a resource server, so the request it processes already is authorized with a Bearer token).
• MicroserviceBiduleApplication in their own names: a new Bearer token is acquired using client-credentials flow.

The MicroserviceMachinApplication exposes an OpenAPI document from which we can generate the following:

@HttpExchange
public interface MachinApi {
  @GetExchange("/truc")
  String getTruc();
}

The MicroserviceBiduleApplication exposes an OpenAPI document from which we can generate the following:

@HttpExchange
public interface BiduleApi {
  @GetExchange("/chose")
  String getChose();
}

MicroserviceChouetteApplication collaborates with the two REST APIs above as follows:

@RestController
@RequiredArgsConstructor
public class ChouetteController {
  private final MachinApi machinApi;
  private final BiduleApi biduleApi;

  @GetMapping("/chouette-truc")
  public String getChouetteTruc() {
    return machinApi.getTruc();
  }

  @GetMapping("/chouette-chose")
  public String getChouetteChose() {
    return biduleApi.getChose();
  }
}

This requires implementations for MachinApi and BiduleApi to be exposed as beans, internally using a RestClient or WebClient instance to retrieve REST resources from other services - authorizing these requests with Bearer tokens.

Common security configuration

issuer: https://oidc.c4-soft.com/auth/realms/rest-showcase
bidule-api-port: 8081
machin-api-port: 8082

server:
  port: 8083

spring:
  application:
    name: bidule-api
  security:
    oauth2:
      client:
        provider:
          sso:
           issuer-uri: ${issuer}
        registration:
          bidule-registration:
            provider: sso
            authorization-grant-type: client_credentials
            client-id: chouette-api
            client-secret: change-me
            scope: openid
      resourceserver:
        jwt:
          issuer-uri: ${issuer}

REST configuration with just "official" 3.4.0-RC1 starters

I believe that we can hardly be more synthetic than the following for having MachinApi and BiduleApi implementations generated by HttpServiceProxyFactory, using RestClient instances configured with the required ClientHttpRequestInterceptors:

bidule-base-uri:  http://localhost:${bidule-api-port}
machin-base-uri:  http://localhost:${machin-api-port}

@Configuration
public class RestConfiguration {

  @Bean
  RestClient machinClient(@Value("${machin-base-uri}") URI machinBaseUri) {
    return RestClient.builder().baseUrl(machinBaseUri)
        .requestInterceptor(forwardingClientHttpRequestInterceptor()).build();
  }

  @Bean
  MachinApi machinApi(RestClient machinClient) {
    return HttpServiceProxyFactory.builderFor(RestClientAdapter.create(machinClient)).build()
        .createClient(MachinApi.class);
  }

  @Bean
  RestClient biduleClient(@Value("${bidule-base-uri}") URI biduleBaseUri,
      OAuth2AuthorizedClientManager authorizedClientManager,
      OAuth2AuthorizedClientRepository authorizedClientRepository) {
    return RestClient.builder().baseUrl(biduleBaseUri)
        .requestInterceptor(registrationClientHttpRequestInterceptor(authorizedClientManager,
            authorizedClientRepository, "bidule-registration"))
        .build();
  }

  @Bean
  BiduleApi biduleApi(RestClient biduleClient) {
    return HttpServiceProxyFactory.builderFor(RestClientAdapter.create(biduleClient)).build()
        .createClient(BiduleApi.class);
  }

  ClientHttpRequestInterceptor forwardingClientHttpRequestInterceptor() {
    return (HttpRequest request, byte[] body, ClientHttpRequestExecution execution) -> {
      final var auth = SecurityContextHolder.getContext().getAuthentication();
      if (auth != null && auth.getPrincipal() instanceof AbstractOAuth2Token oauth2Token) {
        request.getHeaders().setBearerAuth(oauth2Token.getTokenValue());
      }
      return execution.execute(request, body);
    };
  }

  ClientHttpRequestInterceptor registrationClientHttpRequestInterceptor(
      OAuth2AuthorizedClientManager authorizedClientManager,
      OAuth2AuthorizedClientRepository authorizedClientRepository, String registrationId) {
    final var interceptor = new OAuth2ClientHttpRequestInterceptor(authorizedClientManager);
    interceptor.setClientRegistrationIdResolver((HttpRequest request) -> registrationId);
    interceptor.setAuthorizationFailureHandler(
        OAuth2ClientHttpRequestInterceptor.authorizationFailureHandler(authorizedClientRepository));
    return interceptor;
  }
}

Things get even more complicated if the ClientHttpRequestFactory needs configuration for connect timeout, read timeout, or HTTP or SOCKS proxy (reach a remote service like Google API).

REST configuration with spring-addons-starter-rest

The RestConfiguration becomes:

com:
  c4-soft:
    springaddons:
      rest:
        client:
          machin-client:
            base-url: ${machin-base-uri}
            authorization:
              oauth2:
                forward-bearer: true
          bidule-client:
            base-url: ${bidule-base-uri}
            authorization:
              oauth2:
                oauth2-registration-id: bidule-registration

@Configuration
public class RestConfiguration {
  @Bean
  BiduleApi biduleApi(RestClient biduleClient) throws Exception {
    return new RestClientHttpExchangeProxyFactoryBean<>(BiduleApi.class, biduleClient).getObject();
  }

  @Bean
  MachinApi machinApi(RestClient machinClient) throws Exception {
    return new RestClientHttpExchangeProxyFactoryBean<>(MachinApi.class, machinClient).getObject();
  }
}

Features

What is already implemented:

• Expose RestClient or WebClient named beans preconfigured with:
    - A base URI that is likely to change from one deployment to another.
    - Request authorization with a choice of Basic and Bearer, and for the latter, the choice of forwarding the token in the security context of a resource server, or obtained using an OAuth2 client registration ID.
    - Set Proxy-Authorization header and configure a ClientHttpRequestFactory for HTTP or SOCKS proxy. Enabled by default if the standard HTTP_PROXY and NO_PROXY environment variables or custom application properties are set, but can be disabled on any auto-configured client.
• Clients are RestClient by default in servlets and WebClients in Webflux apps, but any client can be switched to WebClient in servlets.
• Choice to expose the builder instead of an already built RestClient or WebClient. This can be useful when some more configuration is needed than what the starter implements.
• The default REST client bean name is the camelCase version of its ID in properties (with Builder suffix if expose-builder=true). A custom name can be defined in properties

Room for improvement: remove the need for the generated @HttpExchange proxies beans definition in the RestConfiguration. I haven't found how to properly post-process the BeanDefinitionRegistry. The additional properties could look like the following:

        service:
          machin-api:
            client-bean-name: machinClient
            http-exchange-class: com.c4soft.showcase.rest.MachinApi
          bidule-api:
            client-bean-name: biduleClient
            http-exchange-class: com.c4soft.showcase.rest.BiduleApi

The great point of using a client bean name (rather than a key under the client properties), is that it allows to use any REST client, which could be a bean exposed using an auto-configured builder or a completely hand-crafted one.

Additional context

I already asked for this in Spring Security issues. @sjohnr wrote that such auto-configuration requests better fit here, and also that this shouldn't be implemented in the "official" framework because this would be "programming with yaml".

I have a different opinion about such auto-configuration. To me, it is about:

• Meeting the DRY principle. I don't want to repeat such Java configuration in each of my resource servers collaborating with other REST API(s).
• Limiting static configuration in application code.
• Flattening the learning curve and improving developer productivity. IDEs autocompletion and documentation features help much more when writing YAML configuration than when implementing Java configuration as the one above, which is far from trivial: we have to remember the name of several classes, which factory to use, which default property to override, which configuration trick is available in servlets or reactive apps, etc.
• Reducing the impact of breaking changes with future Spring Security and Spring Web(flux) versions.

My starter is just fine for me and the (very) few teams getting to know it and accepting to use it, but I'm sure that many more would be glad to benefit from such auto-configuration using just the official Boot starters.

[philwebb]

Thanks for the detailed write up. Issues #21322 and #31337 are somewhat related.

[sjohnr]

Thanks for moving this topic over here @ch4mpy, and thanks for being willing to provide detailed feedback.

I already asked for this in Spring Security issues. @sjohnr wrote that such auto-configuration requests better fit here, and also that this shouldn't be implemented in the "official" framework because this would be "programming with yaml".

I have a different opinion about such auto-configuration.

I haven't used the spring-addons project so my thoughts about it are mostly at the conceptual level. Having said that, I think that providing very detailed auto-configuration based solely on properties is only one way of providing convenience features like this. Sometimes, it will solve a particular problem very well. Other times, it might struggle to meet the requirements of a particular use case. When it fails to meet a particular requirement, I imagine it could quickly break down and require quite a pivot to a different way of configuring the application that is night and day from where the "yaml only" configuration began.

Because of that, I imagine that it might lead to a particular style of architecting, arranging, or configuring applications to suit preference. I wonder whether such an opinionated set of arrangements would fit well inside of Spring Boot, or if instead they are best suited to what I would call a meta-framework (like spring-addons)?

IDEs autocompletion and documentation features help much more when writing YAML configuration than when implementing Java configuration as the one above, which is far from trivial: we have to remember the name of several classes, which factory to use, which default property to override, which configuration trick is available in servlets or reactive apps, etc.

Any time verbose configuration of technical components is required there is some difficulty and complexity. However, YAML is not the only place where such developer productivity improvements are gained, nor is it always capable of expressing the entire application's configuration. So again, I think this becomes about preference.

Room for improvement: remove the need for the generated @HttpExchange proxies beans definition in the RestConfiguration. I haven't found how to properly post-process the BeanDefinitionRegistry. The additional properties could look like the following:

        service:
          machin-api:
            client-bean-name: machinClient
            http-exchange-class: com.c4soft.showcase.rest.MachinApi
          bidule-api:
            client-bean-name: biduleClient
            http-exchange-class: com.c4soft.showcase.rest.BiduleApi

This seems very similar actually (in my mind at least) to what we have/had with XML configuration using namespaces. I don't know that this is really an intended paradigm for auto-configuration features of Spring Boot. I'm open to hearing other perspectives on the matter.

In any case, I think conversation on #31337 illustrates quite well how much balance is required in finding general solutions for reducing boilerplate configuration. In such cases, if we can make it easier to produce meta-frameworks that tailor to a particular configuration preference, I think it benefits the entire community. Large companies which tend to produce their own meta-frameworks have an easier time maintaining them, and small companies and separate open source projects can more easily experiment and iterate on ideas and even easily change directions when needed.

---

Regarding this issue, I'd be in favor of seeing a request like this broken down into some more discrete building blocks that ease or simplify multiple types of configuration approaches (not just configuration properties-based). So my question would be, are there any building blocks (components, factory methods, etc.) that might improve the situation for producing auto-configuration for @HttpExchange proxies with OAuth2 that are not already being discussed in #31337?

[ch4mpy]

Any time verbose configuration of technical components is required there is some difficulty and complexity.

Shouldn't frameworks precisely aim at reducing verbosity, difficulty, and complexity?

To me, configuring clients request authorization with a Basic or Bearer header shouldn't be verbose, difficult, or complex. Basic auth needs no more than a pair of credentials and Bearer auth a token that we can get only from the security context or using a client registration. With the YAML above, we provide just that. The Java configuration is quite more involved.

I have the same observation for HTTP proxy configuration: this should be a snap, especially when the HTTP_PROXY and NO_PROXY environment variables are correctly set. I have about 200 lines of code for that.

Ideally, selecting the kind of authorization to use (Basic or Bearer) and providing its details (credentials or access token source) would be independent of the underlying client implementation and framework version. Same for HTTP proxy configuration and stuff like connection & read timeouts. The YAML above achieves such a de-coupling and so do most of the existing "official" Boot properties.

When it fails to meet a particular requirement, I imagine it could quickly break down and require quite a pivot to a different way of configuring the application that is night and day from where the "yaml only" configuration began

Given "my" starter offers the option to expose the builders instead of already built instances, one can take auto-configuration to the point he likes and add his custom needs the exact same way he would without auto-configuration: as it is already documented in Spring Security or Spring Web documentation.

building blocks (components, factory methods, etc.) that might improve the situation for producing auto-configuration for @HttpExchange proxies with OAuth2 that are not already being discussed in #31337?

As I understand this other ticket, it is about auto-detecting @HttpExchange interfaces and creating the proxies with no more configuration than a base package (not even YAML). The "real world" use cases I have at hand break this approach because of their needs for the underlying client configuration: client requests almost always need authorization, a single client per application is rarely enough, and configuring a client per @HttpExchange proxy would be cumbersome.

So far, I focused on RestClient and WebClient auto-configuration because it's where I had important verbosity and complexity gains.

Here are some of the building blocks I wrote to ease clients auto-configuration:

• a ClientHttpRequestInterceptor to authorize RestClient requests with the Bearer in the security context of a resource server
• an OAuth2ClientHttpRequestInterceptor factory requiring no more than the ID of the registration to use
• ExchangeFilterFunction factories requiring no more than the ID of the registration to use (servlet and WebFlux)
• a ClientHttpRequestInterceptor and an ExchangeFilterFunction to set Basic authorization
• a support class to help merging the "standard" HTTP_PROXY and NO_PROXY environment variables with custom environment properties
• a ClientHttpRequestFactory which conditionally applies proxy configuration to a request (proxy is not set when the request URI matches a nonProxyHostsPattern)

All are in the spring-addons-starter-rest sources

[sjohnr]

Here are some of the building blocks I wrote to ease clients auto-configuration:

Thanks @ch4mpy, that's helpful.

I don't want to speak for the Boot team, but I think general support from Spring Boot for some of those items might still be tricky. I like the idea of enhancing builders, but I don't yet see configuration properties as a general solution.

I'll spend some time thinking about this from a Spring Security perspective as I have some ideas that could be a middle ground. I'll update this issue with those ideas once they're fully formed. This may take a bit so please be patient with me.

[philwebb]

from Spring Boot for some of those items might still be tricky

That's my initial feeling as well, but I must admit I haven't had the time yet to look too closely. We'll wait until we get feedback from @sjohnr about the Spring Security side before we do anything in Boot.

[ch4mpy]

@sjohnr In my comment above, I pointed to the master branch of spring-addons. That was a mistake as what I mentioned changed a lot in the branch I created for Boot 3.4 and Security 6.4.

I updated the link, but if you have a look at the source of the building blocks I listed, please be sure to read from the 7.9 branch.

[ch4mpy]

@philwebb some building blocks aren't directly related to Spring Security and might be investigated in parallel. Notably:

• The HTTP proxy configuration. The ClientHttpRequestFactory I use to set the Proxy-Authorization header and to apply the NO_PROXY env variable (or non-proxy-pattern property) might be the most complicated "building block" (nothing scary).
• The bean definition registry post-processing to add the auto-configured RestClient and WebClient definitions. I'm beginning with this kind of bean factory manipulation and it's very likely that things can be improved there. However, it can be seen in these source files that the auto-configuration is not that tricky once the building blocks are there.

[ch4mpy]

I forgot to mention the main reason why I favored YAML (over the Java annotations asked in #31337) for REST clients and @HttpExchange proxies configuration.

In my use-cases, the @HttpExchange interfaces are almost always generated by the openapi-generator-maven-plugin from an OpenAPI spec. Decorating generated code seems a bad idea and extending these generated interfaces to add Java annotations is cumbersome.

Plus, YAML makes it easy to adapt a deployment to a target environment with things like a base URL, HTTP proxy, or read & connection timeouts.

[snicoll]

@sjohnr any feedback for us?

[sjohnr]

Hi @snicoll!

The only feedback I can give at this point is that due to how Spring Security-specific it is, I don't see this as something that would exist directly in Spring Boot. If something like this exists somewhere, I think it would be best done on top of Boot. However, such a place doesn't exist (in the Spring portfolio) currently.

I will say that we have been prototyping some ideas that involve more general configuration use cases with Spring Boot, auto-configuration, and configuration properties, but they do not currently address this specific issue. We might in the future look at this issue and take inspiration from it as part of a theme for configuring OAuth2 Client with auto-configuration, but that is not very far along yet in terms of design as it's a bit bigger effort to come up with something. If we get farther along and have a place for such things, I think this issue would be more applicable over there.

Also, if this issue resonated strongly with many users, I would expect more feedback on it by now. The lack of feedback suggests this isn't a priority for users at this point. Regardless of where we go from here, we'll definitely keep this issue in mind. However, I don't have any concrete action items to suggest at this time.

Does the Boot team have any additional thoughts on this (and/or my earlier comment)?

[ch4mpy]

we have been prototyping some ideas that involve more general configuration use cases with Spring Boot, auto-configuration, and configuration properties

@sjohnr considering that you're recommending to discard this feature request, is there any place where I can find discussion & code snippets? Maybe I can find there ideas to improve "my" Boot starters as I might have to maintain it on the long term.

if this issue resonated strongly with many users, I would expect more feedback on it by now

I'm not convinced that a single feature request on one of the many Spring issue trackers is the best place to get representative feedback from representative Spring Boot users. This other ticket shows some interest from the community for @HttpExchange proxies auto-configuration to replace @FeignClient.

What would @HttpExchange proxies auto-configuration be about if it does not cover the RestClient / WebClient? And if it does, how useful can it be if request authorization isn't covered? Same thing for the underlying HTTP client tuning (HTTP proxy, timeouts, SSL certificates validation, etc.).

I cannot count the times I was told that my starters solve problems, but are a no-go because they do not have the "Spring label".

@snicoll, I beg you, please have someone you trust writing micro-services daily - and deploying them to production - (or not "micro", but still collaborating with several REST APIs) evaluate how much time and burden can be saved using something similar to spring-addons-starter-rest (published on maven-central and documented there) when requirements go beyond those of a 15 minutes tutorial. By that I mean needing some of the following:

• configuration of the base URL of an API in properties: it is very likely to change from one deployment to another
• request authorization, either using:
  • an OAuth2 client registration
  • forwarding the Bearer token in the context of a resource server (send a sub-request on behalf of the resource owner of the parent request)
  • static API key
  • Basic auth
• configuration of connection and read timeouts
• going through an HTTP proxy. This can be declined in environments with the HTTP_PROXY and NO_PROXY variables and those without it but with application properties.
• disabling SSL certificates validation for RestClients used to query servers on a local network with self-signed certicates not registered with JRE cacerts.
• switching the ClientHttpRequestFactory implementation used by RestClient between JdkClientHttpRequestFactory, HttpComponentsClientHttpRequestFactory, and JettyClientHttpRequestFactory. This one can seem strange; until you have to work with Microsoft middleware that doesn't support standard headers automatically set by the JDK's HTTP client, or want to easily disable SSL certificates hostname verification for some REST clients (but not do it globally)...