Transport: Add the CACert configuration option to the client and transport

When the option is set, the transport will perform the neccessary ceremony
for adding the custom certificate authority to the TLSClientConfig.

It returns an error for transports other than http.Transport.

An executable example has been added to _examples/security/tls_with_ca.go.

Closes #106

Author: karmi

README.md

@@ -106,7 +106,7 @@ log.Println(res)
 When you export the `ELASTICSEARCH_URL` environment variable,
 it will be used to set the cluster endpoint(s). Separate multiple adresses by a comma.
 
-To set the cluster endpoint(s) programatically, pass them in the configuration object
+To set the cluster endpoint(s) programatically, pass a configuration object
 to the `elasticsearch.NewClient()` function.
 
 ```golang
@@ -115,34 +115,55 @@ cfg := elasticsearch.Config{
     "http://localhost:9200",
     "http://localhost:9201",
   },
+  // ...
 }
 es, err := elasticsearch.NewClient(cfg)
-// ...
 ```
 
-To configure the HTTP settings, pass a [`http.Transport`](https://golang.org/pkg/net/http/#Transport)
-object in the configuration object (the values are for illustrative purposes only).
+To set the username and password, include them in the endpoint URL,
+or use the corresponding configuration options.
+
+```golang
+cfg := elasticsearch.Config{
+  // ...
+  Username: "foo",
+  Password: "bar",
+}
+```
+
+To set a custom certificate authority used to sign the certificates of cluster nodes,
+use the `CACert` configuration option.
+
+```golang
+cert, _ := ioutil.ReadFile(*cacert)
+
+cfg := elasticsearch.Config{
+  // ...
+  CACert: cert,
+}
+```
+
+To configure other HTTP settings, pass an [`http.Transport`](https://golang.org/pkg/net/http/#Transport)
+object in the configuration object.
 
 ```golang
 cfg := elasticsearch.Config{
   Transport: &http.Transport{
     MaxIdleConnsPerHost:   10,
     ResponseHeaderTimeout: time.Second,
-    DialContext:           (&net.Dialer{Timeout: time.Second}).DialContext,
     TLSClientConfig: &tls.Config{
       MinVersion: tls.VersionTLS11,
       // ...
     },
+    // ...
   },
 }
-
-es, err := elasticsearch.NewClient(cfg)
-// ...
 ```
 
 See the [`_examples/configuration.go`](_examples/configuration.go) and
 [`_examples/customization.go`](_examples/customization.go) files for
 more examples of configuration and customization of the client.
+See the [`_examples/security`](_examples/security) for an example of a security configuration.
 
 The following example demonstrates a more complex usage. It fetches the Elasticsearch version from the cluster, indexes a couple of documents concurrently, and prints the search results, using a lightweight wrapper around the response body.
 
@@ -335,7 +356,7 @@ The `esutil` package provides convenience helpers for working with the client. A
 
 ## Examples
 
-The **[`_examples`](./_examples)** folder contains a number of recipes and comprehensive examples to get you started with the client, including configuration and customization of the client, mocking the transport for unit tests, embedding the client in a custom type, building queries, performing requests, and parsing the responses.
+The **[`_examples`](./_examples)** folder contains a number of recipes and comprehensive examples to get you started with the client, including configuration and customization of the client, using a custom certificate authority (CA) for security (TLS), mocking the transport for unit tests, embedding the client in a custom type, building queries, performing requests individually and in bulk, and parsing the responses.
 
 <!-- ----------------------------------------------------------------------------------------------- -->
 

_examples/security/README.md

@@ -0,0 +1,76 @@
+# Example: Security
+
+This example demonstrates how to use Transport Layer Security (TLS) for
+encrypting and verifying the communication with an Elasticsearch cluster
+by passing a custom certificate authority to the client.
+
+## Creating the certificates for the cluster nodes
+
+Generate the certificates using the
+[`elasticsearch-certutil`](https://www.elastic.co/guide/en/elasticsearch/reference/current/certutil.html)
+tool:
+
+```bash
+make certificates
+```
+
+> See the [_Encrypting communications in an Elasticsearch Docker Container_](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls-docker.html) tutorial for a complete overview.
+
+Start the cluster with full security configuration:
+
+```bash
+make cluster
+```
+
+See the [`elasticsearch-cluster.yml`](elasticsearch-cluster.yml) file for details.
+
+Use `curl` to verify access to the cluster:
+
+```
+curl --cacert certificates/ca/ca.crt https://elastic:elastic@localhost:9200
+```
+
+> NOTE: On Mac OS X, you may need to add the certificate to the Keychain with `security add-trusted-cert -p ssl certificates/ca/ca.crt`. To remove it, run `security remove-trusted-cert certificates/ca/ca.crt`.
+
+
+## Using the client configuration option
+
+To pass the certificate authority (CA) to the client, so it can verify the server certificate,
+use the `elasticsearch.Config.CACert` configuration option:
+
+```go
+// --> Read the certificate from file
+cert, _ := ioutil.ReadFile(*cacert)
+
+es, _ := elasticsearch.NewClient(
+	elasticsearch.Config{
+		// ...
+
+		// --> Pass the certificate to the client
+		CACert: cert,
+	})
+```
+
+Run the full example:
+
+```bash
+go run tls_with_ca.go
+
+# [200 OK] {
+# ...
+```
+
+
+## Manual transport configuration
+
+To configure the transport manually, use the
+`(*http.Transport).TLSClientConfig.RootCAs.AppendCertsFromPEM()` method.
+
+Run the full example:
+
+```bash
+go run tls_configure_ca.go
+
+# [200 OK] {
+# ...
+```

_examples/security/tls_with_ca.go

@@ -0,0 +1,57 @@
+// Licensed to Elasticsearch B.V. under one or more agreements.
+// Elasticsearch B.V. licenses this file to you under the Apache 2.0 License.
+// See the LICENSE file in the project root for more information.
+
+// +build ignore
+
+package main
+
+import (
+	"flag"
+	"io/ioutil"
+	"log"
+
+	"github.com/elastic/go-elasticsearch/v8"
+)
+
+func main() {
+	log.SetFlags(0)
+
+	var (
+		err error
+
+		// --> Configure the path to the certificate authority and the password
+		//
+		cacert   = flag.String("cacert", "certificates/ca/ca.crt", "Path to the file with certificate authority")
+		password = flag.String("password", "elastic", "Elasticsearch password")
+	)
+	flag.Parse()
+
+	// --> Read the certificate from file
+	//
+	cert, err := ioutil.ReadFile(*cacert)
+	if err != nil {
+		log.Fatalf("ERROR: Unable to read CA from %q: %s", *cacert, err)
+	}
+
+	es, err := elasticsearch.NewClient(
+		elasticsearch.Config{
+			Addresses: []string{"https://localhost:9200"},
+			Username:  "elastic",
+			Password:  *password,
+
+			// --> Pass the certificate to the client
+			//
+			CACert: cert,
+		})
+	if err != nil {
+		log.Fatalf("ERROR: Unable to create client: %s", err)
+	}
+
+	res, err := es.Info()
+	if err != nil {
+		log.Fatalf("ERROR: Unable to get response: %s", err)
+	}
+
+	log.Println(res)
+}

elasticsearch.go

@@ -37,6 +37,11 @@ type Config struct {
 	CloudID string // Endpoint for the Elastic Service (https://elastic.co/cloud).
 	APIKey  string // Base64-encoded token for authorization; if set, overrides username and password.
 
+	// PEM-encoded certificate authorities.
+	// When set, an empty certificate pool will be created, and the certificates will be appended to it.
+	// The option is only valid when the transport is not specified, or when it's http.Transport.
+	CACert []byte
+
 	RetryOnStatus        []int // List of status codes for retry. Default: 502, 503, 504.
 	DisableRetry         bool  // Default: false.
 	EnableRetryOnTimeout bool  // Default: false.
@@ -134,6 +139,8 @@ func NewClient(cfg Config) (*Client, error) {
 		Password: cfg.Password,
 		APIKey:   cfg.APIKey,
 
+		CACert: cfg.CACert,
+
 		RetryOnStatus:        cfg.RetryOnStatus,
 		DisableRetry:         cfg.DisableRetry,
 		EnableRetryOnTimeout: cfg.EnableRetryOnTimeout,
@@ -150,6 +157,9 @@ func NewClient(cfg Config) (*Client, error) {
 		Selector:           cfg.Selector,
 		ConnectionPoolFunc: cfg.ConnectionPoolFunc,
 	})
+	if err != nil {
+		return nil, fmt.Errorf("error creating transport: %s", err)
+	}
 
 	client := &Client{Transport: tp, API: esapi.New(tp)}
 

estransport/estransport.go

@@ -6,6 +6,8 @@ package estransport
 
 import (
 	"bytes"
+	"crypto/x509"
+	"errors"
 	"fmt"
 	"io"
 	"io/ioutil"
@@ -52,6 +54,8 @@ type Config struct {
 	Password string
 	APIKey   string
 
+	CACert []byte
+
 	RetryOnStatus        []int
 	DisableRetry         bool
 	EnableRetryOnTimeout bool
@@ -105,6 +109,22 @@ func New(cfg Config) (*Client, error) {
 		cfg.Transport = http.DefaultTransport
 	}
 
+	if cfg.CACert != nil {
+		httpTransport, ok := cfg.Transport.(*http.Transport)
+		if !ok {
+			return nil, fmt.Errorf("unable to set CA certificate for transport of type %T", cfg.Transport)
+		}
+
+		httpTransport = httpTransport.Clone()
+		httpTransport.TLSClientConfig.RootCAs = x509.NewCertPool()
+
+		if ok := httpTransport.TLSClientConfig.RootCAs.AppendCertsFromPEM(cfg.CACert); !ok {
+			return nil, errors.New("unable to add CA certificate")
+		}
+
+		cfg.Transport = httpTransport
+	}
+
 	if len(cfg.RetryOnStatus) == 0 {
 		cfg.RetryOnStatus = defaultRetryOnStatus[:]
 	}