Exclude the value of a certain key, from getting encoded

[vanenshi]

Is there a way to exclude the value of a key from getting encoded?
I tried using a custom encode function, but that function runs on value and key separately, and there is no way to understand the key of a certain value inside it

[vanenshi]

one simple solution would be to include the prefix here as well

qs/lib/stringify.js

Lines 123 to 129 in 981ce09

	if (isNonNullishPrimitive(obj) || utils.isBuffer(obj)) {
	if (encoder) {
	var keyValue = encodeValuesOnly ? prefix : encoder(prefix, defaults.encoder, charset, 'key', format);
	return [formatter(keyValue) + '=' + formatter(encoder(obj, defaults.encoder, charset, 'value', format))];
	}
	return [formatter(prefix) + '=' + formatter(String(obj))];
	}

to this:

    if (isNonNullishPrimitive(obj) || utils.isBuffer(obj)) {
        if (encoder) {
            var keyValue = encodeValuesOnly ? prefix : encoder(prefix, defaults.encoder, charset, 'key',  format, undefined);
            return [formatter(keyValue) + '=' + formatter(encoder(obj, defaults.encoder, charset, 'value', format, prefix))];
        }
        return [formatter(prefix) + '=' + formatter(String(obj))];
    }

without introducing any breaking changes to the library

[ljharb]

Can you explain your use case?

[vanenshi]

@ljharb sure
I am using a private api that return me a "next token" on paginated results. This api encode this value, and when I send it through query params for the next page, qs encoded again. so I wanted to skip the value encode by using the key (prefix)

I know this is a very rare use case, but adding this simple parameter to custom encoder solved my problem. And it's not going to introduce a breaking change

This will give developers a new way to customize the behavior of the qs

[ljharb]

Can you provide an example token, so i can see the format involved?

[vanenshi]

Definitely

first request will return this payload:

"pagedData": {
    "nextToken": "%7b%22pk%22%3a%7b%22B%22%3anull%2c%22BOOL%22%3afalse%2c%22IsBOOLSet%22%3afalse%2c%22BS%22%3a%5b%5d%2c%22L%22%3a%5b%5d%2c%22IsLSet%22%3afalse%2c%22M%22%3a%7b%7d%2c%22IsMSet%22%3afalse%2c%22N%22%3anull%2c%22NS%22%3a%5b%5d%2c%22NULL%22%3afalse%2c%22S%22%3a%22messages%23b49dbe1eb0af4de89879bbdb553f825d%22%2c%22SS%22%3a%5b%5d%7d%2c%22sk%22%3a%7b%22B%22%3anull%2c%22BOOL%22%3afalse%2c%22IsBOOLSet%22%3afalse%2c%22BS%22%3a%5b%5d%2c%22L%22%3a%5b%5d%2c%22IsLSet%22%3afalse%2c%22M%22%3a%7b%7d%2c%22IsMSet%22%3afalse%2c%22N%22%3anull%2c%22NS%22%3a%5b%5d%2c%22NULL%22%3afalse%2c%22S%22%3a%221706985547187%22%2c%22SS%22%3a%5b%5d%7d%7d",
  }

and I have to send it as it is, which means a request like this

https://<API-Url>/chat/v1/rooms/<Room-Id>/messages?limit=5&nextToken=%7b%22pk%22%3a%7b%22B%22%3anull%2c%22BOOL%22%3afalse%2c%22IsBOOLSet%22%3afalse%2c%22BS%22%3a%5b%5d%2c%22L%22%3a%5b%5d%2c%22IsLSet%22%3afalse%2c%22M%22%3a%7b%7d%2c%22IsMSet%22%3afalse%2c%22N%22%3anull%2c%22NS%22%3a%5b%5d%2c%22NULL%22%3afalse%2c%22S%22%3a%22messages%23b49dbe1eb0af4de89879bbdb553f825d%22%2c%22SS%22%3a%5b%5d%7d%2c%22sk%22%3a%7b%22B%22%3anull%2c%22BOOL%22%3afalse%2c%22IsBOOLSet%22%3afalse%2c%22BS%22%3a%5b%5d%2c%22L%22%3a%5b%5d%2c%22IsLSet%22%3afalse%2c%22M%22%3a%7b%7d%2c%22IsMSet%22%3afalse%2c%22N%22%3anull%2c%22NS%22%3a%5b%5d%2c%22NULL%22%3afalse%2c%22S%22%3a%221706985547187%22%2c%22SS%22%3a%5b%5d%7d%7d

using this code

const listOfExcludedKeys = ['nextToken'];

const str = stringify(params, {
        encoder(str, defaultEncoder, charset, type, format, prefix) {
          // parameters like  'nextToken' are already encoded
          // we need to skip them.
          if (prefix && listOfExcludedKeys.includes(prefix)) return str;

          return defaultEncoder(str, defaultEncoder, charset);
        },
      });

but if i disable my customer encoder, this will be the result:

https://<API-Url>/chat/v1/rooms/<Room-Id>/messages?limit=5&nextToken=%257b%2522pk%2522%253a%257b%2522B%2522%253anull%252c%2522BOOL%2522%253afalse%252c%2522IsBOOLSet%2522%253afalse%252c%2522BS%2522%253a%255b%255d%252c%2522L%2522%253a%255b%255d%252c%2522IsLSet%2522%253afalse%252c%2522M%2522%253a%257b%257d%252c%2522IsMSet%2522%253afalse%252c%2522N%2522%253anull%252c%2522NS%2522%253a%255b%255d%252c%2522NULL%2522%253afalse%252c%2522S%2522%253a%2522messages%2523b49dbe1eb0af4de89879bbdb553f825d%2522%252c%2522SS%2522%253a%255b%255d%257d%252c%2522sk%2522%253a%257b%2522B%2522%253anull%252c%2522BOOL%2522%253afalse%252c%2522IsBOOLSet%2522%253afalse%252c%2522BS%2522%253a%255b%255d%252c%2522L%2522%253a%255b%255d%252c%2522IsLSet%2522%253afalse%252c%2522M%2522%253a%257b%257d%252c%2522IsMSet%2522%253afalse%252c%2522N%2522%253anull%252c%2522NS%2522%253a%255b%255d%252c%2522NULL%2522%253afalse%252c%2522S%2522%253a%25221706985547187%2522%252c%2522SS%2522%253a%255b%255d%257d%257d

it's getting double encoded

[ljharb]

Could you decode the key and then when qs re-encodes it, it'd be what you want?

Perhaps using the filter option?

[vanenshi]

That was my first solution @ljharb, and it was working perfectly
But it was only going to solve this specific use case.
the reason I opened this issue was to propose a new feature, that might solve other use cases too

[ljharb]

It's useful to note the current workaround, at least.

Presumably this API is sending a pre-encoded value in the response - why are they doing that? Generally I'd expect an API to only use encoding on the outer level of a response.

[vanenshi]

Yep, what you are saying is totally correct, but sometimes we are dealing with external APIs that we have no control over. having a library that lets me customize the functionality is great

[Atomic71]

@vanenshi I was looking for the option you're proposing exactly, to prevent the encoding of "cursor" sent with params, that also gets returned from the API.

The cursor looks like this: WyJuZXh0gIjIwIiwMjQtMTAtDRUMTA6MMTU6MzEuNDQ1NTE5IihgMjIxET1=, which encodes the = to %3D.

The fact that there wasn't such an option, which to me seemed like a common option to have for a library of this type, it made me think that I'm doing something very deviant from the norm.

[vanenshi]

Same here @Atomic71, I end up adding this feature as a patch to our codebase