Iterator on JSON objects

[Neradoc]

This adds 2 ways to iterate on an object's items following the APIs of dict().

• The main iterator is for key in obj it iterates over the keys without consuming or loading the values. The current key's value can then be accessed with obj[key].
• The TransientObject.items() method will iterate on item tuples fetching both key and value with for key, value in obj.items(). Because it fetches every value (be it a basic type or a Transient instance), it will use more memory.

So this works, although note that you can only get the key once unless it's a container too.

for key in stream["some_dict"]:
    if key == "needle":
        print(stream["some_dict"][key])

The new example json_stream_local_file_advanced.py uses the iterator to find keys that match a list of expected keys and returns a python dict that contains only those items. It's an example of how to retrieve a set of keys when the order is unknown.

I am not sure how to best document the iterator outside of the items() property.

[justmobilize]

@Neradoc will this also make it easy to implement:

for key, value in stream["some_dict"].items():

[Neradoc]

items() is implemented using the new _next_key() method and self[key].
Since it retrieves the values however, I advise using the other form when looking for a key.
It's still better memory-wise than loading the whole dictionary of course.

[tannewt]

Does the original library do this? https://github.com/daggaz/json-stream (Do we care if it doesn't?)

[Neradoc]

The json_stream library has iterators, but doesn't do obj[key] like in my example.
This works in both:

import json_stream
source = io.BytesIO(b'{"a":1,"b":2,"c":3}')
stream = json_stream.load(source)

for key, value in stream.items():
    print(key, value)

so does:

for key in stream:
    print(key)

This only works in this version:

source = io.BytesIO(b'{"a":1,"b":2,"c":3}')
stream = json_stream.load(source)

for key in stream:
    print(key, stream[key])

json_stream.base.TransientAccessException: a not found in transient JSON stream or already passed in this stream

I would argue that being able to loop through keys and access the value only when needed is a good feature to have. Granted in most cases the values might be numbers, small strings or Transient subclasses, but it could be a big string.

---

Another difference introduced in #7 is the ability to access stream["dict"] multiple times as long as we are still "in that dict". This fails with the json_stream library:

source = io.BytesIO(b'{"a":{"x":10,"y":20},"b":2,"c":3}')
stream = json_stream.load(source)
print(stream["a"]["x"])
print(stream["a"]["y"])

   stream["a"]["y"]
    ~~~~~~^^^^^
[...]
json_stream.base.TransientAccessException: a not found in transient JSON stream or already passed in this stream

That's mostly a convenience one, since we can always save stream["a"] in a variable rather than reuse the notation.
And it only applies if the value (stream["a"]) is another Transient container (list or dict). Which makes sense because we need to keep a (small) container object in memory (at this point in the stream) to be able to skip to the end of it, but we don't want to cache literal values, since again they could be big strings.

[tannewt]

Cool. Thank you!