Wrapping results in Hashie::Mash 3.5.1 causes logs to be flooded with warnings

[lparry]

Originally opened on the wrong repo: elastic/elasticsearch-ruby#398

I updated our bundle today which included hashie@3.5.1, and found our logs are now completely overwhelmed with warnings:

WARN -- : You are setting a key that conflicts with a built-in method Hashie::Mash#sort defined in Enumerable. This can cause unexpected behavior when accessing the key via as a property. You can still access the key via the #[] method.

I believe this is being caused by the "sort" key that comes back in the response body. eg: "sort":[1.121931762,1477653971450]. If I remove any use of sort from the query the warnings go away.

we're doing our searching via elasticsearch/persistence/repository if that has any bearing on things

I suspect it's related to this PR: hashie/hashie#381. I don't have a good suggestion how it might be fixed other than to stop using Hashie, but that seems like a pretty large change.

---

Moved from elastic/elasticsearch-ruby#398

[karmi]

Yes, this will be tricky to solve, because Hashie::Mash is used as a "freeform" wrapper around deeply structured Hashes coming from Elasticsearch, enabling method access to the structures.

On the original ticket, @gfx suggested using Struct. That would be technically possible, I guess, but crazily obnoxious -- Hashie::Mash is used as a "free-form" wrapper around Hashes coming from Elasticsearch, no matter which internal structure they have, and when using Struct, we would have to define the structure first.

Using OpenStruct would seem as a workable replacement for Hashie::Mash, because you can pass a Hash to it and it will create method accessors. However, this does not apply to an embedded Hash:

o = OpenStruct.new foo: 'bar', moo: { koo: 'loo' }
o.foo
# => "bar"
o.moo
# => {:koo=>"loo"}
o.moo.koo
NoMethodError: undefined method `koo' for {:koo=>"loo"}:Hash

And so this is problematic, because what we do almost always is something like response.aggregations.mytags.buckets.first.etc.etc.etc.

Let's keep the issue open, and let's gather feedback.

What should work is to somehow wrap Hashie::Mash in another class, and define the offending method there -- but I can't go checking the source now.

[zealot128]

Workaround for now, setting

Hashie.logger = Logger.new('/dev/null')

in an initializer silences that output.

The best fix would be to replace all "Hashie::Mash.new" with an own subclassed Hashie::Mash like Omniauth did:
https://github.com/omniauth/omniauth/pull/874/files

[michaelherold]

I released Hashie 3.5.2 today that should allow you to more easily handle this.

As @zealot128 said, if you subclass Mash you can access the new disable_warnings method to disable warnings. I would recommend bumping the Hashie requirement to 3.5.2+ to make it so you don't have to do the torturous method that I used in OmniAuth due to their supporting old versions of Hashie.

Please let me know if I can be of assistance on this.

[karmi]

Hi Michael, many thanks for the info! I think we will go through the subclass route, using the disable_warnings method you've pointed out.

I've also tried some sketches with more acrobatics to remove the sort key from the arguments being passed to initializer, so we bypass the warnings on older versions, but had to abandon it for a while. I'll post the code here if I'm successful later.

(Regarding bumping the requirement, I'm usually very conservative with that, notably for popular libraries like Hashie, because you never know what other gems are people using, and a strict requirement might very easily put them into an un-resolvable gem version specification...)

Thanks again!

[karmi]

@lparry, thanks for the initial report and debug info!

@michaelherold, many thanks for the pointers and the quick release of a fixed version!

I've added a HashWrapper subclass in 5732fa3, and used it as the wrapper across the elasticsearch-model and elasticsearch-persistence gems. As I've said, I've not defined the version requirement because of worries it might get people into dependency hell situations. I guess most people will upgrade Hashie to the fixed version sooner than later anyway.

Please try out the version in master, it should be quiet now.

[michaelherold]

That looks good, @karmi! The only versions of Hashie where that won't fully fix the problem are 3.5.0 - 3.5.2. The new 3.5.3 will handle that fine. There were a bunch of hurdles to jump because we didn't have good integration tests in place. Hopefully, that won't cause too many issues since most version constraints that get people to 3.5.x will allow them to get to 3.5.3 without trouble.

I'll add it to the to-do list to get integration tests in place for the Elastic gems since I know a lot of people transitively use Hashie through your projects.

Please let us know if you run into any more trouble!

[lparry]

I've tested master against our codebase and this does appear to fix the excessive logging. thanks! any chance you could push a 0.1.10 version of the elasticsearch-persistence gem up to rubygems when you get a chance please?

[carolliu]

Hi, will the fix to be released?