This repository was archived by the owner on Jul 9, 2024. It is now read-only.
Switch hashing algorithm to Jump Consistent Hash for more uniform distribution #38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pair: @dcorbacho. Part of #37.
with more examples to come as time permits
michaelklishin
changed the title
|
Merging per verbal approvement from @dcorbacho. |
This was referenced Aug 20, 2018
michaelklishin
added a commit
that referenced
this pull request
Aug 21, 2018
In some environments, namely our Concourse containers, with *some* iterations of the test the value exceeds the reference value of p-value = 0.01. This may be specific to OTP 19.3 or certain platforms. This is not something that I can reproduce in a number of OTP 21 environments. References #37, #38.
michaelklishin
added a commit
that referenced
this pull request
Aug 21, 2018
michaelklishin
added a commit
that referenced
this pull request
Aug 21, 2018
…reorganise tests We still depend on the PRNG to provide a reasonably uniform distribution of inputs (e.g. routing keys) but things pass in at least 3 different environments reliably with 150K iterations. Pair: @dcorbacho. References #37, #38.
michaelklishin
added a commit
that referenced
this pull request
Aug 21, 2018
In some environments, namely our Concourse containers, with *some* iterations of the test the value exceeds the reference value of p-value = 0.01. This may be specific to OTP 19.3 or certain platforms. This is not something that I can reproduce in a number of OTP 21 environments. References #37, #38. (cherry picked from commit d7a89cd)
michaelklishin
added a commit
that referenced
this pull request
Aug 21, 2018
…reorganise tests We still depend on the PRNG to provide a reasonably uniform distribution of inputs (e.g. routing keys) but things pass in at least 3 different environments reliably with 150K iterations. Pair: @dcorbacho. References #37, #38. (cherry picked from commit e081baa)
11 tasks
michaelklishin
added a commit
that referenced
this pull request
Aug 24, 2018
There can be more than one bucket per queue, so we ended up with potential extra updates that resulted in incorrect (negative) bucket numbers. As part of this we considered using an alternative data model: folding two tables into one that looks like this: exchange => map(bucket => queue) This would greatly simplify binding management and be roughly as efficient for routing except for one thing: updating the rest of the ring will be a linear operation over all buckets. So that change alone would be insufficient but remains to be an idea to improve on in the future (e.g. by using a tree). Pair: @dcorbacho. References #37, #38. [#159822323]
michaelklishin
added a commit
that referenced
this pull request
Aug 28, 2018
This implementation is significantly simpler and doesn't perform nearly as many Mnesia operations. Pair: @dcorbacho. References #37, #38. [#159822323]
michaelklishin
added a commit
that referenced
this pull request
Aug 31, 2018
There can be more than one bucket per queue, so we ended up with potential extra updates that resulted in incorrect (negative) bucket numbers. As part of this we considered using an alternative data model: folding two tables into one that looks like this: exchange => map(bucket => queue) This would greatly simplify binding management and be roughly as efficient for routing except for one thing: updating the rest of the ring will be a linear operation over all buckets. So that change alone would be insufficient but remains to be an idea to improve on in the future (e.g. by using a tree). Pair: @dcorbacho. References #37, #38. [#159822323] (cherry picked from commit 5cab6eb)
michaelklishin
added a commit
that referenced
this pull request
Aug 31, 2018
This implementation is significantly simpler and doesn't perform nearly as many Mnesia operations. Pair: @dcorbacho. References #37, #38. [#159822323] (cherry picked from commit 0dba5a4)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed Changes
See #37 for background. This change makes the plugin provide a much improved distribution uniformity (proven by a Chi-squared test with p-value of < 0.01 as demonstrated in #37) at the cost of most costly binding and unbinding operations. More tests and documentation updates are WIP.
Pair: @dcorbacho.
Types of Changes
Checklist
CONTRIBUTING.mddocumentFurther Comments
Closes #37.
[#159822323]