Skip to content
This repository was archived by the owner on Sep 21, 2021. It is now read-only.

Commit 4907674

Browse files
polyfractalpolyfractal
committed
add percentiles
1 parent 570af4e commit 4907674

4 files changed

+296
-1
lines changed

‎300_Aggregations/65_percentiles.asciidoc

+293
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,293 @@
1+
=== Percentiles
2+
3+
The other approximate metric offered by Elasticsearch is the `percentiles` metric.
4+
Percentiles show the point at which a certain percentage of observed values occur.
5+
For example, the 95th percentile is the value which is greater than 95% of the
6+
data.
7+
8+
Percentiles are often used to find outliers. In normal distributions, the 0.13th and 99.87th percentiles represents three standard deviations from the mean. Any data which falls outside three standard deviations is often considered an anomaly because it
9+
is so different from the average value.
10+
11+
To be more concrete, imagine that you are running a large website and it is your
12+
job to guarantee fast response times to visitors. You must therefore monitor
13+
your website latency to determine if you are meeting your goal.
14+
15+
A common metric to use in this scenario is the average latency. But this is actually
16+
a poor choice (despite being common), because averages can easily hide outliers.
17+
A median metric also suffers the same problem. You could try a maximum, but this
18+
metric is easily skewed by just a single outlier.
19+
20+
This graph visualizes the problem. If you rely on simple metrics like mean or median, you might see a graph that looks like this:
21+
22+
image::images/300_65_percentile1.png
23+
24+
Everything looks fine. There is a slight bump, but nothing to be concerned about.
25+
But if we load up the 99th percentile (the value which accounts for the slowest 1%
26+
of latencies), we see an entirely different story:
27+
28+
image::images/300_65_percentile2.png
29+
30+
Woah! At 9:30 AM, the mean is only 75ms. As a system administrator, you wouldn't
31+
look at this value twice. Everything normal! But the 99th percentile is telling
32+
you that 1% of your customers are seeing latency in excess of 850ms...a very
33+
different story.
34+
35+
This is just one use-case for a percentile. They can also be used to quickly
36+
eyeball the distribution of data, check for skew or bimodalities, etc.
37+
38+
==== Percentile Metric
39+
40+
Let's load a new dataset (the cars data isn't going to work well for percentiles).
41+
We are going to index a bunch of website latencies and run a few percentiles over
42+
it.
43+
44+
[source,js]
45+
----
46+
POST /website/logs/_bulk
47+
{ "index": {}}
48+
{ "latency" : 100, "zone" : "US", "timestamp" : "2014-10-28" }
49+
{ "index": {}}
50+
{ "latency" : 80, "zone" : "US", "timestamp" : "2014-10-29" }
51+
{ "index": {}}
52+
{ "latency" : 99, "zone" : "US", "timestamp" : "2014-10-29" }
53+
{ "index": {}}
54+
{ "latency" : 102, "zone" : "US", "timestamp" : "2014-10-28" }
55+
{ "index": {}}
56+
{ "latency" : 75, "zone" : "US", "timestamp" : "2014-10-28" }
57+
{ "index": {}}
58+
{ "latency" : 82, "zone" : "US", "timestamp" : "2014-10-29" }
59+
{ "index": {}}
60+
{ "latency" : 100, "zone" : "EU", "timestamp" : "2014-10-28" }
61+
{ "index": {}}
62+
{ "latency" : 280, "zone" : "EU", "timestamp" : "2014-10-29" }
63+
{ "index": {}}
64+
{ "latency" : 155, "zone" : "EU", "timestamp" : "2014-10-29" }
65+
{ "index": {}}
66+
{ "latency" : 623, "zone" : "EU", "timestamp" : "2014-10-28" }
67+
{ "index": {}}
68+
{ "latency" : 380, "zone" : "EU", "timestamp" : "2014-10-28" }
69+
{ "index": {}}
70+
{ "latency" : 319, "zone" : "EU", "timestamp" : "2014-10-29" }
71+
----
72+
// SENSE: 300_Aggregations/65_percentiles.json
73+
74+
This data contains three values: a latency, a datacenter "zone", and a date
75+
timestamp. Let's run a percentiles over the whole data-set to get a feel for
76+
the distribution:
77+
78+
[source,js]
79+
----
80+
GET /website/logs/_search?search_type=count
81+
{
82+
"aggs" : {
83+
"load_times" : {
84+
"percentiles" : {
85+
"field" : "latency" <1>
86+
}
87+
},
88+
"avg_load_time" : {
89+
"avg" : {
90+
"field" : "latency" <2>
91+
}
92+
}
93+
}
94+
}
95+
----
96+
// SENSE: 300_Aggregations/65_percentiles.json
97+
<1> The `percentiles` metric is applied to the "latency" field
98+
<2> For comparison, we also execute an `avg` metric on the same field
99+
100+
By default, the `percentiles` metric will return an array of pre-defined percentiles:
101+
`[1, 5, 25, 50, 75, 95, 99]`. These represent common percentiles that people are
102+
interested in -- the extreme percentiles at either end of the spectrum, and a
103+
few in the middle. In the response, we see that the fastest latency is around 75ms,
104+
while the slowest is almost 600ms. In contrast, the average is sitting near
105+
200ms, which is much less informative:
106+
107+
[source,js]
108+
----
109+
...
110+
"aggregations": {
111+
"load_times": {
112+
"values": {
113+
"1.0": 75.55,
114+
"5.0": 77.75,
115+
"25.0": 94.75,
116+
"50.0": 101,
117+
"75.0": 289.75,
118+
"95.0": 489.34999999999985,
119+
"99.0": 596.2700000000002
120+
}
121+
},
122+
"avg_load_time": {
123+
"value": 199.58333333333334
124+
}
125+
}
126+
----
127+
128+
So there is clearly a wide distribution in latencies, let's look and see if it is
129+
correlated to the geographic "zone" of the datacenter:
130+
131+
[source,js]
132+
----
133+
GET /website/logs/_search?search_type=count
134+
{
135+
"aggs" : {
136+
"zones" : {
137+
"terms" : {
138+
"field" : "zone" <1>
139+
},
140+
"aggs" : {
141+
"load_times" : {
142+
"percentiles" : { <2>
143+
"field" : "latency",
144+
"percents" : [50, 95.0, 99.0] <3>
145+
}
146+
},
147+
"load_avg" : {
148+
"avg" : {
149+
"field" : "latency"
150+
}
151+
}
152+
}
153+
}
154+
}
155+
}
156+
----
157+
// SENSE: 300_Aggregations/65_percentiles.json
158+
<1> First separate we separate our latencies into buckets depending on their zone
159+
<2> Then calculate the percentiles per zone
160+
<3> The "percents" parameter accepts an array of percentiles that we want returned,
161+
since we are only interested in slow latencies
162+
163+
From the response, we can see the EU zone is much slower than the US zone. On the
164+
US side, the 50th percentile is very close to the 99th percentile...and both are
165+
close to the average.
166+
167+
In contrast, the EU zone has a large difference between the 50th and 99th
168+
percentile. It is now obvious that the EU zone is dragging down the latency
169+
statistics, and we know that 50% of the EU zone is seeing 300ms+ latencies.
170+
171+
[source,js]
172+
----
173+
...
174+
"aggregations": {
175+
"zones": {
176+
"buckets": [
177+
{
178+
"key": "eu",
179+
"doc_count": 6,
180+
"load_times": {
181+
"values": {
182+
"50.0": 299.5,
183+
"95.0": 562.25,
184+
"99.0": 610.85
185+
}
186+
},
187+
"load_avg": {
188+
"value": 309.5
189+
}
190+
},
191+
{
192+
"key": "us",
193+
"doc_count": 6,
194+
"load_times": {
195+
"values": {
196+
"50.0": 90.5,
197+
"95.0": 101.5,
198+
"99.0": 101.9
199+
}
200+
},
201+
"load_avg": {
202+
"value": 89.66666666666667
203+
}
204+
}
205+
]
206+
}
207+
}
208+
...
209+
----
210+
211+
==== Percentile Ranks
212+
213+
There is another, closely related metric called `percentile_rank`. The
214+
`percentiles` metric tells you the value at which X% is less than. The `percentile_ranks`
215+
tells you what percentile a specific value belongs too. It is basically a two-
216+
way relationship depending on what data you need. For example:
217+
218+
- The 50th percentile is 119
219+
- 119's percentile rank is 50
220+
221+
So imagine that our website must maintain an SLA of 210ms response times or less.
222+
You would like to know what percentage of requests are actually meeting that SLA.
223+
For this, you can apply the `percentile_ranks` metric instead of `percentiles`:
224+
225+
[source,js]
226+
----
227+
GET /website/logs/_search?search_type=count
228+
{
229+
"aggs" : {
230+
"zones" : {
231+
"terms" : {
232+
"field" : "zone"
233+
},
234+
"aggs" : {
235+
"load_times" : {
236+
"percentile_ranks" : {
237+
"field" : "latency",
238+
"values" : [210] <1>
239+
}
240+
}
241+
}
242+
}
243+
}
244+
}
245+
----
246+
// SENSE: 300_Aggregations/65_percentiles.json
247+
<1> The `percentile_ranks` metric accepts an array of values that you want ranks for
248+
249+
250+
TODO did this actually make it into 1.3?
251+
252+
==== Understanding the Tradeoffs
253+
254+
Like cardinality, calculating percentiles requires an approximate algorithm.
255+
The naive implementation would maintain a sorted list of all values...but this
256+
clearly is not possible when you have billions of values distributed across
257+
dozens of nodes.
258+
259+
Instead, percentiles uses an algorithm called TDigest (introduced by Ted Dunning
260+
in https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf:[Computing Accurate Quantiles using T-Digests]). Like HyperLogLOg, it isn't
261+
necessary to understand the full technical details, but it is good to know
262+
the properties of the algorithm:
263+
264+
- Percentile accuracy is proportional to how "extreme" the percentile is. This
265+
means that percentiles such as 99th are more accurate than less extreme percentiles,
266+
such as the median. This is just a property of how the data structure works, but
267+
it happens to be a nice property because most people care about extreme percentiles
268+
269+
- For small sets of values, percentiles are highly accurate. If the dataset is
270+
small enough, the percentiles may be 100% accurate.
271+
272+
- As the quantity of values in a bucket grows, the algorithm begins to
273+
approximate the percentiles. It is effectively trading accuracy for memory
274+
savings. The exact level of inaccuracy is difficult to generalize, since it
275+
depends on your data distribution and volume of data being aggregated
276+
277+
Similar to `cardinality`, you can control the memory : accuracy ratio by changing
278+
a parameter: `compression`.
279+
280+
The TDigest algorithm uses a number of "nodes" to approximate percentiles 
281+
-— the more nodes available, the higher the accuracy (and large memory footprint) proportional to the volume of data. The compression parameter limits the maximum
282+
number of nodes to 20 * compression.
283+
284+
Therefore, by increasing the compression value, you can increase the accuracy of
285+
your percentiles at the cost of more memory. Larger compression values also
286+
make the algorithm slower since the underlying tree data structure grows in size, resulting in more expensive operations. The default compression value is 100.
287+
288+
A "node" uses roughly 32 bytes of memory, so under worst-case scenarios (large
289+
amount of data which arrives sorted and in-order) the default settings will
290+
produce a TDigest roughly 64KB in size. In practice data tends to be more
291+
random and the TDigest will use less memory.
292+
293+

‎304_Approximate_Aggregations.asciidoc

+3-1
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,6 @@
33

44
include::300_Aggregations/55_approx_intro.asciidoc[]
55

6-
include::300_Aggregations/60_cardinality.asciidoc[]
6+
include::300_Aggregations/60_cardinality.asciidoc[]
7+
8+
include::300_Aggregations/65_percentiles.asciidoc[]

‎images/300_65_percentile1.png

23.8 KB
Loading

‎images/300_65_percentile2.png

33.3 KB
Loading

0 commit comments

Comments
 (0)