Skip to content

Commit df40807

Browse files
swiftixswiftix
committed
Provide a description about the optimizer counters analysis
It described the overall idea behind the optimizer counters, how to collect and record them and how to analyze them
1 parent 74908ce commit df40807

File tree

1 file changed

+348
-0
lines changed

1 file changed

+348
-0
lines changed

‎docs/OptimizerCountersAnalysis.md

+348
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,348 @@
1+
# Optimizer Counter Analysis
2+
3+
It is possible possible by means of providing some special command-line
4+
options to ask the Swift compiler to produce different statistics about
5+
the optimizer counters. Optimizer counters are most typically counters
6+
representing different aspects of a SIL representation for a Swift module
7+
being compiled, e.g. the number of SIL basic blocks or instructions.
8+
These counters may also reveal some details about transformations
9+
and optimizations performed on SIL, e.g. the duration of an optimization
10+
or how much memory was consumed by the compiler. Therefore having the
11+
information about the changes of optimizer counters over time allows for
12+
analysis of changes to the SIL representation during the compilation.
13+
14+
This document describes how you collect and analyze the counters produced by
15+
the optimizer of the Swift compiler. This analysis is useful if you need to get
16+
a detailed insight into the optimizer passes, their effect on the SIL code,
17+
compile times, compiler's memory consumption, etc. For example, you can find
18+
out which optimization passes or phases of optimization produce most new SIL
19+
instructions or delete most SIL functions.
20+
21+
## Table of contents
22+
<!-- TOC -->
23+
- [Optimizer Counter Analysis](#optimizer-counter-analysis)
24+
- [Table of contents](#table-of-contents)
25+
- [Which optimizer counters are available for recording](#which-optimizer-counters-are-available-for-recording)
26+
- [How and when the optimizer counters are collected and recorded](#how-and-when-the-optimizer-counters-are-collected-and-recorded)
27+
- [Differences between different modes of optimizer counters collection](#differences-between-different-modes-of-optimizer-counters-collection)
28+
- [Module level counters](#module-level-counters)
29+
- [Function level counters](#function-level-counters)
30+
- [Instruction level counters](#instruction-level-counters)
31+
- [Configuring which counters changes should be recorded](#configuring-which-counters-changes-should-be-recorded)
32+
- [On-line and off-line processing of optimizer counters](#on-line-and-off-line-processing-of-optimizer-counters)
33+
- [Specifying where the optimizer counters should be stored](#specifying-where-the-optimizer-counters-should-be-stored)
34+
- [The format of the recorded optimizer counters](#the-format-of-the-recorded-optimizer-counters)
35+
- [Storing the produced statistics into a database](#storing-the-produced-statistics-into-a-database)
36+
- [The database schema for counters](#the-database-schema-for-counters)
37+
- [Analyzing collected counters using SQL](#analyzing-collected-counters-using-sql)
38+
- [Examples of interesting SQL queries](#examples-of-interesting-sql-queries)
39+
- [Compute aggregate times for each optimization pass/transformation](#compute-aggregate-times-for-each-optimization-passtransformation)
40+
- [Which pass created/deleted most functions?](#which-pass-createddeleted-most-functions)
41+
- [Which pass at which optimization pipeline stage created/deleted most functions?](#which-pass-at-which-optimization-pipeline-stage-createddeleted-most-functions)
42+
- [Which pass created/removed most instructions?](#which-pass-createdremoved-most-instructions)
43+
- [Which pass at which stage created/removed most instructions?](#which-pass-at-which-stage-createdremoved-most-instructions)
44+
- [Which functions were changed most by which stage when it comes to the instruction counts](#which-functions-were-changed-most-by-which-stage-when-it-comes-to-the-instruction-counts)
45+
- [Get the number of instructions at the beginning and at the end of the optimization pipeline for each function](#get-the-number-of-instructions-at-the-beginning-and-at-the-end-of-the-optimization-pipeline-for-each-function)
46+
- [Show functions which have the biggest size at the end of the optimization pipeline](#show-functions-which-have-the-biggest-size-at-the-end-of-the-optimization-pipeline)
47+
- [Which optimization pipeline stage took most time?](#which-optimization-pipeline-stage-took-most-time)
48+
- [Which stage added/removed most instructions (in term of deltas)?](#which-stage-addedremoved-most-instructions-in-term-of-deltas)
49+
<!-- /TOC -->
50+
51+
## Which optimizer counters are available for recording
52+
53+
The following statistics can be recorded:
54+
55+
* For SILFunctions: the number of SIL basic blocks for each SILFunction, the
56+
number of SIL instructions, the number of SILInstructions of a specific
57+
kind (e.g. a number of alloc_ref instructions)
58+
59+
* For SILModules: the number of SIL basic blocks in the SILModule, the number
60+
of SIL instructions, the number of SILFunctions, the number of
61+
SILInstructions of a specific kind (e.g. a number of alloc_ref
62+
instructions) the amount of memory used by the compiler.
63+
64+
## How and when the optimizer counters are collected and recorded
65+
66+
The pass manager of the SIL optimizer invokes special hooks before and after it
67+
executes any optimization transformation (often called an optimization pass).
68+
69+
The hook checks if there are any changes of counter values since the last time
70+
it was invoked. If there are any changes, then the counters are re-computed.
71+
They are first re-computed for SILFunctions and then for the SILModule being
72+
compiled. The re-computation algorithm is trying to be incremental and fast by
73+
re-computing only the minimal amount of counters instead of scanning the whole
74+
SILModule every time.
75+
76+
Those counters that are changed are reported if they satisfy different
77+
filtering conditions, which are configurable. For example, you may want to see
78+
only counters that have changed by more than 50% since last time.
79+
Alternatively, you may want to log as many counters as possible, in which case
80+
they will be logged on every invocation of the hook.
81+
82+
If there were no changes to the counters that satisfy the filtering conditions,
83+
those changes would not be logged. This means that the final set of logged
84+
changes may be incomplete, i.e. it would not reflect all changes that happened
85+
to those counters.
86+
87+
## Differences between different modes of optimizer counters collection
88+
89+
You can collect optimizer counters at different levels of granularity depending
90+
on your needs. The granularity also affects the amount of recorded data that
91+
will be produced, because the amount of recorded data grows if you decide to
92+
collect more fine-grained counters.
93+
94+
### Module level counters
95+
The most coarse-grained counters are the module level counters, which are
96+
enabled by using `-Xllvm -sil-stats-modules` command-line option. They are
97+
usually logged only if a given optimizer counter changed a lot for the whole
98+
SILModule, which does not happen that often, because most optimization passes
99+
perform just very small transformations on a single function and thus do not
100+
significantly change any module-wide counters.
101+
102+
### Function level counters
103+
The next level of granularity are SILFunction counters, which are enabled by
104+
using `-Xllvm -sil-stats-functions` command-line option. They track statistics
105+
for SILFunctions. Every SILFunction has its own set of these counters.
106+
Obviously, interesting changes to these counters happen more often than to the
107+
module-wide counters.
108+
109+
### Instruction level counters
110+
The finest level of granularity are SILInstruction counters, which are enabled
111+
by using `-Xllvm -sil-stats-only-instructions` command-line option. You can use
112+
them to e.g. collect statistics about how many specific SIL instructions are
113+
used by a given SILFunction or a SILModule. For example, you can count how many
114+
`alloc_ref` instructions occur in a given SILFunction or SILModule. If you are
115+
interested in collecting the stats only for some specific SIL instructions, you
116+
can use a comma-separated list of instructions as a value of the option,
117+
e.g. `-Xllvm -sil-stats-only-instructions=alloc_ref,alloc_stack`. If you need to
118+
collect stats about all kinds of SIL instructions, you can use this syntax:
119+
`-Xllvm -sil-stats-only-instructions=all`.
120+
121+
## Configuring which counters changes should be recorded
122+
123+
The user has a possibility to configure a number of thresholds, which control
124+
what needs to be recorded. Many of those thresholds are formulated for the
125+
deltas, e.g. the delta should be more than 50%.
126+
127+
The value of the delta for a given old and new values of a counter are computed
128+
using the following simple formula:
129+
130+
`delta = 100% * (new_value - old_value)/old_value`
131+
132+
So, a delta basically reflects how big is the change of the counter value
133+
when expressed as a percentage of the old value.
134+
135+
TBD Provide more information about different command-line options for
136+
configuring the thresholds.
137+
138+
## On-line and off-line processing of optimizer counters
139+
140+
As explained above, some of the counters filtering happens on-line at
141+
compile-time already. If this is enough for your purposes, you can use them "as
142+
is".
143+
144+
But in many cases, you may want to perform more complex analysis of the
145+
collected counters, e.g. you may want to aggregate them by the optimization
146+
pass or by a stage of the optimization pipeline, etc. This is not directly
147+
supported by the on-line filtering mode. Instead, you can record all the
148+
interesting counters and then post-process it off-line by first storing them
149+
into a SQLite database by means if a special utility and then using the regular
150+
SQL queries to perform any kind of analysis and aggregation that you may need.
151+
152+
## Specifying where the optimizer counters should be stored
153+
154+
By default, all the collected statistics are written to the
155+
standard error.
156+
157+
But it is possible to write into a custom file by specifying the following
158+
command-line option:
159+
160+
`-Xllvm -sil-stats-output-file=your_file_name`
161+
162+
## The format of the recorded optimizer counters
163+
164+
The counters are recorded using a simple CSV (comma separated value) format.
165+
Each line represents a single counter value or a counter value change.
166+
167+
For counter value updates, the CSV line looks like this:
168+
* `Kind, CounterName, StageName, TransformName,
169+
TransformPassNumber, DeltaValue, OldCounterValue,
170+
NewCounterValue, Duration, Symbol`
171+
172+
And for counter stats it looks like this:
173+
* `Kind, CounterName, StageName, TransformName,
174+
TransformPassNumber, CounterValue, Duration, Symbol`
175+
176+
where the names used above have the following meaning:
177+
178+
* `Kind` is one of `function`, `module`, `function_history`.
179+
* `function` and
180+
`module` correspond directly to the module-level and function-level counters
181+
* `function_history` corresponds to the verbose mode of function
182+
counters collection, when changes to the SILFunction counters are logged
183+
unconditionally, without any on-line filtering.
184+
* `CounterName` is typically one of `block`, `inst`, `function`, `memory`,
185+
or `inst_instruction_name` if you collect counters for specific kinds of SIL
186+
instructions.
187+
* `Symbol` is e.g. the name of a function
188+
* `StageName` is the name of the current optimizer pipeline stage
189+
* `TransformName` is the name of the current optimizer transformation/pass
190+
* `Duration` is the duration of the transformation
191+
* `TransformPassNumber` is the optimizer pass number. It is useful if you
192+
want to reproduce the result later using
193+
`-Xllvm -sil-opt-pass-count -Xllvm TransformPassNumber`
194+
195+
## Storing the produced statistics into a database
196+
197+
To store the set of produced counters into a database, you can use the
198+
following command:
199+
200+
`utils/optimizer_counters_to_sql.py csv_file_with_counters your_database.db`
201+
202+
## The database schema for counters
203+
204+
The database uses a very simple self-explaining schema:
205+
206+
```sql
207+
CREATE TABLE Counters(
208+
Id INTEGER PRIMARY KEY AUTOINCREMENT,
209+
Stage TEXT NOT NULL,
210+
Transform TEXT NOT NULL,
211+
Kind TEXT,
212+
Counter TEXT NOT NULL,
213+
PassNum INT NOT NULL,
214+
Delta NUMBER,
215+
Old INT,
216+
New INT,
217+
Duration INT,
218+
Symbol TEXT NOT NULL DEFAULT '');
219+
```
220+
221+
## Analyzing collected counters using SQL
222+
223+
First, you need to connect to your database, so that you can issue SQL queries.
224+
This can be accomplished e.g. by the following command:
225+
226+
`sqlite3 your_database.db`
227+
228+
After executing this command, you will be presented with a SQLite command
229+
prompt and you can start entering SQL queries. Each query should end with
230+
a semicolon.
231+
232+
### Examples of interesting SQL queries
233+
234+
SQL gives you a lot of freedom to perform different kinds of analysis. Below
235+
you can find some examples of typical queries, which you can use "as is" or as
236+
a basis for formulating more complex queries.
237+
238+
#### Compute aggregate times for each optimization pass/transformation
239+
240+
```sql
241+
select C.Transform, sum(C.Duration)
242+
from Counters C
243+
where C.counter = 'inst' and C.kind = 'module'
244+
group by C.Transform;
245+
```
246+
247+
#### Which pass created/deleted most functions?
248+
```sql
249+
select C.Transform, sum(C.Delta)
250+
from Counters C
251+
where C.counter = 'functions' and C.kind = 'module'
252+
group by C.Transform;
253+
```
254+
255+
#### Which pass at which optimization pipeline stage created/deleted most functions?
256+
```sql
257+
select C.Stage, C.Transform, sum(C.Delta)
258+
from Counters C where C.counter = 'functions' and C.kind = 'module'
259+
group by C.Stage, C.Transform;
260+
```
261+
262+
#### Which pass created/removed most instructions?
263+
264+
```sql
265+
# Sort by biggest changes
266+
select C.Transform, sum(C.Delta)
267+
from Counters C where C.counter = 'inst' and C.kind = 'module'
268+
group by C.Transform
269+
order by abs(sum(C.Delta));
270+
```
271+
272+
#### Which pass at which stage created/removed most instructions?
273+
```sql
274+
# Sort by biggest changes
275+
select C.Stage, C.Transform, sum(C.Delta)
276+
from Counters C where C.counter = 'inst' and C.kind = 'module'
277+
group by C.Stage, C.Transform
278+
order by abs(sum(C.Delta));
279+
```
280+
281+
#### Which functions were changed most by which stage when it comes to the instruction counts
282+
283+
```sql
284+
select C.Stage, min(C.Old), max(C.Old), Symbol
285+
from Counters C where C.counter = 'inst' and C.kind = 'function_history'
286+
group by C.Symbol, C.Stage
287+
having min(C.Old) <> max(C.Old)
288+
order by abs(max(C.Old)-min(C.Old));
289+
```
290+
291+
#### Get the number of instructions at the beginning and at the end of the optimization pipeline for each function
292+
```sql
293+
select MinOld.Id, MinOld.Old, MaxOld.Id, MaxOld.Old, MinOld.Symbol
294+
from
295+
(
296+
select C.Id, C.Old, C.Symbol
297+
from Counters C where C.counter = 'inst' and C.kind = 'function_history'
298+
group by C.Symbol
299+
having C.Id = max(Id)
300+
) as MaxOld,
301+
(select C.Id, C.Old, C.Symbol
302+
from Counters C
303+
where C.counter = 'inst' and C.kind = 'function_history'
304+
group by C.Symbol
305+
having C.Id = min(Id)
306+
) as MinOld
307+
where MinOld.Symbol == MaxOld.Symbol;
308+
```
309+
310+
#### Show functions which have the biggest size at the end of the optimization pipeline
311+
```sql
312+
select MinOld.Id, MinOld.Old, MaxOld.Id, MaxOld.Old, MinOld.Symbol
313+
from
314+
(
315+
select C.Id, C.Old, C.Symbol
316+
from Counters C
317+
where C.counter = 'inst' and C.kind = 'function_history'
318+
group by C.Symbol
319+
having C.Id = max(Id)
320+
) as MaxOld,
321+
(
322+
select C.Id, C.Old, C.Symbol
323+
from Counters C
324+
where C.counter = 'inst' and C.kind = 'function_history'
325+
group by C.Symbol
326+
having C.Id = min(Id)
327+
) as MinOld
328+
where MinOld.Symbol == MaxOld.Symbol
329+
order by MaxOld.Old;
330+
```
331+
332+
#### Which optimization pipeline stage took most time?
333+
```sql
334+
select sum(Duration), Stage
335+
from Counters C
336+
where C.counter = 'inst' and C.kind = 'module'
337+
group by Stage
338+
order by sum(C.Duration);
339+
```
340+
341+
#### Which stage added/removed most instructions (in term of deltas)?
342+
```sql
343+
select sum(Delta), Stage
344+
from Counters C where C.counter = 'inst' and C.kind = 'module'
345+
group by Stage
346+
order by sum(C.Delta);
347+
```
348+

0 commit comments

Comments
 (0)