File tree Expand file tree Collapse file tree 1 file changed +30
-0
lines changed Expand file tree Collapse file tree 1 file changed +30
-0
lines changed Original file line number Diff line number Diff line change 1
1
# Pairwise t-test
2
2
3
3
Scripts to perform pairwise t-test on TREC run files.
4
+
5
+ ### Requirements
6
+
7
+ * R
8
+ * [ reshape2] [ reshape2 ]
9
+ * [ gdeval.pl] [ gdeval ]
10
+ * [ trec\_ eval] [ treceval ]
11
+
12
+ [ reshape2 ] : https://cran.r-project.org/web/packages/reshape2/index.html
13
+ [ gdeval ] : https://github.com/lgrz/trec-web-2013
14
+ [ treceval ] : https://trec.nist.gov/trec_eval
15
+
16
+ ### Usage
17
+
18
+ There are two bash scripts to run. First run ` pairwise-eval.sh ` to evaluate the
19
+ TREC run files. Then run ` pairwise-ttest.sh ` to compute statistical
20
+ significance.
21
+
22
+ To compute a pairwise t-test of all run files in the ` runs ` directory for
23
+ NDCG@10 using ` foo.qrels ` (which contains the relevance judgments), run
24
+ the following:
25
+
26
+ ```
27
+ ./pairwise-eval.sh ndcg 10 foo.qrels runs/*.run
28
+ ./pairwise-ttest.sh runs/*.run.ndcg10
29
+ cat result.txt
30
+ ```
31
+
32
+ The ` pairwise-eval.sh ` script can compute ERR, NDCG and MAP. ` gdeval.pl ` is
33
+ used for ERR and NDCG, while ` trec_eval ` is used MAP.
You can’t perform that action at this time.
0 commit comments