Skip to content

Commit 90cd68e

Browse files
committed
Update readme
1 parent 9f02663 commit 90cd68e

File tree

1 file changed

+30
-0
lines changed

1 file changed

+30
-0
lines changed

README.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,33 @@
11
# Pairwise t-test
22

33
Scripts to perform pairwise t-test on TREC run files.
4+
5+
### Requirements
6+
7+
* R
8+
* [reshape2][reshape2]
9+
* [gdeval.pl][gdeval]
10+
* [trec\_eval][treceval]
11+
12+
[reshape2]: https://cran.r-project.org/web/packages/reshape2/index.html
13+
[gdeval]: https://github.com/lgrz/trec-web-2013
14+
[treceval]: https://trec.nist.gov/trec_eval
15+
16+
### Usage
17+
18+
There are two bash scripts to run. First run `pairwise-eval.sh` to evaluate the
19+
TREC run files. Then run `pairwise-ttest.sh` to compute statistical
20+
significance.
21+
22+
To compute a pairwise t-test of all run files in the `runs` directory for
23+
NDCG@10 using `foo.qrels` (which contains the relevance judgments), run
24+
the following:
25+
26+
```
27+
./pairwise-eval.sh ndcg 10 foo.qrels runs/*.run
28+
./pairwise-ttest.sh runs/*.run.ndcg10
29+
cat result.txt
30+
```
31+
32+
The `pairwise-eval.sh` script can compute ERR, NDCG and MAP. `gdeval.pl` is
33+
used for ERR and NDCG, while `trec_eval` is used MAP.

0 commit comments

Comments
 (0)