Skip to content

Commit 6a783fd

Browse files
rasbtrasbt
committed
a better tfidf example p236-p240
1 parent 52109a8 commit 6a783fd

File tree

2 files changed

+145
-27
lines changed

2 files changed

+145
-27
lines changed

‎code/ch08/ch08.ipynb

+144-26
Original file line numberDiff line numberDiff line change
@@ -44,16 +44,16 @@
4444
"output_type": "stream",
4545
"text": [
4646
"Sebastian Raschka \n",
47-
"Last updated: 01/20/2016 \n",
47+
"last updated: 2016-06-05 \n",
4848
"\n",
4949
"CPython 3.5.1\n",
50-
"IPython 4.0.1\n",
50+
"IPython 4.2.0\n",
5151
"\n",
52-
"numpy 1.10.1\n",
53-
"pandas 0.17.1\n",
54-
"matplotlib 1.5.0\n",
55-
"scikit-learn 0.17\n",
56-
"nltk 3.1\n"
52+
"numpy 1.11.0\n",
53+
"pandas 0.18.0\n",
54+
"matplotlib 1.5.1\n",
55+
"scikit-learn 0.17.1\n",
56+
"nltk 3.2.1\n"
5757
]
5858
}
5959
],
@@ -342,9 +342,19 @@
342342
"## Transforming documents into feature vectors"
343343
]
344344
},
345+
{
346+
"cell_type": "markdown",
347+
"metadata": {},
348+
"source": [
349+
"By calling the fit_transform method on CountVectorizer, we just constructed the vocabulary of the bag-of-words model and transformed the following three sentences into sparse feature vectors:\n",
350+
"1. The sun is shining\n",
351+
"2. The weather is sweet\n",
352+
"3. The sun is shining, the weather is sweet, and one and one is two\n"
353+
]
354+
},
345355
{
346356
"cell_type": "code",
347-
"execution_count": 7,
357+
"execution_count": 15,
348358
"metadata": {
349359
"collapsed": false
350360
},
@@ -356,10 +366,17 @@
356366
"docs = np.array([\n",
357367
" 'The sun is shining',\n",
358368
" 'The weather is sweet',\n",
359-
" 'The sun is shining and the weather is sweet'])\n",
369+
" 'The sun is shining, the weather is sweet, and one and one is two'])\n",
360370
"bag = count.fit_transform(docs)"
361371
]
362372
},
373+
{
374+
"cell_type": "markdown",
375+
"metadata": {},
376+
"source": [
377+
"Now let us print the contents of the vocabulary to get a better understanding of the underlying concepts:"
378+
]
379+
},
363380
{
364381
"cell_type": "code",
365382
"execution_count": 8,
@@ -371,14 +388,28 @@
371388
"name": "stdout",
372389
"output_type": "stream",
373390
"text": [
374-
"{'sweet': 4, 'and': 0, 'is': 1, 'shining': 2, 'sun': 3, 'the': 5, 'weather': 6}\n"
391+
"{'sun': 4, 'and': 0, 'is': 1, 'the': 6, 'shining': 3, 'two': 7, 'sweet': 5, 'weather': 8, 'one': 2}\n"
375392
]
376393
}
377394
],
378395
"source": [
379396
"print(count.vocabulary_)"
380397
]
381398
},
399+
{
400+
"cell_type": "markdown",
401+
"metadata": {},
402+
"source": [
403+
"As we can see from executing the preceding command, the vocabulary is stored in a Python dictionary, which maps the unique words that are mapped to integer indices. Next let us print the feature vectors that we just created:"
404+
]
405+
},
406+
{
407+
"cell_type": "markdown",
408+
"metadata": {},
409+
"source": [
410+
"Each index position in the feature vectors shown here corresponds to the integer values that are stored as dictionary items in the CountVectorizer vocabulary. For example, the rst feature at index position 0 resembles the count of the word and, which only occurs in the last document, and the word is at index position 1 (the 2nd feature in the document vectors) occurs in all three sentences. Those values in the feature vectors are also called the raw term frequencies: *tf (t,d)*—the number of times a term t occurs in a document *d*."
411+
]
412+
},
382413
{
383414
"cell_type": "code",
384415
"execution_count": 9,
@@ -390,9 +421,9 @@
390421
"name": "stdout",
391422
"output_type": "stream",
392423
"text": [
393-
"[[0 1 1 1 0 1 0]\n",
394-
" [0 1 0 0 1 1 1]\n",
395-
" [1 2 1 1 1 2 1]]\n"
424+
"[[0 1 0 1 1 0 1 0 0]\n",
425+
" [0 1 0 0 0 1 1 0 1]\n",
426+
" [2 3 2 1 1 1 2 1 1]]\n"
396427
]
397428
}
398429
],
@@ -425,6 +456,24 @@
425456
"np.set_printoptions(precision=2)"
426457
]
427458
},
459+
{
460+
"cell_type": "markdown",
461+
"metadata": {},
462+
"source": [
463+
"When we are analyzing text data, we often encounter words that occur across multiple documents from both classes. Those frequently occurring words typically don't contain useful or discriminatory information. In this subsection, we will learn about a useful technique called term frequency-inverse document frequency (tf-idf) that can be used to downweight those frequently occurring words in the feature vectors. The tf-idf can be de ned as the product of the term frequency and the inverse document frequency:\n",
464+
"\n",
465+
"$$\\text{tf-idf}(t,d)=\\text{tf (t,d)}\\times \\text{idf}(t,d)$$\n",
466+
"\n",
467+
"Here the tf(t, d) is the term frequency that we introduced in the previous section,\n",
468+
"and the inverse document frequency *idf(t, d)* can be calculated as:\n",
469+
"\n",
470+
"$$\\text{idf}(t,d) = \\text{log}\\frac{n_d}{1+\\text{df}(d, t)},$$\n",
471+
"\n",
472+
"where $n_d$ is the total number of documents, and *df(d, t)* is the number of documents *d* that contain the term *t*. Note that adding the constant 1 to the denominator is optional and serves the purpose of assigning a non-zero value to terms that occur in all training samples; the log is used to ensure that low document frequencies are not given too much weight.\n",
473+
"\n",
474+
"Scikit-learn implements yet another transformer, the `TfidfTransformer`, that takes the raw term frequencies from `CountVectorizer` as input and transforms them into tf-idfs:"
475+
]
476+
},
428477
{
429478
"cell_type": "code",
430479
"execution_count": 11,
@@ -436,9 +485,9 @@
436485
"name": "stdout",
437486
"output_type": "stream",
438487
"text": [
439-
"[[ 0. 0.43 0.56 0.56 0. 0.43 0. ]\n",
440-
" [ 0. 0.43 0. 0. 0.56 0.43 0.56]\n",
441-
" [ 0.4 0.48 0.31 0.31 0.31 0.48 0.31]]\n"
488+
"[[ 0. 0.43 0. 0.56 0.56 0. 0.43 0. 0. ]\n",
489+
" [ 0. 0.43 0. 0. 0. 0.56 0.43 0. 0.56]\n",
490+
" [ 0.5 0.45 0.5 0.19 0.19 0.19 0.3 0.25 0.19]]\n"
442491
]
443492
}
444493
],
@@ -449,9 +498,53 @@
449498
"print(tfidf.fit_transform(count.fit_transform(docs)).toarray())"
450499
]
451500
},
501+
{
502+
"cell_type": "markdown",
503+
"metadata": {},
504+
"source": [
505+
"As we saw in the previous subsection, the word is had the largest term frequency in the 3rd document, being the most frequently occurring word. However, after transforming the same feature vector into tf-idfs, we see that the word is is\n",
506+
"now associated with a relatively small tf-idf (0.45) in document 3 since it is\n",
507+
"also contained in documents 1 and 2 and thus is unlikely to contain any useful, discriminatory information.\n"
508+
]
509+
},
510+
{
511+
"cell_type": "markdown",
512+
"metadata": {},
513+
"source": [
514+
"However, if we'd manually calculated the tf-idfs of the individual terms in our feature vectors, we'd have noticed that the `TfidfTransformer` calculates the tf-idfs slightly differently compared to the standard textbook equations that we de ned earlier. The equations for the idf and tf-idf that were implemented in scikit-learn are:"
515+
]
516+
},
517+
{
518+
"cell_type": "markdown",
519+
"metadata": {},
520+
"source": [
521+
"$$\\text{idf} (t,d) = log\\frac{1 + n_d}{1 + \\text{df}(d, t)}$$\n",
522+
"\n",
523+
"The tf-idf equation that was implemented in scikit-learn is as follows:\n",
524+
"\n",
525+
"$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d)+1)$$\n",
526+
"\n",
527+
"While it is also more typical to normalize the raw term frequencies before calculating the tf-idfs, the `TfidfTransformer` normalizes the tf-idfs directly.\n",
528+
"\n",
529+
"By default (`norm='l2'`), scikit-learn's TfidfTransformer applies the L2-normalization, which returns a vector of length 1 by dividing an un-normalized feature vector *v* by its L2-norm:\n",
530+
"\n",
531+
"$$v_{\\text{norm}} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v_{1}^{2} + v_{2}^{2} + \\dots + v_{n}^{2}}} = \\frac{v}{\\big (\\sum_{i=1}^{n} v_{i}^{2}\\big)^\\frac{1}{2}}$$\n",
532+
"\n",
533+
"To make sure that we understand how TfidfTransformer works, let us walk\n",
534+
"through an example and calculate the tf-idf of the word is in the 3rd document.\n",
535+
"\n",
536+
"The word is has a term frequency of 3 (tf = 3) in document 3, and the document frequency of this term is 3 since the term is occurs in all three documents (df = 3). Thus, we can calculate the idf as follows:\n",
537+
"\n",
538+
"$$\\text{idf}(\"is\", d3) = log \\frac{1+3}{1+3} = 0$$\n",
539+
"\n",
540+
"Now in order to calculate the tf-idf, we simply need to add 1 to the inverse document frequency and multiply it by the term frequency:\n",
541+
"\n",
542+
"$$\\text{tf-idf}(\"is\",d3)= 3 \\times (0+1) = 3$$"
543+
]
544+
},
452545
{
453546
"cell_type": "code",
454-
"execution_count": 12,
547+
"execution_count": 16,
455548
"metadata": {
456549
"collapsed": false
457550
},
@@ -460,18 +553,43 @@
460553
"name": "stdout",
461554
"output_type": "stream",
462555
"text": [
463-
"tf-idf of term \"is\" = 2.00\n"
556+
"tf-idf of term \"is\" = 3.00\n"
464557
]
465558
}
466559
],
467560
"source": [
468-
"tf_is = 2 \n",
561+
"tf_is = 3\n",
469562
"n_docs = 3\n",
470563
"idf_is = np.log((n_docs+1) / (3+1))\n",
471564
"tfidf_is = tf_is * (idf_is + 1)\n",
472565
"print('tf-idf of term \"is\" = %.2f' % tfidf_is)"
473566
]
474567
},
568+
{
569+
"cell_type": "markdown",
570+
"metadata": {},
571+
"source": [
572+
"If we repeated these calculations for all terms in the 3rd document, we'd obtain the following tf-idf vectors: [3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]. However, we notice that the values in this feature vector are different from the values that we obtained from the TfidfTransformer that we used previously. The nal step that we are missing in this tf-idf calculation is the L2-normalization, which can be applied as follows:"
573+
]
574+
},
575+
{
576+
"cell_type": "markdown",
577+
"metadata": {},
578+
"source": [
579+
"$$\\text{tfi-df}_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n",
580+
"\n",
581+
"$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n",
582+
"\n",
583+
"$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$"
584+
]
585+
},
586+
{
587+
"cell_type": "markdown",
588+
"metadata": {},
589+
"source": [
590+
"As we can see, the results match the results returned by scikit-learn's `TfidfTransformer` (below). Since we now understand how tf-idfs are calculated, let us proceed to the next sections and apply those concepts to the movie review dataset."
591+
]
592+
},
475593
{
476594
"cell_type": "code",
477595
"execution_count": 13,
@@ -482,7 +600,7 @@
482600
{
483601
"data": {
484602
"text/plain": [
485-
"array([ 1.69, 2. , 1.29, 1.29, 1.29, 2. , 1.29])"
603+
"array([ 3.39, 3. , 3.39, 1.29, 1.29, 1.29, 2. , 1.69, 1.29])"
486604
]
487605
},
488606
"execution_count": 13,
@@ -506,7 +624,7 @@
506624
{
507625
"data": {
508626
"text/plain": [
509-
"array([ 0.4 , 0.48, 0.31, 0.31, 0.31, 0.48, 0.31])"
627+
"array([ 0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])"
510628
]
511629
},
512630
"execution_count": 14,
@@ -1289,21 +1407,21 @@
12891407
],
12901408
"metadata": {
12911409
"kernelspec": {
1292-
"display_name": "Python 2",
1410+
"display_name": "Python 3",
12931411
"language": "python",
1294-
"name": "python2"
1412+
"name": "python3"
12951413
},
12961414
"language_info": {
12971415
"codemirror_mode": {
12981416
"name": "ipython",
1299-
"version": 2
1417+
"version": 3
13001418
},
13011419
"file_extension": ".py",
13021420
"mimetype": "text/x-python",
13031421
"name": "python",
13041422
"nbconvert_exporter": "python",
1305-
"pygments_lexer": "ipython2",
1306-
"version": "2.7.11"
1423+
"pygments_lexer": "ipython3",
1424+
"version": "3.5.1"
13071425
}
13081426
},
13091427
"nbformat": 4,

‎docs/errata.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ I would be happy if you just write me a short [mail](mailto:mail@sebastianraschk
4848
10. Panos N. ($2.00)
4949
19. F. Liu ($2.00)
5050
20. Stefan P. ($2.00)
51-
20. Claude C. ($5.00)
51+
20. Claude C. ($6.00)
5252
11. Elias R. ($1.00)
5353
12. Haitham H. Saleh ($1.00)
5454
13. Muqueet M. ($1.00)

0 commit comments

Comments
 (0)