Add Isna Persian Dataset (LAION-AI#3631)

pourmand1376 · olliestanley · web-flow · commit 0d4adb5f1ad6 · 2023-08-03T14:09:31.000Z
The level of importance of this data is less than Wikipedia. So, I think [this pull request](LAION-AI#3629) should be merged first. I have uploaded the data to [huggingface](https://huggingface.co/datasets/pourmand1376/isna-news) according to Open-assistant's standard. So, it shouldn't need any processing. --------- Co-authored-by: Oliver Stanley <olivergestanley@gmail.com>
diff --git a/data/datasets/__init__.py b/data/datasets/__init__.py
@@ -4,6 +4,7 @@
     "tv_dialogue": "sedthh/tv_dialogue",  # TV and Movie dialogues and transcripts
     "fd_dialogue": "sedthh/fd_dialogue",  # TV and Movie dialogues and transcripts from ForeverDreaming
     "tlcv2.0_oa": "pythainlp/tlcv2.0_oa",  # Thai classical literature texts
+    "fa-isna-news": "pourmand1376/isna-news",  # Isna Persian News
     "fa-wikipedia": "pourmand1376/fa-wikipedia",  # Farsi Wikipedia texts
 }
 
diff --git a/data/datasets/fa-isna-news/README.md b/data/datasets/fa-isna-news/README.md
@@ -0,0 +1,2 @@
+This text-only dataset is crawled from [Isna news](https://isna.ir/). This is
+biggest farsi news agency and thus the text is pretty clean.

Original file line number	Diff line number	Diff line change
`@@ -4,6 +4,7 @@`
`4`	`4`	`"tv_dialogue": "sedthh/tv_dialogue", # TV and Movie dialogues and transcripts`
`5`	`5`	`"fd_dialogue": "sedthh/fd_dialogue", # TV and Movie dialogues and transcripts from ForeverDreaming`
`6`	`6`	`"tlcv2.0_oa": "pythainlp/tlcv2.0_oa", # Thai classical literature texts`
	`7`	`+ "fa-isna-news": "pourmand1376/isna-news", # Isna Persian News`
`7`	`8`	`"fa-wikipedia": "pourmand1376/fa-wikipedia", # Farsi Wikipedia texts`
`8`	`9`	`}`
`9`	`10`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+This text-only dataset is crawled from [Isna news](https://isna.ir/). This is`
	`2`	`+biggest farsi news agency and thus the text is pretty clean.`