-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
batch_refresh Schedule阻塞 #230
Comments
遇到了同样问题 |
想了解一下 |
只是很慢。量化一点的话,感觉是剩1~2个线程在跑。 |
解决方案
|
隔了一年多,发现了问题的原因:mongoDB的pop经常返回同一个代理,常常出现几十个线程验证同一个代理,因而验证效率退化为单线程。应该跟$sample机制有关,在个人电脑上没有这样的问题,但在云主机上会有,可能随机数种子的更新和电脑的运算速度有关。 |
上述版本为1.13, 更换为1.14后问题解决 |
在验证raw_proxy的时候,常常发现raw_proxy堆积,同时伴有以上错误提示。同时会发现refresh新代理感觉回到了单线程的情况。
一旦raw_proxy不堆积,便不会有报错,很奇怪。
求个解决方案。
目前的解决方案是一旦人工发现堆积情况,便新开一个进程,这样就会很快地验证完raw_proxy。
The text was updated successfully, but these errors were encountered: