Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch_refresh Schedule阻塞 #230

Open
likenji opened this issue Dec 11, 2018 · 6 comments
Open

batch_refresh Schedule阻塞 #230

likenji opened this issue Dec 11, 2018 · 6 comments

Comments

@likenji
Copy link

likenji commented Dec 11, 2018

WARNING:apscheduler.scheduler:Execution of job "batch_refresh (trigger: interval[0:01:00], next run at: 2018-12-11 13:14:33 CST)" skipped: maximum number of running instances reached

在验证raw_proxy的时候,常常发现raw_proxy堆积,同时伴有以上错误提示。同时会发现refresh新代理感觉回到了单线程的情况。
一旦raw_proxy不堆积,便不会有报错,很奇怪。
求个解决方案。
目前的解决方案是一旦人工发现堆积情况,便新开一个进程,这样就会很快地验证完raw_proxy。

@ghost
Copy link

ghost commented Dec 11, 2018

WARNING:apscheduler.scheduler:Execution of job "batch_refresh (trigger: interval[0:01:00], next run at: 2018-12-11 13:14:33 CST)" skipped: maximum number of running instances reached

在验证raw_proxy的时候,常常发现raw_proxy堆积,同时伴有以上错误提示。同时会发现refresh新代理感觉回到了单线程的情况。
一旦raw_proxy不堆积,便不会有报错,很奇怪。
求个解决方案。
目前的解决方案是一旦人工发现堆积情况,便新开一个进程,这样就会很快地验证完raw_proxy。

遇到了同样问题

@likenji likenji changed the title batch_refresh batch_refresh Schedule阻塞 Dec 12, 2018
@ozhiwei
Copy link
Contributor

ozhiwei commented Dec 13, 2018

想了解一下
是完全"无法验证raw_proxy"
还是"验证raw_proxy很慢"

@likenji
Copy link
Author

likenji commented Dec 13, 2018

想了解一下
是完全"无法验证raw_proxy"
还是"验证raw_proxy很慢"

只是很慢。量化一点的话,感觉是剩1~2个线程在跑。

@ozhiwei
Copy link
Contributor

ozhiwei commented Dec 13, 2018

解决方案 proxy_pool/Schedule/ProxyRefreshSchedule.py:102

minutes=1 修改成 minutes=30, 这个时间还是要看raw_proxy的大小

@likenji
Copy link
Author

likenji commented Feb 24, 2020

隔了一年多,发现了问题的原因:mongoDB的pop经常返回同一个代理,常常出现几十个线程验证同一个代理,因而验证效率退化为单线程。应该跟$sample机制有关,在个人电脑上没有这样的问题,但在云主机上会有,可能随机数种子的更新和电脑的运算速度有关。

@likenji
Copy link
Author

likenji commented Feb 24, 2020

上述版本为1.13, 更换为1.14后问题解决

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants