Crawler never complete

[ForexTalk11]

Hi,

The crawler never complete because of "Stopped due to reset meta position". Is that expected?

Thanks.

 

[hai]

Hi,

Before crawler finishes its full process, did you click "Generate Crawler File" again? That button doesn't need to be clicked manually.

Otherwise, please check the `var` folder under your plugin, make sure all files in it is writable.

 

[ForexTalk11]

Hi,

No, I don't click "Generate Crawler File". I manually run but it only craw around 1000 and then stuck there forever unless I run manually again.

Also, Threads: 6 and cannot be increase further.

 

[hai]

Is your server a 6-cpu server? honestly 6 threads are high enough already.

What is your crawler setting? `Run Duration` and `Interval Between Runs`?

 

[ForexTalk11]

it's 24CPU server, I just want to test how much it can handle, if it stress the server out I will change it. Unfortunately, thread is not allow update over 6.

`Run Duration` and `Interval Between Runs` use default setting 400 and 600s.

 

[ForexTalk11]

@NiteWave: any imput for the issue please?

Thanks.

 

[hai]

The reset action is triggered when there is a new position reset request, which can be only triggered when you click that button manually.
Can you create a tix that includes your admin and ssh info so we can have a look at it?

 

[ForexTalk11]

As testing, I see that "reset meta position" happen when new tag added. For website which frequently have new contents and tags, the crawls will never completed.

 

[hai]

That is you have purge_all setting for add a new post.
Every time when crawler detected a purge_all action, it will stop the crawling action and restart from the beginning.