Error while getting plenty of data from NCBI

[TurboQiimer]

Hi again guys,

It sounds parameter --p-n-jobs with digit 5 is unable to work in my case when a lot of data is downloading.

While I wanted to download lots of data with gene name, I got this error (it take around 20 min to get the error):

qiime rescript get-ncbi-data \

--p-query dsrB \
--o-sequences RefsequencesdsrB.qza \

--p-n-jobs 5
--o-taxonomy 11.qza
Plugin error from rescript:

A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGKILL(-9)}

Debug info has been saved to /tmp/qiime2-q2cli-err-3qh5xqo6.log

In the guideline it mentions to digit 5 " If you are downloading lots of data, you should set the --p-n-jobs parameter to a number greater than one. Five works well in most cases." It failed then!

The only way that I catch all data is gene name, because accession number and GI are exclusively unique, then I used gene name in order to get all data. Would it be the point? How the problem could be solved? What do you think?

Thanks a lot.
Qiimer

[TurboQiimer]

I used a server (remote machine)to prevent memory-associated limitation. After around two hours, it reported an unknown error.

qiime rescript get-ncbi-data \

--p-query dsrB \
--o-sequences RefsequencesdsrB.qza \

--p-n-jobs 5
--o-taxonomy 11.qza
Plugin error from rescript:

Download did not finish. Reason unknown.

Debug info has been saved to /tmp/qiime2-q2cli-err-we4d1npy.log

I attached this result to complete my report.

Best regards,
Qiimer

[SoilRotifer (Mike Robeson)]

@TurboQiimer

A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGKILL(-9)}

It looks like you figured this out as per:

@TurboQiimer

I used a server (remote machine)to prevent memory-associated limitation

Your next error:

@TurboQiimer

Download did not finish. Reason unknown.

can be problematic to figure out. There are many "out of our hands" reasons that fetching data from NCBI can fail:

rescript get-ncbi-data ‘dict’ object has no attribute ‘add’

Hey @the_dummy, Just to follow on @SoilRotifer’s comments to clarify — we see various server-side issues with NCBI so sometimes using get-ncbi-data can be a bit bumpy depending on: the time of day the size/content of your query which way the wind is blowing (just kidding ) Usually trying again later works (e.g., if you are trying to run the job during peak hours) and we have some pending changes to rescript to make these failures more graceful in the future. Good luck!

I'd suggest trying to add more selective terms to your query, or try to download in batches. This is a great place to start:

Building a COI database from NCBI references

Citation: If you use the following COI resources or RESCRIPt for COI database preparation, please cite the following: Michael S Robeson II, Devon R O’Rourke, Benjamin D Kaehler, Michal Ziemski, Matthew R Dillon, Jeffrey T Foster, Nicholas A Bokulich. RESCRIPt: Reproducible sequence taxonomy reference database management for the masses. bioRxiv 2020.10.05.326504; …

Good luck!