Time required for importing data

I am trying to import 56 fastq piles (paired-end reads with 28 fastq files for forward and reverse reads each) on qiime2 - 2019.4 version. I am referring to the corresponding data importing tutorial for the manifest protocol.
It has been more than 30 minutes but the command is still running. Does it usually take this long? No error has popped up yet neither the imported data. I am stuck. I am using MAC, and the running speed is usually good on this computer.

Hi @SAHIL_JAIN_16110144,
You’re right that this is a bit longer than usual, unless you have some crazy big files, but if it is still running I would let it run until completion or at least let it reach an error that way we can work with the error. If you open a new terminal and run top can you confirm that the job is still running? Or that how much memory is dedicated to the task?

1 Like

I had to stop the run. I reduced the data size to 4 fastq files just to check if there is something wrong with the command. The data got imported and it took almost 20 minutes to import 4 files. This is a huge setback because I have to analyse samples from more than just 1 study (a meta analysis). Does multi core processing work for Qiime?

What is the size of those files? Unless if they are massive files, this runtime is very unusual (importing even large fastqs usually takes seconds or minutes, not hours). I suspect it is a problem with the file size or else with your system, you may want to check with your system administrator to see what is causing the slowdown. This is not an issue with QIIME 2, which you have confirmed is working (though slower than is typical).

Yes this is a routine use of QIIME 2 — yet importing many datasets for meta-analysis usually takes minutes.

Many actions in QIIME 2 can run in parallel, but not importing.

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.