Q2-itsxpress reformat.sh error

Charles_Hauser · July 23, 2018, 6:39pm

Hi Adam

I received an error when attempting to run itsxpress on single-end sequences that had been demultiplexed using :
qiime demux emp-single
–i-seqs seqs.qza
–m-barcodes-file relabeled_sx2_3_mapping.txt
–m-barcodes-column BarcodeSequence
–o-per-sample-sequences seqs.demux.qza

qiime itsxpress trim-single --i-per-sample-sequences seqs.demux.qza --p-region ALL --p-taxa F --p-threads 8 --output-dir ITSXPRESS
Plugin error from itsxpress:

Command ‘[‘reformat.sh’, ‘in=/tmp/qiime2-archive-45x7dl5h/13133601-24ae-4dc2-a3ec-a900cfb9f004/data/5H2_1_L001_R1_001.fastq.gz’, ‘reads=50’]’ returned non-zero exit status 1

Debug info has been saved to /tmp/qiime2-q2cli-err-7vn2og_9.log

The log file indicates a problem with the fastq file:

ERROR:root:There appears to be an issue with your input fastq or fastq.gz file(s).
Traceback (most recent call last):
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/q2cli/commands.py”, line 274, in call
results = action(**arguments)
File “”, line 2, in trim_single
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 232, in bound_callable
output_types, provenance)
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/qiime2/sdk/action.py”, line 367, in callable_executor
output_views = self._callable(**view_args)
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_itsxpress/_itsxpress.py”, line 213, in trim_single
cluster_id=cluster_id)
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_itsxpress/_itsxpress.py”, line 278, in main
threads=threads)
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/q2_itsxpress/_itsxpress.py”, line 96, in _set_fastqs_and_check
itsxpress._check_fastqs(fastq=fastq, fastq2=fastq2)
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/itsxpress/main.py”, line 580, in _check_fastqs
raise e
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/site-packages/itsxpress/main.py”, line 577, in _check_fastqs
p1.check_returncode()
File “/home/charlesh/binf/miniconda/envs/qiime2-2018.6/lib/python3.5/subprocess.py”, line 349, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command ‘[‘reformat.sh’, ‘in=/tmp/qiime2-archive-45x7dl5h/13133601-24ae-4dc2-a3ec-a900cfb9f004/data/5H2_1_L001_R1_001.fastq.gz’, ‘reads=50’]’ returned non-zero exit status 1

Charles

Adam_Rivers · July 24, 2018, 7:33pm

Hi @Charles_Hauser,

The error you got is supposed be thrown if there is a problem with the input fastq file. It’s possible that the input fastq file is not formatted correctly, however @kweber and I saw this same error 2 times with good files in about 800 file runs so it may be an odd intermittent error.

The error occurs in a portion of the program that is meant validate input files so we may be able to disable it temporarily while we determine if the problem is coming from bbtools or the subprocess module. Its also possible that if you run it again it will work fine. I’d try to just rerun it while we investigate. If you would like to share your data files with us we can also work directly with them. Feel free to PM me to coordinate transferring data if you would like to do that.

Adam_Rivers · July 24, 2018, 7:58pm

If I run BBtools reformat.sh many times outside of the itsxpress it periodically throws an error. I’ll contact the BBtools author. In the mean time I’m looking into other fastq validation methods.

Charles_Hauser · July 24, 2018, 9:27pm

Adam

Thanks for your help.
If this is of any help, I have rerun
qiime itsxpress trim-single --i-per-sample-sequences seqs.demux.qza --p-region ALL --p-taxa F --p-threads 8 --output-dir ITSXPRESS

3x and it throws same error.

I was able to run qiime dada2 denoise-single
using the same input file (seqs.demux.qza) without error.

Charles

Adam_Rivers · July 25, 2018, 3:08pm

Hi @Charles_Hauser,

ITSxpress v1.6.4 fixes this issue.

It takes a day or two for my updates to be approved on bioconda but you can get the update now from pip.

First uninstall the conda version of ITSxpress:
conda uninstall itsxpress

Then reinstall it with pip:
pip install itsxpress

Previously, ITSxpress used the BBtools script reformat.sh for FASTQ file validation. Sometimes reformat.sh would encounter a race condition and produce an exception which caused ITSxpress to incorrectly call valid fastq files invalid. The BBtools author just fixed this issue and released v38.17. However, I went ahead and switched to using Biopython for FASTQ file validation because it has more extensive validation.

Thanks for your feedback.

Adam_Rivers · July 25, 2018, 8:28pm

Hi @Charles_Hauser,

Sorry I see now that there’s still an issue with the plugin although the main application works. I’ll update once we fix it.

Adam_Rivers · July 26, 2018, 4:49pm

Okay, v1.6.4 really fixes the issue. Sorry for the delay.

First uninstall the conda version of ITSxpress:
conda uninstall itsxpress
(or if you installed with pip use this command):
pip uninstall itsxpress

Then reinstall it with pip and refresh Qiime

pip install itsxpress
pip install --upgrade q2-itsxpress
qiime dev refresh-cache

system · August 26, 2018, 10:56pm

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.