qiime itsxpress trim-single fatal error: file too small

Hello -

I am trying to trim single-read (R1) ITS sequences (generated using the EMP protocol) with Q2-ITSxpress prior to running dada2. I have run this script successfully on other datasets, but am running into an error this time and I’m not sure what the problem is. I am using Qiime2-2019.4 on Amazon AWS.

Here is the command:

qiime itsxpress trim-single --i-per-sample-sequences demux_forward.qza --p-region ITS1 --p-taxa F --o-trimmed ITSxpress_trimmed.qza --verbose

Here is the output:

vsearch v2.7.0_linux_x86_64, 67.2GB RAM, 8 cores

Fatal error: File too small

ERROR:root:Could not perform clustering with Vsearch. Error from Vsearch was:
vsearch v2.7.0_linux_x86_64, 67.2GB RAM, 8 cores

Fatal error: File too small
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 557, in cluster
p2.check_returncode()
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py”, line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_j7kb6gum/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_j7kb6gum/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_j7kb6gum/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py”, line 311, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-273>”, line 2, in trim_single
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 231, in bound_callable
output_types, provenance)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 365, in callable_executor
output_views = self._callable(**view_args)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py”, line 119, in trim_single
cluster_id=cluster_id)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py”, line 198, in main
sobj.cluster(threads=threads, cluster_id=cluster_id)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 560, in cluster
raise e
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 557, in cluster
p2.check_returncode()
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py”, line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_j7kb6gum/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_j7kb6gum/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_j7kb6gum/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.

Plugin error from itsxpress:

Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_j7kb6gum/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_j7kb6gum/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_j7kb6gum/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.

See above for debug info.

I have checked other discussions on the forum (this and this), and it seems like /tmp/itsxpress_j7kb6gum/seq.fq.gz’ may not have any sequences? Maybe the plug-in is trying to merge reads, despite running the trim-single command? Here is the demuxed qzv demux_forward.qzv (292.4 KB) I validated my demuxed sequences using the qiime tools validate function and all looked good. So, I’m not sure what the problem is. Any insights you could provide would be greatly appreciated!

Chelsea

Hello, I’m just checking back in on this. Any thoughts on what might be the issue and how to proceed? I tried running my demuxed samples through dada2 just to see, and that worked fine, but ITSxpress is still throwing that same error. Thanks!

Hi @ccarey,
Sorry for the delay and thank you for your patience.

I am cc:ing @Adam_Rivers to see if he has any thoughts about this.

Yes that was the issue. I fixed it a few weeks ago, along with a feature addition, but I never pushed out the new version. I’ll try to do that next week, you can get the fixed code from GitHub if you need it right away.

1 Like

Great, thank you! Very helpful.

Hi Adam,

It’s been a while, but I’m finally picking this project back up. With the same command (qiime itsxpress trim-single --i-per-sample-sequences demux_forward.qza --p-region ITS1 --p-taxa F --o-trimmed ITSxpress_trimmed.qza --verbose) I’m still getting the error:

Fatal error: File too small
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 557, in cluster
p2.check_returncode()
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py”, line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_sype0axq/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_sype0axq/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_sype0axq/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py”, line 311, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-273>”, line 2, in trim_single
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 231, in bound_callable
output_types, provenance)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 365, in callable_executor
output_views = self._callable(**view_args)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py”, line 119, in trim_single
cluster_id=cluster_id)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py”, line 198, in main
sobj.cluster(threads=threads, cluster_id=cluster_id)
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 560, in cluster
raise e
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py”, line 557, in cluster
p2.check_returncode()
File “/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py”, line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_sype0axq/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_sype0axq/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_sype0axq/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.

Plugin error from itsxpress:

Command ‘[‘vsearch’, ‘–cluster_size’, ‘/tmp/itsxpress_sype0axq/seq.fq.gz’, ‘–centroids’, ‘/tmp/itsxpress_sype0axq/rep.fa’, ‘–uc’, ‘/tmp/itsxpress_sype0axq/uc.txt’, ‘–strand’, ‘both’, ‘–id’, ‘0.995’, ‘–threads’, ‘1’]’ returned non-zero exit status 1.

As a workaround, I tried standalone ITSxpress and kept running out of space with this command even though I have >60 GB available.

Any help you could provide would be greatly appreciated!
Thanks,
Chelsea

Hi there,
I am also passing through this issue right now.
Tried those suggestions around with no solution too.
Do you guys have any update on that?
Chelsea, did you find your way to fix it?

All the best
Leo

No, I have not found a way around this issue and have not heard an update on how to proceed.

@Adam_Rivers or @Nicholas_Bokulich - any thoughts?

Thank you Chelsea and an update:
From yesterday I’ve been working on it and found out that running standalone itsxpress works. I am now going downstream the pipeline to evaluate the process.

Command (in an active qiime2 environment) for single-end reads:
itsxpress --fastq read.fastq.gz --single_end --outfile read.trim.fastq.gz --region ITS2 --taxa Fungi --cluster_id 1 --threads 15

Then reimport the fastqs to qiime.

Cheers,

Thanks! I’ve tried to run standalone but every time I do I run out of storage despite having >60 GB available. Does this sound right based on your experience?

Well, based on my experience with other issues it seems like :

If you’re running out of storage maybe your directory where Qiime is installed is full, so itsxpress (which is installed whithin Qiime environment) can’t create tmp files.
Check how full are your file systems with df -h
Also, have a look here and see if that’s make sense for your case: No space left on device classifier TMPDIR

If at least itsxpress start running, check this line from the log in the screen:
2019-12-10 16:26:58,419: INFO Temporary directory is: /any/path/itsxpress_jnvs8k7r
and see if /any/path/ points to a place in a full disk. That is the default diretory Qiime uses for storing tmp files.

I have got into this issue and in my case the /home/ was full because that’s were Conda-qiime is installed. I then set the TMP directory to a directory placed in a larger disk and it worked.
Cheers

The tempdir issue seems like the most plausible explanation. Itsxpress also has a —tempdir flag so you can set it to a new location.

Also, itsxpress 1.8.0 was just released with support for reversed orientation primers and better error handling with empty files. They are available from github or pypi now and will be available on conda in the next day or so.

1 Like

I am downloading it!

Do you mean, for example, forward primers in reverse orientantion in the 3’ end?

Primers where the forward primer goes in the reverse direction in the genome, like the Taylor et al primer set:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5118932/

2 Likes