qiime itsxpress trim-single fatal error: file too small

Library Plugin Support
1

Hello -

I am trying to trim single-read (R1) ITS sequences (generated using the EMP protocol) with Q2-ITSxpress prior to running dada2. I have run this script successfully on other datasets, but am running into an error this time and I'm not sure what the problem is. I am using Qiime2-2019.4 on Amazon AWS.

Here is the command:

qiime itsxpress trim-single --i-per-sample-sequences demux_forward.qza --p-region ITS1 --p-taxa F --o-trimmed ITSxpress_trimmed.qza --verbose

Here is the output:

vsearch v2.7.0_linux_x86_64, 67.2GB RAM, 8 cores

Fatal error: File too small

ERROR:root:Could not perform clustering with Vsearch. Error from Vsearch was:
vsearch v2.7.0_linux_x86_64, 67.2GB RAM, 8 cores

Fatal error: File too small
Traceback (most recent call last):
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 557, in cluster
p2.check_returncode()
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command '['vsearch', '--cluster_size', '/tmp/itsxpress_j7kb6gum/seq.fq.gz', '--centroids', '/tmp/itsxpress_j7kb6gum/rep.fa', '--uc', '/tmp/itsxpress_j7kb6gum/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in call
results = action(**arguments)
File "</home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-273>", line 2, in trim_single
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in callable_executor
output_views = self._callable(**view_args)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py", line 119, in trim_single
cluster_id=cluster_id)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py", line 198, in main
sobj.cluster(threads=threads, cluster_id=cluster_id)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 560, in cluster
raise e
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 557, in cluster
p2.check_returncode()
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command '['vsearch', '--cluster_size', '/tmp/itsxpress_j7kb6gum/seq.fq.gz', '--centroids', '/tmp/itsxpress_j7kb6gum/rep.fa', '--uc', '/tmp/itsxpress_j7kb6gum/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.

Plugin error from itsxpress:

Command '['vsearch', '--cluster_size', '/tmp/itsxpress_j7kb6gum/seq.fq.gz', '--centroids', '/tmp/itsxpress_j7kb6gum/rep.fa', '--uc', '/tmp/itsxpress_j7kb6gum/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.

See above for debug info.

I have checked other discussions on the forum (this and this), and it seems like /tmp/itsxpress_j7kb6gum/seq.fq.gz' may not have any sequences? Maybe the plug-in is trying to merge reads, despite running the trim-single command? Here is the demuxed qzv demux_forward.qzv (292.4 KB) I validated my demuxed sequences using the qiime tools validate function and all looked good. So, I'm not sure what the problem is. Any insights you could provide would be greatly appreciated!

Chelsea

2

Hello, I'm just checking back in on this. Any thoughts on what might be the issue and how to proceed? I tried running my demuxed samples through dada2 just to see, and that worked fine, but ITSxpress is still throwing that same error. Thanks!

3

Hi @ccarey,
Sorry for the delay and thank you for your patience.

I am cc:ing @Adam_Rivers to see if he has any thoughts about this.

4

Yes that was the issue. I fixed it a few weeks ago, along with a feature addition, but I never pushed out the new version. I'll try to do that next week, you can get the fixed code from GitHub if you need it right away.

1 Like
5

Great, thank you! Very helpful.

6

Hi Adam,

It's been a while, but I'm finally picking this project back up. With the same command (qiime itsxpress trim-single --i-per-sample-sequences demux_forward.qza --p-region ITS1 --p-taxa F --o-trimmed ITSxpress_trimmed.qza --verbose) I'm still getting the error:

Fatal error: File too small
Traceback (most recent call last):
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 557, in cluster
p2.check_returncode()
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command '['vsearch', '--cluster_size', '/tmp/itsxpress_sype0axq/seq.fq.gz', '--centroids', '/tmp/itsxpress_sype0axq/rep.fa', '--uc', '/tmp/itsxpress_sype0axq/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/commands.py", line 311, in call
results = action(**arguments)
File "</home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/decorator.py:decorator-gen-273>", line 2, in trim_single
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 231, in bound_callable
output_types, provenance)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/action.py", line 365, in callable_executor
output_views = self._callable(**view_args)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py", line 119, in trim_single
cluster_id=cluster_id)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/q2_itsxpress/_itsxpress.py", line 198, in main
sobj.cluster(threads=threads, cluster_id=cluster_id)
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 560, in cluster
raise e
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/site-packages/itsxpress/main.py", line 557, in cluster
p2.check_returncode()
File "/home/qiime2/miniconda/envs/qiime2-2019.4/lib/python3.6/subprocess.py", line 369, in check_returncode
self.stderr)
subprocess.CalledProcessError: Command '['vsearch', '--cluster_size', '/tmp/itsxpress_sype0axq/seq.fq.gz', '--centroids', '/tmp/itsxpress_sype0axq/rep.fa', '--uc', '/tmp/itsxpress_sype0axq/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.

Plugin error from itsxpress:

Command '['vsearch', '--cluster_size', '/tmp/itsxpress_sype0axq/seq.fq.gz', '--centroids', '/tmp/itsxpress_sype0axq/rep.fa', '--uc', '/tmp/itsxpress_sype0axq/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.

As a workaround, I tried standalone ITSxpress and kept running out of space with this command even though I have >60 GB available.

Any help you could provide would be greatly appreciated!
Thanks,
Chelsea

7

Hi there,
I am also passing through this issue right now.
Tried those suggestions around with no solution too.
Do you guys have any update on that?
Chelsea, did you find your way to fix it?

All the best
Leo

8

No, I have not found a way around this issue and have not heard an update on how to proceed.

@Adam_Rivers or @Nicholas_Bokulich - any thoughts?

9

Thank you Chelsea and an update:
From yesterday I've been working on it and found out that running standalone itsxpress works. I am now going downstream the pipeline to evaluate the process.

Command (in an active qiime2 environment) for single-end reads:
itsxpress --fastq read.fastq.gz --single_end --outfile read.trim.fastq.gz --region ITS2 --taxa Fungi --cluster_id 1 --threads 15

Then reimport the fastqs to qiime.

Cheers,

10

Thanks! I've tried to run standalone but every time I do I run out of storage despite having >60 GB available. Does this sound right based on your experience?

11

Well, based on my experience with other issues it seems like :

If you're running out of storage maybe your directory where Qiime is installed is full, so itsxpress (which is installed whithin Qiime environment) can't create tmp files.
Check how full are your file systems with df -h
Also, have a look here and see if that's make sense for your case: No space left on device classifier TMPDIR

If at least itsxpress start running, check this line from the log in the screen:
2019-12-10 16:26:58,419: INFO Temporary directory is: /any/path/itsxpress_jnvs8k7r
and see if /any/path/ points to a place in a full disk. That is the default diretory Qiime uses for storing tmp files.

I have got into this issue and in my case the /home/ was full because that's were Conda-qiime is installed. I then set the TMP directory to a directory placed in a larger disk and it worked.
Cheers

12

The tempdir issue seems like the most plausible explanation. Itsxpress also has a —tempdir flag so you can set it to a new location.

Also, itsxpress 1.8.0 was just released with support for reversed orientation primers and better error handling with empty files. They are available from github or pypi now and will be available on conda in the next day or so.

1 Like
13

I am downloading it!

Do you mean, for example, forward primers in reverse orientantion in the 3' end?

14

Primers where the forward primer goes in the reverse direction in the genome, like the Taylor et al primer set:Accurate Estimation of Fungal Diversity and Abundance through Improved Lineage-Specific Primers Optimized for Illumina Amplicon Sequencing - PMC

2 Likes
15

I was able to run standalone itsxpress successfully by changing the TMP directory. So thank you for that recommendation! I re-imported the sequences and am now running into issues when trying to run the demux command:

qiime demux emp-single --m-barcodes-column BarcodeSequence --i-seqs imported_data_forward.qza --o-per-sample-sequences demux_forward.qza --p-rev-comp-mapping-barcodes --o-error-correction-details demux_error.qza --p-no-golay-error-correction --m-barcodes-file Carey_ITS_Mappingfile.txt

This is the error:

Plugin error from demux: Mismatched sequence ids: M02149:257:000000000-CBYGW:1:1101:17084:1398 and M02149:257:000000000-CBYGW:1:1101:18572:1830

I've looked through other related posts and it seems like the barcode and sequence files don't match up when using the file that was trimmed with ITSxpress. I checked how many lines are in each file per this post and it seems I've lost quite a bit from the sequences file after running through ITSxpress:

wc -l barcodes.fastq: 74,129,196
wc -l Undetermined_S0_L001_R1_001.fastq (raw sequences): 74,129,196
wc -l sequences.fastq (file I'm using that was run through ITSxpress): 48,958,160

Any thoughts on what might be going on and how to fix it?

Thanks!!
Chelsea

16

I may not understand the question but I think demux should be run before using itsxpress.

1 Like
17

Yeah, that makes sense and is the way I was doing it with the q2-itsxpress plugin, but standalone ITSxpress only seems to take .fastq files while demux produces a .qza file. If I export the .qza file it gives me all of my samples demultiplexed in .fastq format, but doesn't give just one sequences.fastq file that I can then pass through ITSxpress. Do I have to run it separately for each sample? That doesn't seem right.. what am I missing? Thanks!!

18

Just following up again. Any thoughts @Adam_Rivers?

1 Like
19

Hi Chelsea and Adam

I'm in same predicament:

  • qiime itsxpress run on an app on cyverse throws error others have encountered (below)
  • running standalone in same app on non-demultiplexed seqs works,
  • unable to demultiplex resulting ITS2 reads due to mismatch in barcode & sx IDs

Plugin error from demux:

Mismatched sequence ids: ICKS16J02J28PP and ICKS16J02F6FMM

Debug info has been saved to /tmp/qiime2-q2cli-err-n0_y48ig.log

Is it necessary to run on each demultiplexed sample?

error from qiime itsxpress:
Command '['vsearch', '--cluster_size', '/tmp/itsxpress_snqx84k1/seq.fq.gz', '--centroids', '/tmp/itsxpress_snqx84k1/rep.fa', '--uc', '/tmp/itsxpress_snqx84k1/uc.txt', '--strand', 'both', '--id', '0.995', '--threads', '1']' returned non-zero exit status 1.

Debug info has been saved to /tmp/qiime2-q2cli-err-00p6lr_m.log

Charles

20

If I look at files in the vsearch /tmp/itsxpress dir there are sequences for both .r1, and .r2 , but the command executed was for single-end reads:

extract ITS2 seqs - this fails

!qiime itsxpress trim-single
--i-per-sample-sequences bastrop-sx2-outputs/sx2.demux.qza
--p-region ITS2
--p-taxa F
--o-trimmed bastrop-sx2-outputs/sx2.demux.ITS2.qza

/tmp/itsxpress dir

ls -l /tmp/itsxpress_fp0p5hbq/
total 6872
-rw-r--r-- 1 qiime2 qiime2 0 Mar 19 16:18 rep.fa
-rw-r--r-- 1 qiime2 qiime2 20 Mar 19 16:18 seq.fq.gz
-rw-r--r-- 1 qiime2 qiime2 3524967 Mar 19 16:18 seq_r1.fq.gz
-rw-r--r-- 1 qiime2 qiime2 3502500 Mar 19 16:18 seq_r2.fq.gz
-rw-r--r-- 1 qiime2 qiime2 0 Mar 19 16:18 uc.txt