demux summarize error after importing data (QIIME2 2022.11)

[ryunheekim]

Hello, everyone!

I'm super new to QIIME2 and I got the same error messages when conducting 'demux summarize' several times.

I think I've gotten the sequencing data which have been already demultiplexed ('sequences are separated into different files based on the kind of sample.'), so I jumped into the 'demux summarize' step to see the quality scores after importing data.

Here is the code I used, the error messages, and the log I got.

Code)
qiime demux summarize --i-data paired-end-demux.qza --o-visualization demux.qzv

Error messages)
Plugin error from demux:
CRC check failed 0xe020340 != 0xac99c05f
Debug info has been saved to /tmp/qiime2-q2cli-err-f9s_ppuj.log

Log)
Traceback (most recent call last):
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 352, in call
results = action(**arguments)
File "", line 2, in summarize
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 443, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_demux/_summarize/_visualizer.py", line 140, in summarize
for seq in _read_fastq_seqs(filename):

Followings are troubleshootings I tried after googling;

1. Re-download fastq files (.fq.gz) from the sequencing center
2. Re-import data (.fq.gz --> .qza)

BUT It didn't work at all ...

I would be really grateful if you provide any suggestions for me.
Thanks in advance..!

P.S. There were no problems at all at the importing data step.

Here are codes and some parts of manifest file used in the 'importing data' step;

Code)
qiime tools import
--type 'SampleData[PairedEndSequencesWithQuality]'
--input-path microbiome-data-manifest
--output-path paired-end-demux.qza
--input-format PairedEndFastqManifestPhred33V2

Manifest file)

Sample-id/ forward-absolute-filepath/ reverse-absolute-filepath (column separated)
Sample-1/ $PWD/microbiome-data/CM1B_1.fq.gz/ $PWD/microbiome-data/CM1B_2.fq.gz
...

[lizgehret]

Hi @ryunheekim,

Welcome to the forum!

Can you please re-run your command with the --verbose flag? This will provide additional de-bugging information so we can better see what's causing this error. Thanks!

[ryunheekim]

Thank you!

I re-runned with the "--verbose" flag as you suggested and the followings are what I got.

Traceback (most recent call last):
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 352, in call
results = action(**arguments)
File "", line 2, in summarize
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 234, in bound_callable
outputs = self.callable_executor(scope, callable_args,
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/qiime2/sdk/action.py", line 443, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_demux/_summarize/_visualizer.py", line 140, in summarize
for seq in _read_fastq_seqs(filename):
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2_demux/_demux.py", line 63, in _read_fastq_seqs
for seq_header, seq, qual_header, qual in itertools.zip_longest(*[fh] * 4):
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/gzip.py", line 305, in read1
return self._buffer.read1(size)
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/_compression.py", line 68, in readinto
data = self.read(len(byte_view))
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/gzip.py", line 470, in read
self._read_eof()
File "/home/ryunhee/anaconda3/envs/qiime2-2022.11/lib/python3.8/gzip.py", line 516, in _read_eof
raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32),
gzip.BadGzipFile: CRC check failed 0xe020340 != 0xac99c05f

Plugin error from demux:

CRC check failed 0xe020340 != 0xac99c05f

See above for debug info.

[lizgehret]

Hi @ryunheekim,

Thanks for sharing this log! It looks like this is the source of the issue:

raise BadGzipFile("CRC check failed %s != %s" % (hex(crc32),
gzip.BadGzipFile: CRC check failed 0xe020340 != 0xac99c05f

Do you mind sharing your paired-end-demux.qza file for me to take a look at? You can send it to me in a private message or upload via Google Drive or Dropbox - whatever you'd prefer

Cheers

[ryunheekim]

Dear Liz Gehret,

I sent you a message with the Dropbox link.

If you have anything to comment on about the files, please let me know.

Thank you

[lizgehret]

Hi @ryunheekim,

Thanks for providing those files! I'm seeing the same error that you are; this looks to be caused by some missing or corrupt data in one or more of your fastq files. You mentioned that you had already re-downloaded your fastq files from your sequencing center - I would confirm with them that these files haven't been modified in any way since they were originally uploaded.

It's also possible that the method of file transfer you are using is corrupting the files in some way. If possible, I would try downloading them once more, and then running qiime tools validate on your freshly imported .qza file.

Hope this helps, please let us know what you hear back from the sequencing center!

[ryunheekim]

Hello, Liz Gehret,

The sequencing center confirmed that there are no errors in the fastq files.

Afterward, I tried re-installing qiime2, re-downloading the files and changed the way of saving the files (flash drive --> Dropbox). This worked!

So now I can perform subsequent qiime2 plugins without errors.

Again, I really appreciate your help.