Compressed file ended before the end-of-stream marker was reached (Song et al)

[kaitlineblen27]

Hello I am second year an undergraduate research assistant, I am working on importing the fastq files from Song et al 2020 (Comparative Analyses of Vertebrate Gut Microbiomes Reveal Convergence between Birds and Bats) to do analysis in Qiime2. I downloaded the fastq files from ENA and unzipped them. I have also created a manifest file. However, I now receiving an error when trying to complete the demux summarize step. Here is the code I used:
qiime tools import
--type 'SampleData[SequencesWithQuality]'
--input-path mapping-file-song.txt
--output-path song-single-end-demux.qza
--input-format SingleEndFastqManifestPhred33V2
Then, with the following syntax:
qiime demux summarize
--i-data song-single-end-demux.qza
--o-visualization song-single-end-demux.qzv

I have the following error:
Plugin error from demux:
Compressed file ended before the end-of-stream marker was reached
Debug info has been saved to /tmp/qiime2-q2cli-err-drk_ev69.log

I have tried looking at previous forum posts and I was wondering if there is code I can run to test the integrity of the fastq files? Any other suggestions?

[colinbrislawn (Colin J Brislawn)]

Hello @kaitlineblen27,

Welcome to the forums!

@kaitlineblen27

Compressed file ended before the end-of-stream marker was reached
...
I have tried looking at previous forum posts and I was wondering if there is code I can run to test the integrity of the fastq files? Any other suggestions?

Yes, that 'end-of-stream' error is from an incomplete fastq file. I would just download those fastq files again and see if downloads them in full this time.

If you keep getting the same error, it's possible Song and friends uploaded an incomplete fastq file, which is a bigger problem.

But hopefully a new download will fix the issue.

[kaitlineblen27]

Thanks for your reply! I have already tried redownloading and unzipping the files again. Is there anything I can do to weed out the incomplete/corrupted fastq file(s)?

[colinbrislawn (Colin J Brislawn)]

Yes, there are ways to repair a, uh... unexpected fastq file.

Check our repair.sh, which is part of the BBTools package.

Read those docs, install BBTools with conda install bbmap, and see what happens.

(We aren't experts in BBTools over here, but hopefully we can help you get your data into Qiime2.)