Some background: I’m using a publicly available dataset that I’ve converted into a manifest over R by combining metadata and Fastq files. However, since my manifest contains over 3000 samples, I’ve divided the manifest into 6 smaller manifests (sub-manifests) as to facilitate the process of executing commands on the Cluster I’m using.
In the process of running a summarize step to look at base quality and per-sample read depth:
where “N” is the number of the smaller/sub-manifest. 4 of these manifests had absolutely no problem; 2 of them, however, report an “end of stream marker error” (debug info saved to /tmp/qiime2-q2cli-err-xa0hj1j4.log).
I tried looking into this error and comparing it to similar errors on this forum, but what confuses me is how only 2 of my sub-manifests are producing errors.
Hello @Benji. Can you post the full contents of that .log file here? If you no longer have access to that file then can you please rerun the command with the --verbose flag and paste the results here. Thank you.
Traceback (most recent call last):
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/q2cli/commands.py”, line 328, in call
results = action(**arguments)
File “</n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/decorator.py:decorator-gen-436>”, line 2, in summarize
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 245, in bound_callable
output_types, provenance)
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 452, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_demux/_summarize/_visualizer.py”, line 159, in summarize
quality_scores, min_seq_len = _subsample_paired(sample_map)
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_demux/_summarize/_visualizer.py”, line 61, in _subsample_paired
for i, (fseq, rseq) in enumerate(file_pair):
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/site-packages/q2_demux/_demux.py”, line 36, in _read_fastq_seqs
for seq_header, seq, qual_header, qual in itertools.zip_longest(*[fh] * 4):
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/gzip.py”, line 289, in read1
return self._buffer.read1(size)
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/_compression.py”, line 68, in readinto
data = self.read(len(byte_view))
File “/n/home02/bho/.conda/envs/qiime2-2020.2/lib/python3.6/gzip.py”, line 482, in read
raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
Hello @Benji. That makes it sound to me as though something is wrong with sequences_N.qza. The part I’m focusing on is “Compressed file ended before the end-of-stream marker was reached.” Files usually have a piece of data explicitly marking where their end should be, but for some reason your artifact is missing that marker suggesting that some portion of the end of the file has been cut off by something. These things happen from time to time most frequently when you’re downloading something and the download goes wrong. If you downloaded sequences_N.qza I would suggest re downloading it if possible. If you generated it yourself locally then this error is a little more peculiar, but all the same I would suggest re generating it if possible.