No such file or directory : error at demux summarize for demuxed/imported data

Hi,

We have already demultiplexed 250bp pairedend samples of 124 all sequenced on one lane. I used the following command to import it ( not sure about source-format since couldn’t find the detailed explanation of all importable-types)

qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path fastq
–source-format CasavaOneEightLanelessPerSampleDirFmt
–output-path demux-paired-end.qza

This runs with no error and then when I want to continue with demux summarize:

Summarize demuxed data

qiime demux summarize
–i-data demux-paired-end.qza
–o-visualization demux-paired-end.qzv

I get the following error:


Plugin error from demux:

[Errno 2] No such file or directory: ‘/var/folders/wx/rk1376mx19j5w4l4czbdzs11lq284c/T/qiime2-archive-ydpb09_c/f92fbb59-3a31-4f66-9285-a70764f4b722/data/MJneg-control_S121_L001_R1_001.fastq.gz’

Debug info has been saved to /var/folders/wx/rk1376mx19j5w4l4czbdzs11lq284c/T/qiime2-q2cli-err-5sncf3lo.log


I was able to export the data and it seams like all the files are there.the MANIFEST file content starts with:

sample-id,filename,direction
MJneg-control_S121,MJneg-control_S121_L001_R1_001.fastq.gz,forward
MJneg-control_S121,MJneg-control_S121_L001_R2_001.fastq.gz,reverse
MJpos-control_S122,MJpos-control_S122_L001_R1_001.fastq.gz,forward
MJpos-control_S122,MJpos-control_S122_L001_R2_001.fastq.gz,reverse


I also tried to run dada2:

Denoise the data. Looking at the quality plots , half of the samples do not need trimming, but some need around ranging bw 350-450.

250 bp Paired end reads

qiime dada2 denoise-single
–i-demultiplexed-seqs demux-paired-end.qza
–p-trim-left 0
–p-trunc-len 450
–o-representative-sequences rep-seqs-dada2.qza
–o-table table-dada2.qza

throws the following error:


Plugin error from dada2:

A path specified in the manifest does not exist or is not accessible: /var/folders/wx/rk1376mx19j5w4l4czbdzs11lq284c/T/qiime2-archive-njbviyfg/f92fbb59-3a31-4f66-9285-a70764f4b722/data/MJneg-control_S121_L001_R1_001.fastq.gz

Debug info has been saved to /var/folders/wx/rk1376mx19j5w4l4czbdzs11lq284c/T/qiime2-q2cli-err-fbzuql0u.log


So it’sa similar issue of not being able to locate fastq files in the artifact. Is this some type of a bug or what am I doing wrong. Please HELP…

Thanks

Hi @selen! I think something went awry during the import step, causing downstream analyses to fail. Can you please send me the following info?

  1. The output from running qiime info
  2. The output from running ls fastq/ (assuming that’s the directory name containing the fastq files you’re wanting to import).
  3. The fastq manifest file you’re using (please attach the entire file if possible).

Thanks!

  1. The output from running qiime info
    (qiime2-2017.12) Fields-MacBook-Pro:scripts yilm03$ qiime info
    System versions
    Python version: 3.5.4
    QIIME 2 release: 2017.12
    QIIME 2 version: 2017.12.1
    q2cli version: 2017.12.0

Installed plugins
alignment 2017.12.0
composition 2017.12.0
cutadapt 2017.12.0
dada2 2017.12.1
deblur 2017.12.0
demux 2017.12.0
diversity 2017.12.0
emperor 2017.12.0
feature-classifier 2017.12.0
feature-table 2017.12.0
gneiss 2017.12.0
longitudinal 2017.12.0
metadata 2017.12.0
phylogeny 2017.12.0
quality-control 2017.12.0
quality-filter 2017.12.0
sample-classifier 2017.12.0
taxa 2017.12.0
types 2017.12.0
vsearch 2017.12.0

Application config directory
/Users/yilm03/Library/Application Support/q2cli

Getting help
To get help with QIIME 2, visit https://qiime2.org

Citing QIIME 2
If you use QIIME 2 in any published work, you should cite QIIME 2 and the plugins that you used. To display the citations for QIIME 2 and all installed plugins, run:

qiime info --citations

MJneg-control_S121_L001_R1_001.fastq.gz
MJneg-control_S121_L001_R2_001.fastq.gz
MJpos-control_S122_L001_R1_001.fastq.gz
MJpos-control_S122_L001_R2_001.fastq.gz
NM-15-121_S1_L001_R1_001.fastq.gz
NM-15-121_S1_L001_R2_001.fastq.gz
NM-15-121_S31_L001_R1_001.fastq.gz
NM-15-121_S31_L001_R2_001.fastq.gz
NM-15-121_S61_L001_R1_001.fastq.gz
NM-15-121_S61_L001_R2_001.fastq.gz
NM-15-121_S91_L001_R1_001.fastq.gz
NM-15-121_S91_L001_R2_001.fastq.gz
NM-15-124_S2_L001_R1_001.fastq.gz
NM-15-124_S2_L001_R2_001.fastq.gz
NM-15-124_S32_L001_R1_001.fastq.gz
NM-15-124_S32_L001_R2_001.fastq.gz
NM-15-124_S62_L001_R1_001.fastq.gz
NM-15-124_S62_L001_R2_001.fastq.gz
NM-15-124_S92_L001_R1_001.fastq.gz
NM-15-124_S92_L001_R2_001.fastq.gz
NM-15-128_S101_L001_R1_001.fastq.gz
NM-15-128_S101_L001_R2_001.fastq.gz
NM-15-128_S11_L001_R1_001.fastq.gz
NM-15-128_S11_L001_R2_001.fastq.gz
NM-15-128_S41_L001_R1_001.fastq.gz
NM-15-128_S41_L001_R2_001.fastq.gz
NM-15-128_S71_L001_R1_001.fastq.gz
NM-15-128_S71_L001_R2_001.fastq.gz
NM-15-134_S111_L001_R1_001.fastq.gz
NM-15-134_S111_L001_R2_001.fastq.gz
NM-15-134_S21_L001_R1_001.fastq.gz
NM-15-134_S21_L001_R2_001.fastq.gz
NM-15-134_S51_L001_R1_001.fastq.gz
NM-15-134_S51_L001_R2_001.fastq.gz
NM-15-134_S81_L001_R1_001.fastq.gz
NM-15-134_S81_L001_R2_001.fastq.gz
NM-15-138_S33_L001_R1_001.fastq.gz
NM-15-138_S33_L001_R2_001.fastq.gz
NM-15-138_S3_L001_R1_001.fastq.gz
NM-15-138_S3_L001_R2_001.fastq.gz
NM-15-138_S63_L001_R1_001.fastq.gz
NM-15-138_S63_L001_R2_001.fastq.gz
NM-15-138_S93_L001_R1_001.fastq.gz
NM-15-138_S93_L001_R2_001.fastq.gz
NM-15-141_S112_L001_R1_001.fastq.gz
NM-15-141_S112_L001_R2_001.fastq.gz
NM-15-141_S22_L001_R1_001.fastq.gz
NM-15-141_S22_L001_R2_001.fastq.gz
NM-15-141_S52_L001_R1_001.fastq.gz
NM-15-141_S52_L001_R2_001.fastq.gz
NM-15-141_S82_L001_R1_001.fastq.gz
NM-15-141_S82_L001_R2_001.fastq.gz
NM-15-145_S102_L001_R1_001.fastq.gz
NM-15-145_S102_L001_R2_001.fastq.gz
NM-15-145_S12_L001_R1_001.fastq.gz
NM-15-145_S12_L001_R2_001.fastq.gz
NM-15-145_S42_L001_R1_001.fastq.gz
NM-15-145_S42_L001_R2_001.fastq.gz
NM-15-145_S72_L001_R1_001.fastq.gz
NM-15-145_S72_L001_R2_001.fastq.gz
NM-15-150_S34_L001_R1_001.fastq.gz
NM-15-150_S34_L001_R2_001.fastq.gz
NM-15-150_S4_L001_R1_001.fastq.gz
NM-15-150_S4_L001_R2_001.fastq.gz
NM-15-150_S64_L001_R1_001.fastq.gz
NM-15-150_S64_L001_R2_001.fastq.gz
NM-15-150_S94_L001_R1_001.fastq.gz
NM-15-150_S94_L001_R2_001.fastq.gz
NM-15-157_S113_L001_R1_001.fastq.gz
NM-15-157_S113_L001_R2_001.fastq.gz
NM-15-157_S23_L001_R1_001.fastq.gz
NM-15-157_S23_L001_R2_001.fastq.gz
NM-15-157_S53_L001_R1_001.fastq.gz
NM-15-157_S53_L001_R2_001.fastq.gz
NM-15-157_S83_L001_R1_001.fastq.gz
NM-15-157_S83_L001_R2_001.fastq.gz
NM-15-160_S103_L001_R1_001.fastq.gz
NM-15-160_S103_L001_R2_001.fastq.gz
NM-15-160_S13_L001_R1_001.fastq.gz
NM-15-160_S13_L001_R2_001.fastq.gz
NM-15-160_S43_L001_R1_001.fastq.gz
NM-15-160_S43_L001_R2_001.fastq.gz
NM-15-160_S73_L001_R1_001.fastq.gz
NM-15-160_S73_L001_R2_001.fastq.gz
NM-15-166_S114_L001_R1_001.fastq.gz
NM-15-166_S114_L001_R2_001.fastq.gz
NM-15-166_S24_L001_R1_001.fastq.gz
NM-15-166_S24_L001_R2_001.fastq.gz
NM-15-166_S54_L001_R1_001.fastq.gz
NM-15-166_S54_L001_R2_001.fastq.gz
NM-15-166_S84_L001_R1_001.fastq.gz
NM-15-166_S84_L001_R2_001.fastq.gz
NM-15-170_S115_L001_R1_001.fastq.gz
NM-15-170_S115_L001_R2_001.fastq.gz
NM-15-170_S25_L001_R1_001.fastq.gz
NM-15-170_S25_L001_R2_001.fastq.gz
NM-15-170_S55_L001_R1_001.fastq.gz
NM-15-170_S55_L001_R2_001.fastq.gz
NM-15-170_S85_L001_R1_001.fastq.gz
NM-15-170_S85_L001_R2_001.fastq.gz
NM-15-173_S116_L001_R1_001.fastq.gz
NM-15-173_S116_L001_R2_001.fastq.gz
NM-15-173_S26_L001_R1_001.fastq.gz
NM-15-173_S26_L001_R2_001.fastq.gz
NM-15-173_S56_L001_R1_001.fastq.gz
NM-15-173_S56_L001_R2_001.fastq.gz
NM-15-173_S86_L001_R1_001.fastq.gz
NM-15-173_S86_L001_R2_001.fastq.gz
NM-15-179_S104_L001_R1_001.fastq.gz
NM-15-179_S104_L001_R2_001.fastq.gz
NM-15-179_S14_L001_R1_001.fastq.gz
NM-15-179_S14_L001_R2_001.fastq.gz
NM-15-179_S44_L001_R1_001.fastq.gz
NM-15-179_S44_L001_R2_001.fastq.gz
NM-15-179_S74_L001_R1_001.fastq.gz
NM-15-179_S74_L001_R2_001.fastq.gz
NM-15-182_S35_L001_R1_001.fastq.gz
NM-15-182_S35_L001_R2_001.fastq.gz
NM-15-182_S5_L001_R1_001.fastq.gz
NM-15-182_S5_L001_R2_001.fastq.gz
NM-15-182_S65_L001_R1_001.fastq.gz
NM-15-182_S65_L001_R2_001.fastq.gz
NM-15-182_S95_L001_R1_001.fastq.gz
NM-15-182_S95_L001_R2_001.fastq.gz
NM-15-186_S117_L001_R1_001.fastq.gz
NM-15-186_S117_L001_R2_001.fastq.gz
NM-15-186_S27_L001_R1_001.fastq.gz
NM-15-186_S27_L001_R2_001.fastq.gz
NM-15-186_S57_L001_R1_001.fastq.gz
NM-15-186_S57_L001_R2_001.fastq.gz
NM-15-186_S87_L001_R1_001.fastq.gz
NM-15-186_S87_L001_R2_001.fastq.gz
NM-15-187_S36_L001_R1_001.fastq.gz
NM-15-187_S36_L001_R2_001.fastq.gz
NM-15-187_S66_L001_R1_001.fastq.gz
NM-15-187_S66_L001_R2_001.fastq.gz
NM-15-187_S6_L001_R1_001.fastq.gz
NM-15-187_S6_L001_R2_001.fastq.gz
NM-15-187_S96_L001_R1_001.fastq.gz
NM-15-187_S96_L001_R2_001.fastq.gz
NM-15-195_S105_L001_R1_001.fastq.gz
NM-15-195_S105_L001_R2_001.fastq.gz
NM-15-195_S15_L001_R1_001.fastq.gz
NM-15-195_S15_L001_R2_001.fastq.gz
NM-15-195_S45_L001_R1_001.fastq.gz
NM-15-195_S45_L001_R2_001.fastq.gz
NM-15-195_S75_L001_R1_001.fastq.gz
NM-15-195_S75_L001_R2_001.fastq.gz
NM-15-196_S106_L001_R1_001.fastq.gz
NM-15-196_S106_L001_R2_001.fastq.gz
NM-15-196_S16_L001_R1_001.fastq.gz
NM-15-196_S16_L001_R2_001.fastq.gz
NM-15-196_S46_L001_R1_001.fastq.gz
NM-15-196_S46_L001_R2_001.fastq.gz
NM-15-196_S76_L001_R1_001.fastq.gz
NM-15-196_S76_L001_R2_001.fastq.gz
NM-15-200_S118_L001_R1_001.fastq.gz
NM-15-200_S118_L001_R2_001.fastq.gz
NM-15-200_S28_L001_R1_001.fastq.gz
NM-15-200_S28_L001_R2_001.fastq.gz
NM-15-200_S58_L001_R1_001.fastq.gz
NM-15-200_S58_L001_R2_001.fastq.gz
NM-15-200_S88_L001_R1_001.fastq.gz
NM-15-200_S88_L001_R2_001.fastq.gz
NM-15-202_S119_L001_R1_001.fastq.gz
NM-15-202_S119_L001_R2_001.fastq.gz
NM-15-202_S29_L001_R1_001.fastq.gz
NM-15-202_S29_L001_R2_001.fastq.gz
NM-15-202_S59_L001_R1_001.fastq.gz
NM-15-202_S59_L001_R2_001.fastq.gz
NM-15-202_S89_L001_R1_001.fastq.gz
NM-15-202_S89_L001_R2_001.fastq.gz
NM-15-212_S37_L001_R1_001.fastq.gz
NM-15-212_S37_L001_R2_001.fastq.gz
NM-15-212_S67_L001_R1_001.fastq.gz
NM-15-212_S67_L001_R2_001.fastq.gz
NM-15-212_S7_L001_R1_001.fastq.gz
NM-15-212_S7_L001_R2_001.fastq.gz
NM-15-212_S97_L001_R1_001.fastq.gz
NM-15-212_S97_L001_R2_001.fastq.gz
NM-15-217_S107_L001_R1_001.fastq.gz
NM-15-217_S107_L001_R2_001.fastq.gz
NM-15-217_S17_L001_R1_001.fastq.gz
NM-15-217_S17_L001_R2_001.fastq.gz
NM-15-217_S47_L001_R1_001.fastq.gz
NM-15-217_S47_L001_R2_001.fastq.gz
NM-15-217_S77_L001_R1_001.fastq.gz
NM-15-217_S77_L001_R2_001.fastq.gz
NM-15-218_S108_L001_R1_001.fastq.gz
NM-15-218_S108_L001_R2_001.fastq.gz
NM-15-218_S18_L001_R1_001.fastq.gz
NM-15-218_S18_L001_R2_001.fastq.gz
NM-15-218_S48_L001_R1_001.fastq.gz
NM-15-218_S48_L001_R2_001.fastq.gz
NM-15-218_S78_L001_R1_001.fastq.gz
NM-15-218_S78_L001_R2_001.fastq.gz
NM-15-219_S120_L001_R1_001.fastq.gz
NM-15-219_S120_L001_R2_001.fastq.gz
NM-15-219_S30_L001_R1_001.fastq.gz
NM-15-219_S30_L001_R2_001.fastq.gz
NM-15-219_S60_L001_R1_001.fastq.gz
NM-15-219_S60_L001_R2_001.fastq.gz
NM-15-219_S90_L001_R1_001.fastq.gz
NM-15-219_S90_L001_R2_001.fastq.gz
NM-15-221_S109_L001_R1_001.fastq.gz
NM-15-221_S109_L001_R2_001.fastq.gz
NM-15-221_S19_L001_R1_001.fastq.gz
NM-15-221_S19_L001_R2_001.fastq.gz
NM-15-221_S49_L001_R1_001.fastq.gz
NM-15-221_S49_L001_R2_001.fastq.gz
NM-15-221_S79_L001_R1_001.fastq.gz
NM-15-221_S79_L001_R2_001.fastq.gz
NM-15-224_S38_L001_R1_001.fastq.gz
NM-15-224_S38_L001_R2_001.fastq.gz
NM-15-224_S68_L001_R1_001.fastq.gz
NM-15-224_S68_L001_R2_001.fastq.gz
NM-15-224_S8_L001_R1_001.fastq.gz
NM-15-224_S8_L001_R2_001.fastq.gz
NM-15-224_S98_L001_R1_001.fastq.gz
NM-15-224_S98_L001_R2_001.fastq.gz
NM-15-237_S39_L001_R1_001.fastq.gz
NM-15-237_S39_L001_R2_001.fastq.gz
NM-15-237_S69_L001_R1_001.fastq.gz
NM-15-237_S69_L001_R2_001.fastq.gz
NM-15-237_S99_L001_R1_001.fastq.gz
NM-15-237_S99_L001_R2_001.fastq.gz
NM-15-237_S9_L001_R1_001.fastq.gz
NM-15-237_S9_L001_R2_001.fastq.gz
NM-15-240_S100_L001_R1_001.fastq.gz
NM-15-240_S100_L001_R2_001.fastq.gz
NM-15-240_S10_L001_R1_001.fastq.gz
NM-15-240_S10_L001_R2_001.fastq.gz
NM-15-240_S40_L001_R1_001.fastq.gz
NM-15-240_S40_L001_R2_001.fastq.gz
NM-15-240_S70_L001_R1_001.fastq.gz
NM-15-240_S70_L001_R2_001.fastq.gz
NM-15-242_S110_L001_R1_001.fastq.gz
NM-15-242_S110_L001_R2_001.fastq.gz
NM-15-242_S20_L001_R1_001.fastq.gz
NM-15-242_S20_L001_R2_001.fastq.gz
NM-15-242_S50_L001_R1_001.fastq.gz
NM-15-242_S50_L001_R2_001.fastq.gz
NM-15-242_S80_L001_R1_001.fastq.gz
NM-15-242_S80_L001_R2_001.fastq.gz

I didn't provide any manifest file, but here is part of the MANIFEST file I see when I export back the artifact.

sample-id,filename,direction
MJneg-control_S121,MJneg-control_S121_L001_R1_001.fastq.gz,forward
MJneg-control_S121,MJneg-control_S121_L001_R2_001.fastq.gz,reverse
MJpos-control_S122,MJpos-control_S122_L001_R1_001.fastq.gz,forward
MJpos-control_S122,MJpos-control_S122_L001_R2_001.fastq.gz,reverse
NM-15-121_S1,NM-15-121_S1_L001_R1_001.fastq.gz,forward
NM-15-121_S1,NM-15-121_S1_L001_R2_001.fastq.gz,reverse
NM-15-121_S31,NM-15-121_S31_L001_R1_001.fastq.gz,forward
NM-15-121_S31,NM-15-121_S31_L001_R2_001.fastq.gz,reverse
....................

Thanks for the details @selen! Based on the filenames in the fastq/ directory, you should be able to use the CasavaOneEightSingleLanePerSampleDirFmt to import your data:

qiime tools import \
--type 'SampleData[PairedEndSequencesWithQuality]' \
--input-path fastq \
--source-format CasavaOneEightSingleLanePerSampleDirFmt \
--output-path demux-paired-end.qza

That was my first attempt. Unfortunately I got the following error…Maybe naming convention problem?

There was a problem importing /Users/yilm03/Work/Datasets/Michael_Bailey/Microbiome_16S_mouse/fastq/nomouse:

/Users/yilm03/Work/Datasets/Michael_Bailey/Microbiome_16S_mouse/fastq/nomouse is not a(n) CasavaOneEightSingleLanePerSampleDirFmt:

Duplicate samples in forward reads: {‘NM-15-195’, ‘NM-15-179’, ‘NM-15-141’, ‘NM-15-124’, ‘NM-15-166’, ‘NM-15-212’, ‘NM-15-145’, ‘NM-15-196’, ‘NM-15-160’, ‘NM-15-242’, ‘NM-15-221’, ‘NM-15-170’, ‘NM-15-200’, ‘NM-15-150’, ‘NM-15-240’, ‘NM-15-134’, ‘NM-15-187’, ‘NM-15-217’, ‘NM-15-128’, ‘NM-15-182’, ‘NM-15-224’, ‘NM-15-138’, ‘NM-15-237’, ‘NM-15-219’, ‘NM-15-218’, ‘NM-15-157’, ‘NM-15-121’, ‘NM-15-173’, ‘NM-15-202’, ‘NM-15-186’}

It looks like some of your samples have more than one pair of forward/reverse reads files associated with them. For example, the NM-15-195 sample has the following files associated with it:

NM-15-195_S105_L001_R1_001.fastq.gz
NM-15-195_S105_L001_R2_001.fastq.gz
NM-15-195_S15_L001_R1_001.fastq.gz
NM-15-195_S15_L001_R2_001.fastq.gz
NM-15-195_S45_L001_R1_001.fastq.gz
NM-15-195_S45_L001_R2_001.fastq.gz
NM-15-195_S75_L001_R1_001.fastq.gz
NM-15-195_S75_L001_R2_001.fastq.gz

QIIME 2 expects demultiplexed paired-end data to have exactly one forward reads file and one reverse reads file per sample. The sample above has 4 pairs of forward/reverse reads files associated with it. Are each of these forward/reverse files from different sequencing runs? Was the same sample sequenced multiple times, e.g. to produce technical replicates?

They are four different samples that represent the different condition/timepoints. All 124 samples are sequenced on one lane though.

1 Like

ok I just renamed the samples to include postfixes of -1,-2,-3 and -4… Seems like it worked… I will try to do downstream…

2 Likes

Treating them as independent samples sounds like the way to go! If you have any additional issues, we’re here to help (please create new forum topics for any further questions that come up). Thanks!

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.