Hello, so I am new to qiime2 and I have the qiime2-2019.7 on Linux platform. At the moment, I have of forward and reverse fastq files for my samples in two batches. Though I have gone through the moving pictures tutorial, I still have problem importing my fastq files and the metadata.
P.S: I already used fastq processor to process my data to get barcodes, forward and reverse reads.
Thanks
Hi @slamcole,
Welcome to the forum!
Can you clarify for us what format your fastq files are and also what you have tried exactly to this point? Have you gone through the importing tutorial?
Do you have one single forward, reverse, and barcode file for all of your samples or one of each per sample?
Hi @Mehrbod_Estaki, initially assumed my fastq files were in two groups that was why I tried to merge them. but on closer look, what I have with are single forward, reverse and barcodes fastq.gz files and the sample-metadata.tsv which I got after running my raw fastq files through the FASTQ processor.
What I am trying to do right now is to import what I have as paired-end sequences.qza using the EMP protocol. But I kept getting this error
“There was a problem importing emp-paired-end-sequences: Missing one or more files for EMPPairedENDDirFmt: 'forward.fastq.gz”
Hi @slamcole,
Can you please provide us with the exact commands (copy & paste) you are using and the qiime2 version?
My first reaction is that the folder in which you are referring to in your command does not contain the filename forward.fastq.qz as it is expecting. The tutorial I linked should have a direct example of importing using this approach too.
Hi, so i was successful in making the emp-paired-end-sequences.qza. The challenge I have right now is semultiplexing. I am using the Atacama soil soil microbiome tutorial but I give getting the error plug in error from demux:
source activate (qiime2-2018.8) qiime2@qiime2core2018-8:~ source activate qiime2-2019.7
mkdir (qiime2-2019.7) qiime2@qiime2core2018-8:~ mkdir microbiome-data-analysis
(qiime2-2019.7) qiime2@qiime2core2018-8:~ mkdir paired-end-sequences
(qiime2-2019.7) qiime2@qiime2core2018-8:~ cd microbiome-data-analysis
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ mkdir paired-end-sequences
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime tools import \
--type EMPPairedEndSequences
--input-path emp-paired-end-sequences
--output-path emp-paired-end-sequences.qza
Usage: qiime tools import [OPTIONS]
Import data to create a new QIIME 2 Artifact. See https://docs.qiime2.org/
for usage examples and details on the file types and associated semantic
types that can be imported.
Options:
--type TEXT The semantic type of the artifact that will be
created upon importing. Use --show-importable-types
to see what importable semantic types are available
in the current deployment. [required]
--input-path PATH Path to file or directory that should be imported.
[required]
--output-path ARTIFACT Path where output artifact should be written.
[required]
--input-format TEXT The format of the data to be imported. If not
provided, data must be in the format expected by the
semantic type provided via --type.
--show-importable-types Show the semantic types that can be supplied to
--type to import data into an artifact.
--show-importable-formats
Show formats that can be supplied to --input-format
to import data into an artifact.
--help Show this message and exit.
There was a problem with the command:
(1/1) Invalid value for "--input-path": Path "emp-paired-end-sequences" does
not exist.
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ mkdir emp-paired-end-sequences
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime tools import --type EMPPairedEndSequences --input-path emp-paired-end-sequences --output-path emp-paired-end-sequences.qza
Imported emp-paired-end-sequences as EMPPairedEndDirFmt to emp-paired-end-sequences.qza
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired \
--i-seqs emp-paired-end-sequences.qza
--m-barcodes-file sample-metadata tsv
--m-barcodes-column BarcoseSequence
--p-no-golay-error-correction
--p-no-rev-comp-mapping-barcodes
--o-per-sample-sequences demux.qza
--o-error-correction-details demux-details.qza
There was an issue with loading the file sample-metadata as metadata:
Metadata file path doesn't exist, or the path points to something other than a file. Please check that the path exists, has read permissions, and points to a regular file (not a directory): sample-metadata
There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org
Find details on QIIME 2 metadata requirements here: Metadata in QIIME 2 — QIIME 2 2019.7.0 documentation
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired --i-seqs emp-paired-end-sequences.qza --m-barcodes-file sample-metadata tsv --m-barcodes-column BarcoseSequence --p-no-golay-error-correction --p-no-rev-comp-mapping-barcodes --o-per-sample-sequences demux.qza --o-error-correction-details demux-details.qza
There was an issue with loading the file sample-metadata as metadata:
Metadata file path doesn't exist, or the path points to something other than a file. Please check that the path exists, has read permissions, and points to a regular file (not a directory): sample-metadata
There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org
Find details on QIIME 2 metadata requirements here: Metadata in QIIME 2 — QIIME 2 2019.7.0 documentation
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired --i-seqs emp-paired-end-sequences.qza --m-barcodes-file sample-metadata.tsv --m-barcodes-column BarcodeSequence --p-no-golay-error-correction --p-no-rev-comp-mapping-barcodes --o-per-sample-sequences demux.qza --o-error-correction-details demux-details.qza
^C
Aborted!
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired \
--m-barcodes-file sample-metadata.tsv
--m-barcodes-column barcode-sequence
--p-rev-comp-mapping-barcodes
--i-seqs emp-paired-end-sequences.qza
--o-per-sample-sequences demux.qza
--o-error-correction-details demux-details.qza^C
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired --m-barcodes-file sample-metadata.tsv --m-barcodes-column barcode-sequence --p-no-rev-comp-mapping-barcodes --p-no-golay-error-correction --i-seqs emp-paired-end-sequences.qza --o-per-sample-sequences demux.qza
^C
Aborted!
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired --m-barcodes-file sample-metadata.tsv --m-barcodes-column BarcodeSequence --p-no-rev-comp-mapping-barcodes --p-no-golay-error-correction --i-seqs emp-paired-end-sequences.qza --o-per-sample-sequences demux.qza
Usage: qiime demux emp-paired [OPTIONS]
Demultiplex paired-end sequence data (i.e., map barcode reads to sample
ids) for data generated with the Earth Microbiome Project (EMP) amplicon
sequencing protocol. Details about this protocol can be found at
Protocols and Standards : earthmicrobiome
Inputs:
--i-seqs ARTIFACT EMPPairedEndSequences
The paired-end sequences to be demultiplexed.
[required]
Parameters:
--m-barcodes-file METADATA
--m-barcodes-column COLUMN MetadataColumn[Categorical]
The sample metadata column containing the per-sample
barcodes. [required]
--p-golay-error-correction / --p-no-golay-error-correction
Perform 12nt Golay error correction on the barcode
reads. [default: True]
--p-rev-comp-barcodes / --p-no-rev-comp-barcodes
If provided, the barcode sequence reads will be
reverse complemented prior to demultiplexing.
[default: False]
--p-rev-comp-mapping-barcodes / --p-no-rev-comp-mapping-barcodes
If provided, the barcode sequences in the sample
metadata will be reverse complemented prior to
demultiplexing. [default: False]
Outputs:
--o-per-sample-sequences ARTIFACT
SampleData[PairedEndSequencesWithQuality]
The resulting demultiplexed sequences. [required]
--o-error-correction-details ARTIFACT ErrorCorrectionDetails
Detail about the barcode error corrections. [required]
Miscellaneous:
--output-dir PATH Output unspecified results to a directory
--verbose / --quiet Display verbose output to stdout and/or stderr during
execution of this action. Or silence output if
execution is successful (silence is golden).
--citations Show citations and exit.
--help Show this message and exit.
There was a problem with the command:
(1/1) Missing option "--o-error-correction-details". ("--output-dir" may
also be used)
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$ qiime demux emp-paired --m-barcodes-file sample-metadata.tsv --m-barcodes-column BarcodeSequence --p-no-rev-comp-mapping-barcodes --p-no-golay-error-correction --i-seqs emp-paired-end-sequences.qza --o-per-sample-sequences demux.qza --o-error-correction-details demux-details.qza
Plugin error from demux:
Debug info has been saved to /tmp/qiime2-q2cli-err-pbds1134.log
^[[A(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$
(qiime2-2019.7) qiime2@qiime2core2018-8:~/microbiome-data-analysis$
The qiime2 version is 2019.7
Hi @slamcole,
That's good! The following should probably be moved to a separate thread since this is a new issue. But before we move it, let's make sure the issue is indeed separate. Can you please show us the content of the folder in which you are running the last command from, type ls there. Also please rerun the very last command that is giving you the error but add the --verbose parameter to it so we get the full error message.
(qiime2-2019.7) qiime2@qiime2core2018-8:~/8-13-2019-analysis$ ls
emp-paired-end-sequences emp-paired-end-sequences.qza sample-metadata.tsv
(qiime2-2019.7) qiime2@qiime2core2018-8:~/8-13-2019-analysis$ qiime demux emp-paired --m-barcodes-file sample-metadata.tsv --m-barcodes-column BarcodeSequence --p-no-golay-error-correction --p-no-rev-comp-barcodes --i-seqs emp-paired-end-sequences.qza --o-per-sample-sequences demux.qza --o-error-correction-details demux-details.qza --verbose
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-437>”, line 2, in emp_paired
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 411, in callable_executor
spec.qiime_type, output_view, spec.view_type, prov)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/result.py”, line 267, in _from_view
result = transformation(view, validate_level)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/transform.py”, line 70, in transformation
new_view = transformer(view)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/core/transform.py”, line 220, in wrapped
file_view = transformer(view)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_demux/_transformer.py”, line 123, in _8
Metadata(data).save(str(ff))
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 364, in init
self._dataframe, self._columns = self._normalize_dataframe(dataframe)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 372, in _normalize_dataframe
metadata_column = self._metadata_column_factory(series)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 384, in _metadata_column_factory
column = CategoricalMetadataColumn(series)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 871, in init
super().init(series.index)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 99, in init
self._validate_index(index, axis=‘id’)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/metadata/metadata.py”, line 175, in _validate_index
if len(index) != len(set(index)):
MemoryError
Plugin error from demux:
See above for debug info.
Hi @slamcole,
Thank you for that. The last error line here would suggest that there is insufficient memory to carry on this command. Even though demux plugin isn’t too memory demanding it’s possible you simply have limited memory allocated to your environment. Can you tell us how much memory and storage you have allocated to your environment that runs qiime2? If this is ran through a personal computer using a Virtual Machine you can change the default settings to allocate more juice. If this is through a remote network you may want to contact the admins for further assistance.
Thank you very much. A colleague of mine looked into it and suggested the same thing. The memory size on my PC is just 118GB and I am sure it is inadequate to run the virtual box. Do you think getting an external hard-drive to attach to the PC will solve the problem.
Thanks
Hi @slamcole,
I’m guessing you mean that the storage on your computer is 118 GB, which is different than the memory (quantified with RAM) which we think is the issue here. If you indeed have 118 GB RAM on your machine then that should be more than enough! But I’m guessing you don’t.
I would recommend recruiting a more powerful machine for this work, or if that is not an option try changing your VirtualBox machine settings so that more memory and storage are being allocated to it, I believe by default VB dedicates only 1 GB of your available RAMs which is not enough.
Good luck and keep us posted.
Thank you.
(qiime2-2019.7) qiime2@qiime2core2018-11:~/8-16-19-afternoon-analysis$ qiime dada2 denoise-paired --i-demultiplexed-seqs demux.qza --p-trim-left-f 20 --p-trim-left-r 20 --p-trunc-len-f 220 --p-trunc-len-r 220 --o-table table.qza --o-representative-sequences rep-seqs.qza --o-denoising-stats denoising-stats.qza --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.
Command: run_dada_paired.R /tmp/tmpshs41zjr/forward /tmp/tmpshs41zjr/reverse /tmp/tmpshs41zjr/output.tsv.biom /tmp/tmpshs41zjr/track.tsv /tmp/tmpshs41zjr/filt_f /tmp/tmpshs41zjr/filt_r 220 220 20 20 2.0 2.0 2 consensus 1.0 1 1000000
R version 3.5.1 (2018-07-02)
Loading required package: Rcpp
Error: package or namespace load failed for ‘dada2’ in loadNamespace(i, c(lib.loc, .libPaths()), versionCheck = vI[[i]]):
there is no package called ‘GenomeInfoDbData’
Execution halted
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py”, line 234, in denoise_paired
run_commands([cmd])
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py”, line 36, in run_commands
subprocess.run(cmd, check=True)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/subprocess.py”, line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘run_dada_paired.R’, ‘/tmp/tmpshs41zjr/forward’, ‘/tmp/tmpshs41zjr/reverse’, ‘/tmp/tmpshs41zjr/output.tsv.biom’, ‘/tmp/tmpshs41zjr/track.tsv’, ‘/tmp/tmpshs41zjr/filt_f’, ‘/tmp/tmpshs41zjr/filt_r’, ‘220’, ‘220’, ‘20’, ‘20’, ‘2.0’, ‘2.0’, ‘2’, ‘consensus’, ‘1.0’, ‘1’, ‘1000000’]’ returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2cli/commands.py”, line 327, in call
results = action(**arguments)
File “</home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/decorator.py:decorator-gen-459>”, line 2, in denoise_paired
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 240, in bound_callable
output_types, provenance)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/qiime2/sdk/action.py”, line 383, in callable_executor
output_views = self._callable(**view_args)
File “/home/qiime2/miniconda/envs/qiime2-2019.7/lib/python3.6/site-packages/q2_dada2/_denoise.py”, line 249, in denoise_paired
" and stderr to learn more." % e.returncode)
Exception: An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
Plugin error from dada2:
An error was encountered while running DADA2 in R (return code 1), please inspect stdout and stderr to learn more.
See above for debug info.
This came up while running the denoising dada2 command
Hi @slamcole!
Check out this part of the error message:
Looks like something isn't right with your environment. It could be that it is "incomplete" (missing dependencies), or, your environment variables could be "polluting" the environment with .
One thing that jumps out at me is this:
It looks like you have manually installed 2019.7 into a 2018.11 virtual machine. I suspect something has gone wrong with that. Can you run the command env and paste the results here? Thanks! 
Oops, very sorry @slamcole, I didn’t mean to abandon this thread. I totally forgot to come back to it or assign myself to check in on later. Just slipped through the crack it would seem. Thanks @thermokarst for taking over it!