Have you reviewed the [QIIME 2 Forum Glossary]?
I have for 2 days searched everything Qiime2 tutoritals, glossary, and forums. I've even done googling for third-party resources.
Version of QIIME 2 you are running, and how it is installed (e.g. Virtualbox, conda, etc.)
q2cli version 2019.4.0
System versions
Python version: 3.6.7
QIIME 2 release: 2019.4
QIIME 2 version: 2019.4.0
q2cli version: 2019.4.0
Installed plugins
alignment: 2019.4.0
composition: 2019.4.0
cutadapt: 2019.4.0
dada2: 2019.4.0
deblur: 2019.4.0
demux: 2019.4.1
diversity: 2019.4.0
emperor: 2019.4.0
feature-classifier: 2019.4.0
feature-table: 2019.4.0
fragment-insertion: 2019.4.0
gneiss: 2019.4.0
longitudinal: 2019.4.0
metadata: 2019.4.0
phylogeny: 2019.4.0
quality-control: 2019.4.0
quality-filter: 2019.4.0
sample-classifier: 2019.4.0
taxa: 2019.4.0
types: 2019.4.1
vsearch: 2019.4.0
What is the exact command or commands you ran? import-failure.txt (41.7 KB)
What is the exact error message?
I've uploaded a text file that is a copy and past of my full terminal window.
I'm new to any type of sequence analysis and so I started with learning how to run a qiime1 pipeline with my PI who is experience with the original version. I wanted to make sure I could understand what the commands were doing. Now I'm trying to figure out qiime2 to future proof my PI's work but I'm having the worst time. The tutorial pages are helpful only why my case is exactly the same and right now mine is not. I did the moving pictures tutorial and that made sense and I was able to do that but now want to try this on my own data that I've analyzed with qiime1. I'm stuck at step 1. importing my data. This is amplicon data with forward and reverse reads in separate files (demultiplexed) and we only want to use the forward. I have tried every iteration of manifest file that has been suggested on nearly 2 dozen sources with no luck. sample_manifest.txt (2.7 KB)
I've tried text, comma separated, tab separated, with file extensions included in the path, without file extensions,etc. I'm not sure what I'm doing wrong for the import. I'm not sure if I'm selecting the proper --type and --format-type or if it is something about my manifest file.
Thanks for your help
@Dana_Roberts,
Sorry to hear you are having a rough time importing your data! And thanks for reading through the forum/googling prior to posting your problem. Here is the error info we need from the file you uploaded:
(qiime2-2019.4) My-Mac:endosymbiont_2019_pipeline danaroberts$ qiime tools import --type 'SampleData[SequencesWithQuality]' --input-path raw_reads/ --input-format FastqGzFormat --output-path qiime2-raw-reads.qza
Traceback (most recent call last):
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/builtin/tools.py", line 152, in import_data
view_type=input_format)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py", line 240, in import_data
return cls._from_view(type_, view, view_type, provenance_capture)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py", line 264, in _from_view
recorder=recorder)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/core/transform.py", line 59, in make_transformation
(self._view_type, other._view_type))
Exception: No transformation from <class 'q2_types.per_sample_sequences._format.FastqGzFormat'> to <class 'q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt'>
An unexpected error has occurred:
No transformation from <class 'q2_types.per_sample_sequences._format.FastqGzFormat'> to <class 'q2_types.per_sample_sequences._format.SingleLanePerSampleSingleEndFastqDirFmt'>
See above for debug info.
So it looks like you are following the correct tutorial for importing a single-end manifest format but your command strays from the tutorial — the error is essentially saying that you are specifying an input-format that is incompatible with your desired output format. Let’s start there. Does this command work for you?
for the manifest file I do have all three headings (sample-id, absolute-path, direction) and I’ve tried typing all that information in by hand in the text editor and making is simple text. I’ve tried tabs, spaces, commas with spaces, commas without spaces. I’ve tried typing it in excel and saving it as text from excel, saving it as a .csv, saving it as a .tsv and none of those have worked.
Yep my bad I should have read more, it was a lot to ingest!
Here is where you input the correct command and got a meaningful error that we can troubleshoot:
qiime tools import \
--type 'SampleData[SequencesWithQuality]' \
--input-path sample_manifest.txt \
--output-path qiime2-raw-reads.qza \
--input-format SingleEndFastqManifestPhred33V2
There was a problem importing sample_manifest.txt:
sample_manifest.txt is not a(n) SingleEndFastqManifestPhred33V2 file:
Filepath on line 1 and column "absolute-filepath" could not be found (/Users/danaroberts/Desktop/endosymbiont_2019_pipeline/raw_reads/G423-035-T4-STR_S35_L001_R1_001.fastq) for sample "T4_STR_S35".
check for typos in your filepaths. What happens if you enter this command?
ls /Users/danaroberts/Desktop/endosymbiont_2019_pipeline/raw_reads/G423-035-T4-STR_S35_L001_R1_001.fastq
(qiime2-2019.4) My-Mac:endosymbiont_2019_pipeline danaroberts$ ls /Users/danaroberts/Desktop/endosymbiont_2019_pipeline/raw_reads/G423-035-T4-STR_S35_L001_R1_001.fastq
ls: /Users/danaroberts/Desktop/endosymbiont_2019_pipeline/raw_reads/G423-035-T4-STR_S35_L001_R1_001.fastq: No such file or directory
looking at my files I see I have mislabeled the proper file directory for those specific files. I will alter the file directory now and see if that fixes my import problems.
I have now fixed my absolute-filepath in the manifest file to be correct - I must have mixed it up trying all the times before.
I re-ran the code you pointed out in your last response (copied here)
There was a problem importing sample_manifest.txt:
sample_manifest.txt is not a(n) SingleEndFastqManifestPhred33V2 file:
Filepath on line 1 and column "absolute-filepath" could not be found (/Users/danaroberts/Desktop/endosymbiont_2019_pipeline/raw_reads/G423-035-T4-STR_S35_L001_R1_001.fastq) for sample "T4_STR_S35".
I got a new error! (I'm so good at getting errors lol)
(qiime2-2019.4) My-Mac:endosymbiont_2019_pipeline danaroberts$ qiime tools import --type 'SampleData[SequncesWithQuality]' --input-path sample_manifest.txt --output-path qiime2-raw-reads.qza --input-format SingleEndFastqManifestPhred33V2
Traceback (most recent call last):
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/q2cli/builtin/tools.py", line 152, in import_data
view_type=input_format)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py", line 240, in import_data
return cls.from_view(type, view, view_type, provenance_capture)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/result.py", line 253, in _from_view
output_dir_fmt = pm.get_directory_format(type)
File "/miniconda3/envs/qiime2-2019.4/lib/python3.6/site-packages/qiime2/sdk/plugin_manager.py", line 157, in get_directory_format
% semantic_type)
TypeError: Semantic type SampleData[SequncesWithQuality] does not have a compatible directory format.
An unexpected error has occurred:
Semantic type SampleData[SequncesWithQuality] does not have a compatible directory format.
SequncesWithQuality should be SequencesWithQuality
No worries about the typos! We are getting there! (Typos are arguably better than the alternative — copying/pasting from a webpage and propagating errors/introducing invisible characters that display on the html!)
There was a problem importing sample_manifest.txt:
sample_manifest.txt is not a(n) SingleEndFastqManifestPhred33V2 file:
Metadata file must be encoded as UTF-8 or ASCII. The following error occurred when decoding the file:
’utf-8’ codec can’t decode byte 0xd0 in position 924: invalid continuation byte
There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org
your manifest file contains an invalid character (and possibly more).
what program did you use to make this manifest? You should try to save your file as utf-8 format... see this topic for details on troubleshooting this problem:
It looks like the dashes in these lines are not really dashes...
G423–041-CN-STR_S41_L001_R1_001.fastq
G423—042-T4-BLB_S42_L001_R1_001.fastq
I was using TextEdit - I have a mac. I copied and pasted my manifest information into word and I was finally able to see the dashes you were talking about (they weren’t obviously visible in the simple text editor). I fixed the dashes and then choose save as and selected plain text .txt. I was then prompted how to format the file and I selected the other option and choose utf-8 from the list. I ran the code again and it was finally a success!
Glad you finally got it working! Don't be afraid to come to us sooner in the future if you have problems (though of course always check the forum history just to make sure it's not an issue that's been solved).
Yes... unfortunately many folks do not share the solution once they work it out. So thank you for describing your steps!