Qiime2 out put error - protocol "QIIME 2 Enables Comprehensive End‐to‐End Analysis of Diverse Microbiome Data and Comparative Studies with Publicly Available Data"

Hello,

I am following the protocol of " QIIME 2 Enables Comprehensive End‐to‐End Analysis of Diverse Microbiome Data and Comparative Studies with Publicly Available Data" and got an error message on the step 3.

Here is the error message:
"An unexpected error has occurred: (opt-einsum 0+untagged.61.gd905544.dirty (/home/qiime2/miniconda/envs/qiime2-2019.10/lib/python3.6/site-packages), Requirement.parse(‘opt-einsum>=2.3.2’), {‘tensorflow’})
See above for debug info.

The code used was:
qiime tools import \

  • ‐‐input‐path manifest.tsv \
  • ‐‐type `SampleData[SequencesWithQuality]’ \
  • ‐‐input‐format SingleEndFastqManifestPhred33V2 \
  • ‐‐output‐path se‐demux.qza

Could someone let me know what went wrong and how to correct it?

Many thanks in advance,
Ying

Hi @yinglu1!

For discussion about a similar error, please see

Friendly reminder: don’t forget to search the forum for previous discussions before you ask questions. :brain: :slightly_smiling_face:

Hi again, @yinglu1.

@thermokarst informed me that your error indicates that you might have manually edited the opt-einsum package manually within your QIIME 2 2019.10 env:

The .dirty part in this part of the message looks like a versioneer -produced tag, and the .dirty in versioneer means there are local edits.

Investigating the issues discussed in the post I linked you to should give you a get starting point/overview regarding problems with corrupt conda environments and how to clean them up and reinstall qiime 2. Starting over with a fresh QIIME 2 installation as well as fresh installations of additional plugins will likely do the trick for you, but please don’t hesitate to follow up if you run into further problems.

2 Likes

Hi @yinglu1,

I think this error may be due to this open issue with Songbird’s installation. Long story short, at one point the opt-einsum package was renamed to opt_einsum, and this breaks a lot of things. I suspect that opt_einsum is being installed in your conda environment.

It’s a bit of a hacky solution because you’ll have both versions of the package, but I think running conda install -c conda-forge opt-einsum within your conda environment may fix this problem.

2 Likes

Thank you All for the kind supporting!

Will installing a most current qiime2 version help to resolve those issues?

Kind Regards,
yinglu1

It might help, but I suspect this error will still happen even with a newer QIIME 2 version. I think running the command in the earlier comment (conda install -c conda-forge opt-einsum) should fix this problem, hopefully! *

* This being said, this command will install a slightly older version of the opt_einsum package (since the opt-einsum conda-forge feedstock apparently stopped being updated at v3.0.0, while the opt_einsum conda-forge feedstock is at v3.3.0) – I don’t think this should cause any problems for Songbird / TensorFlow, but can’t guarantee it.

1 Like

Thank you Marcus!

After installed latest qiime2 version 2020.08, I also installed additional dependencies from this link (https://github.com/qiime2/paper2/blob/master/notebooks/qiime2-protocol-CLI.ipynb) .
conda install songbird -c conda-forge
conda install -c conda-forge redbiom
conda install -c bioconda bowtie2
pip install https://github.com/knights-lab/SHOGUN/archive/master.zip
pip install https://github.com/qiime2/q2-shogun/archive/master.zip
conda install cytoolz
qiime dev refresh-cache

Everything seem okay but the import code has been running for over 16 hours and it’s still running. Is it normal? If not, what should I do?

The import code is:
qiime tools import
–input-path manifest.tsv
–type ‘SampleData[SequencesWithQuality]’
–input-format SingleEndFastqManifestPhred33V2
–output-path se-demux.qza

I haven’t install the (conda install -c conda-forge opt-einsum) yet.

Many thanks in advance!

Sorry, I’m not very familiar with the importing parts of QIIME 2 – not sure if this is normal or not. I’m guessing it is just taking a while, since I know validating / loading sequencing data can be very time-consuming.

1 Like

Silly question, but did you remember to press the Enter/Return key? I just looked at the provenance for the companion site (https://curr-protoc-bioinformatics.qiime2.org/), and the import there took less than 1 second.

Let us know!

:qiime2:

Hi Matthew, Thank you for taking time to help!

I did press the enter key to run the code last time. I also started from fresh by following the link you provided and had a different issue of running import this time. Here is the error message.

There was a problem importing manifest.tsv:

manifest.tsv is not a(n) SingleEndFastqManifestPhred33V2 file:

Found unrecognized ID column name ‘sample-id\tabsolute-filepath’ while searching for header. The first column name in the header defines the ID column, and must be one of these values:

Case-insensitive: ‘feature id’, ‘feature-id’, ‘featureid’, ‘id’, ‘sample id’, ‘sample-id’, ‘sampleid’

Case-sensitive: ‘#OTU ID’, ‘#OTUID’, ‘#Sample ID’, ‘#SampleID’, ‘sample_name’

NOTE: Metadata files must contain tab-separated values.

There may be more errors present in the metadata file. To get a full report, sample/feature metadata files can be validated with Keemei: https://keemei.qiime2.org

Find details on QIIME 2 metadata requirements here: https://docs.qiime2.org/2020.8/tutorials/metadata/

Hi @yinglu1, this part is the issue:

Did you run the command as specified in the tutorial?

echo -e "sample-id\tabsolute-filepath" > manifest.tsv

The -e is very important, in ensures that the \t is escaped to an actual “tab”.

Yes, I did. Here is the list of the file fold and screenshot of manifest.tsv file.


(qiime2-2020.8) [email protected]:~/qiime2‐ecam‐tutorial$ ls
81253.zip manifest.tsv mapping_files metadata.tsv per_sample_FASTQ
(qiime2-2020.8) [email protected]:~/qiime2‐ecam‐tutorial$

For some reason the whitespace values (\t) aren’t being escaped properly, let’s just double check that, first, please run the following and share the results here:

echo -e "hello\tyinglu1\ngoodbye" > test.txt
cat test.txt

:qiime2:

Here is the output:

hello yinglu1
goodbye

I have followed along with 10/5 Qiime2 online workshop tutorial successfully. So, I assume that my windows VB and qiime2 2020.08 setting is correct.

Perfect, so then the whitespace escaping is working as advertised, so maybe you forgot to do that when creating your manifest file? I suggest deleting the manifest file and re-running the command (being very careful to make sure you run the full command specified in the tutorial) - you have successfully demonstrated that you’re able to escape the non-printing whitespace characters in this exercise.

Thanks!

1 Like