Error in command for generating PCoA plot

Lisa_Crummett · September 15, 2017, 5:06pm

Hello,
I would like to generate PCoA plots. Some of my samples do not have a value for the category that I am using for the thirs PCoA axis, whichis "Exper.Day" and I was receiving an error message telling me this. Therefore, I deleted those samples from my metadata file and renamed my metadata file "metadatamesocosymonly.txt" instead of "metadata.txt". I used the following commands:

qiime emperor plot \
  --i-pcoa core-metrics-result/unweighted_unifrac_pcoa_results.qza \
  --m-metadata-file metadataonlymesocosm.txt \
  --p-custom-axis Exper.Day \
  --o-visualization core-metrics-result/unweighted-unifrac-emperor-Exper.Day.qzv

qiime emperor plot \
  --i-pcoa core-metrics-result/bray_curtis_pcoa_results.qza \
  --m-metadata-file metadataonlymesocosm.txt \
  --p-custom-axis Exper.Day \
  --o-visualization core-metrics-result/bray-curtis-emperor-Exper.Day.qzv

I received the following error response and it looks like the Unifrac command was unsuccessful but the Bray Curtis command is still running? Can you confirm this and if so, tell me why the Unifrac command was unsuccessful and also whether or not the Bray Curtis command will also likely be unsuccessful?

Traceback (most recent call last):
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/handlers.py", line 366, in get_value
    artifact = qiime2.Artifact.load(path)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/result.py", line 61, in load
    archiver = archive.Archiver.load(filepath)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/archive/archiver.py", line 289, in load
    archive = cls.get_archive(filepath)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/archive/archiver.py", line 254, in get_archive
    raise ValueError("%s is not a QIIME archive." % filepath)
ValueError: metadataonlymesocosm.txt is not a QIIME archive.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/bin/qiime", line 6, in <module>
    sys.exit(q2cli.__main__.qiime())
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/commands.py", line 189, in __call__
    arguments, missing_in, verbose, quiet = self.handle_in_params(kwargs)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/commands.py", line 281, in handle_in_params
    kwargs, fallback=cmd_fallback
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/handlers.py", line 368, in get_value
    metadata.append(qiime2.Metadata.load(path))
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/metadata.py", line 103, in load
    return cls(df)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/metadata.py", line 39, in __init__
    if illegal_chars & set(value):
TypeError: 'float' object is not iterable
(qiime2-2017.6) qiime2@qiime2core2017-6:/media/sf_Qiime2data/FHLexper/paired-end-sequences/699$ 
(qiime2-2017.6) qiime2@qiime2core2017-6:/media/sf_Qiime2data/FHLexper/paired-end-sequences/699$ qiime emperor plot \
>   --i-pcoa core-metrics-result/bray_curtis_pcoa_results.qza \
>   --m-metadata-file metadataonlymesocosm.txt \
>   --p-custom-axis Exper.Day \
>   --o-visualization core-metrics-result/bray-curtis-emperor-Exper.Day.qzv^C
(qiime2-2017.6) qiime2@qiime2core2017-6:/media/sf_Qiime2data/FHLexper/paired-end-sequences/699$

thermokarst · September 18, 2017, 7:09pm

Hi @Lisa_Crummett, it looks like you are using an unsupported version of QIIME 2 (we are only able to support the latest release right now while we are in this transitional period between QIIME 1 and QIIME 2). Do you see this same error if you rerun with QIIME 2 2017.8?

It looks like something might be wrong with the formatting of your metadata - would it be possible for you to provide a brief sample of metadataonlymesocosm.txt? Thanks!

Lisa_Crummett · September 19, 2017, 1:03am

Hi. I installed Qiime2 via Virtual Box in August following the instructions posted on Installing QIIME 2 using VirtualBox — QIIME 2 2017.8.0 documentation .... So I don't know how I would have ended up with an older version of Qiime2? Did it just recently get updated? It says "qiime2@qiime2core2017-6" in the terminal window. If I need to update it, are there instructions somewhere on how to do this.

As far as my metadata file not being recognized as a Qiime2 artifact... here is a sample of what that file looks like:

#SampleID Treatment Exper.Day Replicate Morning pH
A2 8.0 2 A 7.82
A4 8.0 4 A 8.01
A6 8.0 6 A 7.97
A8 8.0 8 A 8.01
A10 8.0 10 A 8.25
A12 8.0 12 A 8.32
A14 8.0 14 A 8.08
A16 8.0 16 A 8.17
A18 8.0 18 A 8.11
A20 8.0 20 A 8.05
B2 8.0 2 B 7.84
B4 8.0 4 B 7.98
B6 8.0 6 B 8.03
B8 8.0 8 B 8.25
B10 8.0 10 B 8.26
B12 8.0 12 B 8.29
B14 8.0 14 B 8.00
B16 8.0 16 B 8.08
B18 8.0 18 B 8.08
B20 8.0 20 B 7.98

Please advise...

Lisa_Crummett · September 19, 2017, 1:08am

By the way, the variables "Treatment", "Exper.Day, and "Morning pH" are all numerical whereas"Replicate" is categorical and can be A,B,C,D,E, or F. I originally had "Treatment" be categorical as "High pH" or "Low pH" but then changed it to numerical because some of the alpha diversity analyses required that it be numerical.
Thanks.

thermokarst · September 19, 2017, 3:25am

Hi @Lisa_Crummett!

Maybe you copied the wrong link in the download section? QIIME 2017.6 was the current release version at the QIIME 2 Las Vegas workshop, so maybe you had grabbed the old release then? Either way, the link you provided above is indeed the right place to go to find out all about the latest release of QIIME 2!

Maybe your metadata file was tweaked when you copied-and-pasted, but that doesn't look like a tab-separated file to me (e.g. TSV file)! If you are using a tool like Excel or Google Sheets, can you try re-exporting to TSV (tab-separated) and try again? For now, QIIME 2 only supports TSV files for user-supplied metadata. Good luck!

Lisa_Crummett · September 21, 2017, 4:30am

Hi Matt. I did install the latest version of Qiime2 in late August when following the instructions on the website for installing Qiime2 using virtual box. I went back and checked the instructions and the screen shots show Qiime2 Core - 2.0.6. Is it possible that Qiime 2 2017.8 was not made available for virtual box download only native dowload?

Regarding my metadata, I did export my excel data as a tab separated file, I didn't cut and paste anything. Can you explain why it doesn't look like a tab delimited file? What appears to be incorrect in terms of how it is formatted?

Thanks,
Lisa

thermokarst · September 21, 2017, 1:04pm

A few screenshots do show that, but there is also a note in the docs, copied here:

Note, the list of installed plugins (and their versions) will likely differ depending on the version of the VirtualBox virtual machine that you chose to download and install.

This is so that we don't need to recreate screenshots with every release!

No, I don't think it was possible, we announced the VirtualBox build within a day of the main release announcement.

Do you see a page that looks like this when you click on the "VirtualBox Images download link" in the VirtualBox Install Doc?

The https://data.qiime2.org/distro/core/2017.8 is the one for the current release (although at this point you might want to wait until the end of next week to upgrade - QIIME 2017.9 is due out next week).

Yep, sorry I wasn't clear! There are no tabs in that sample you posted - only spaces, which is why I thought it didn't look like a tab-delimited file.

Sorry, I was talking about copy-and-pasting the contents of your metadata file into the Forum, as opposed to attaching your metadata file to your Forum reply --- what I was getting at here is by pasting that snippet of your metadata into the Forum, some of the formatting might have been lost.

Apologies if I missed this in your reply, but were you able to try this? If so and you are still seeing the error, then upgrading to the latest version of QIIME 2 will be the next step here (quite a bit of the Metadata code has changed since 2017.6).

Thanks!

Lisa_Crummett · September 21, 2017, 10:55pm

HI Matt, Yes, the metadata file did get tweaked when I cut and pasted it into my reply to you. It does in fact have tabs in it. I just got the latest version of Qiime2 natively installed on a MacBook Pro so I will have a go at this command using this new set up and see how it goes...

Lisa_Crummett · September 22, 2017, 12:03am

I successfully validated my metadata file using the Keemei and it said that it was a valid Qiime2 mapping file. And yet when I run the command below using the latest version of Qiime2 (2017.8) I get a ValueError saying that, "%s is not a QIIME archive." % filepath..... metadataonlymesocosm2.txt is not a QIIME archive.It also states that Non-string Metadata index values detected. I don't know how I could get this message if my data file was successfully validated with Keemei?
Please advise...

qiime emperor plot
--i-pcoa core-metrics-result/unweighted_unifrac_pcoa_results.qza
--m-metadata-file metadataonlymesocosm2.txt
--p-custom-axis Exper.Day
--o-visualization core-metrics-result/unweighted-unifrac-emperor-Exper.Day.qzv

qiime emperor plot
--i-pcoa core-metrics-result/bray_curtis_pcoa_results.qza
--m-metadata-file metadataonlymesocosm2.txt
--p-custom-axis Exper.Day
--o-visualization core-metrics-result/bray-curtis-emperor-Exper.Day.qzv

thermokarst · September 22, 2017, 12:46am

Hi @Lisa_Crummett - this is interesting --- can you please provide your metadata file so that we can figure out why things look okay to Keemei but not to QIIME 2? If you can do this, feel free to send to me in a direct message. Thank you!

Lisa_Crummett · September 22, 2017, 12:54am

Sure, I have attached my metadata file.

Cheers,
Lisametadataonlymesocosm2.txt (1.2 KB)

thermokarst · September 22, 2017, 3:37am

Hi @Lisa_Crummett! Thanks for sharing your metadata, that was really helpful!

The problem is that your metadata file has several empty rows near the bottom. I took a screenshot of your file opened in my text editor, with some additional syntax highlighting turned on:

The error message you saw about non-string index values, while technically true, isn't a very helpful error message. I opened up an issue on the QIIME 2 issue tracker to start working on updating that error message to something a bit more helpful.

The reason this didn't show up as an error in Keemei has to do with the way Google Sheets treats data, and in particular, how it handles "empty cells." There isn't really a way to disambiguate between user-supplied empty cells, versus the empty cells that already are part of the spreadsheet as additional rows or columns.

Moving forward, if you delete the 4 extra lines at the bottom of the file and run that through QIIME 2 you should be in good shape (I tested that out locally and your metadata worked just fine in QIIME 2). Keep us posted!

Lisa_Crummett · September 22, 2017, 5:16am

Thank you for your handy detective work Matt. I am sorry that I am keeping you so busy with all of my questions!

I deleted those extra rows and resaved the metadata file and tried running the emperor plat command again and I got another error message saying that I had 17 values in the "Exper.Day" column that were non-numeric but this isn't true. This was true several steps back but I deleted those rows so that all of the data was numeric and saved the file as "metadataonlymesocosm2.txt". The bray_curtis_pcoa_results.qza file was not dependent on any previous metadata files right? So why I am I getting this error message:

(qiime2-2017.8) Helps-MacBook-Pro:Qiime2data lisacrummett$ qiime emperor plot \

--i-pcoa core-metrics-result/bray_curtis_pcoa_results.qza
--m-metadata-file metadataonlymesocosm2.txt
--p-custom-axis Exper.Day
--o-visualization core-metrics-result/bray-curtis-emperor-Exper.Day.qzv

Plugin error from emperor:

All values in a custom axis must be numeric, this summary shows the
invalid values and the number of times they were found in column
'Exper.Day': NaN 17

Please advise... you should still have my metadata file if you want to look at it.

thermokarst · September 22, 2017, 12:46pm

No worries - that is what we are here for!

Correct!

This is actually an error message coming from Emperor, not from QIIME 2, and what it is saying is that your bray_curtis_pcoa_results.qza file has 17 samples in it that aren't present in your sample metadata file. This make sense, since you mentioned:

this was true several steps back but I deleted those rows so that all of the data was numeric and saved the file as “metadataonlymesocosm2.txt”.

You have a few options:

Add the samples back to your metadata file and put some sentinel value in those fields (e.g. -1)
Filter the samples that don't have numeric values in their metadata out of your feature table and recompute the core-diversity step using that filtered table.
Maybe there is some other option: pinging @yoshiki, for an assist on emperor specifics.

Thanks @Lisa_Crummett!

Lisa_Crummett · September 22, 2017, 7:26pm

Thanks Matt. I figured that the it was something like that holding it up. I went ahead and added back those missing 17 samples and gave them dummy values and saved that file as "metadata.txt" but after I ran the emperor plot command again, I got an error saying that the file did not exist even though it clearly does exist in the same directory as the rest of the data files:

(qiime2-2017.8) Helps-MacBook-Pro:~ lisacrummett$ qiime emperor plot \

--i-pcoa core-metrics-result/unweighted_unifrac_pcoa_results.qza
--m-metadata-file metadata.txt
--p-custom-axis Exper.Day
--o-visualization core-metrics-result/unweighted-unifrac-emperor.qzv
Usage: qiime emperor plot [OPTIONS]

Error: Invalid value for "--m-metadata-file": Path "metadata.txt" does not exist

thermokarst · September 22, 2017, 9:44pm

Hi @Lisa_Crummett!

This isn't really an error from QIIME 2, just a general command line issue - you told a command (QIIME 2 in this case) that a file (your metadata) was somewhere, with a certain name, but it isn't actually there.

It looks like you forgot to change working directories before running this command: in your first command block in this thread you appear to be working in a directory called Qiime2data. In your most recent post, you are working in your home directory, the shorthand of which is ~. The filepath you specified for your metadata (metadata.txt) doesn't give an absolute location, so your command line shell is going to look for that file in your working directory. Since you are in the wrong directory, it technically doesn't exist, as far as your computer can tell. Again, this isn't related to QIIME 2, but is more of a general computer situation.

Thanks!

Lisa_Crummett · September 22, 2017, 11:24pm

Thanks, I feel silly... I had given the command "cd Qiime2data" at the start but left out the forward slash so it didn't work and I wasn't actually in the correct directory. Ugh. Okay so now I am in the correct directory and I am using a metadata file that has numerical values for every sample "metadata3" but I am still getting an error message that there are 16 values in the "Exper.Day" column that are not numeric. I don't see extra rows in my text file so I don't know what the problem is. I have attached the file here so that you can take a look at it. metadata3.txt (1.7 KB)

thermokarst · September 23, 2017, 4:28am

Okay, so we are back to where we were a few posts ago:

The set of Sample IDs in your metadata aren't the same as the set of Sample IDs in your feature table (which was used to produce the file bray_curtis_pcoa_results.qza).

One way to troubleshoot this is to generate a table summary of your feature table (check out this section in the moving pictures tutorial for some pointers). That viz summarizes samples and features in your feature table, and there is a little CSV download link in the first half of the viz:

In particular, "Frequency per sample detail" is what you are looking for.

Grab this CSV and fire up your favorite spreadsheet editor. Then start overlaying your sample metadata with this file, and make sure that your IDs match --- there are a few ways to do this (sorting comes to mind). Ultimately you are looking for some discrepancies: maybe a missing sample, maybe there is a typo in your metadata, maybe something else altogether.

I have opened another issue on the q2-emperor issue tracker to provide a more detailed error message when the Sample IDs aren't matching (like what you are experiencing here).

Keep us posted, and thanks for sticking with it!

Lisa_Crummett · September 26, 2017, 7:10pm

TOUCHDOWN! It worked! You were right in that some of the names in my metadata file were slightly different than what was in my feature table (not sure how I managed to do that?). Anyhow, I modified the names in my metadata file so that they would match up with my feature table and BAM the pcoa commands worked!

Thank you so much!

thermokarst · September 29, 2017, 7:21pm

The release of QIIME 2 2017.9 includes improvements based on the issues encountered in this thread: when loading TSV-based metadata, QIIME 2 now strips out any rows made entirely of whitespace. As well, the error message raised by QIIME 2 when it has a hard time parsing TSV data has been improved so that it is a bit more concise, and has more specific help text on how to fix the problems observed. Thanks!