Hi. I installed Qiime2 via Virtual Box in August following the instructions posted on https://docs.qiime2.org/2017.8/install/virtual/virtualbox/ … So I don’t know how I would have ended up with an older version of Qiime2? Did it just recently get updated? It says “qiime2@qiime2core2017-6” in the terminal window. If I need to update it, are there instructions somewhere on how to do this.
As far as my metadata file not being recognized as a Qiime2 artifact… here is a sample of what that file looks like:
#SampleID Treatment Exper.Day Replicate Morning pH
A2 8.0 2 A 7.82
A4 8.0 4 A 8.01
A6 8.0 6 A 7.97
A8 8.0 8 A 8.01
A10 8.0 10 A 8.25
A12 8.0 12 A 8.32
A14 8.0 14 A 8.08
A16 8.0 16 A 8.17
A18 8.0 18 A 8.11
A20 8.0 20 A 8.05
B2 8.0 2 B 7.84
B4 8.0 4 B 7.98
B6 8.0 6 B 8.03
B8 8.0 8 B 8.25
B10 8.0 10 B 8.26
B12 8.0 12 B 8.29
B14 8.0 14 B 8.00
B16 8.0 16 B 8.08
B18 8.0 18 B 8.08
B20 8.0 20 B 7.98
By the way, the variables “Treatment”, “Exper.Day, and “Morning pH” are all numerical whereas"Replicate” is categorical and can be A,B,C,D,E, or F. I originally had “Treatment” be categorical as “High pH” or “Low pH” but then changed it to numerical because some of the alpha diversity analyses required that it be numerical.
Thanks.
Maybe you copied the wrong link in the download section? QIIME 2017.6 was the current release version at the QIIME 2 Las Vegas workshop, so maybe you had grabbed the old release then? Either way, the link you provided above is indeed the right place to go to find out all about the latest release of QIIME 2!
Maybe your metadata file was tweaked when you copied-and-pasted, but that doesn't look like a tab-separated file to me (e.g. TSV file)! If you are using a tool like Excel or Google Sheets, can you try re-exporting to TSV (tab-separated) and try again? For now, QIIME 2 only supports TSV files for user-supplied metadata. Good luck!
Hi Matt. I did install the latest version of Qiime2 in late August when following the instructions on the website for installing Qiime2 using virtual box. I went back and checked the instructions and the screen shots show Qiime2 Core - 2.0.6. Is it possible that Qiime 2 2017.8 was not made available for virtual box download only native dowload?
Regarding my metadata, I did export my excel data as a tab separated file, I didn’t cut and paste anything. Can you explain why it doesn’t look like a tab delimited file? What appears to be incorrect in terms of how it is formatted?
A few screenshots do show that, but there is also a note in the docs, copied here:
Note, the list of installed plugins (and their versions) will likely differ depending on the version of the VirtualBox virtual machine that you chose to download and install.
This is so that we don't need to recreate screenshots with every release!
The https://data.qiime2.org/distro/core/2017.8 is the one for the current release (although at this point you might want to wait until the end of next week to upgrade - QIIME 2017.9 is due out next week).
Yep, sorry I wasn't clear! There are no tabs in that sample you posted - only spaces, which is why I thought it didn't look like a tab-delimited file.
Sorry, I was talking about copy-and-pasting the contents of your metadata file into the Forum, as opposed to attaching your metadata file to your Forum reply --- what I was getting at here is by pasting that snippet of your metadata into the Forum, some of the formatting might have been lost.
Apologies if I missed this in your reply, but were you able to try this? If so and you are still seeing the error, then upgrading to the latest version of QIIME 2 will be the next step here (quite a bit of the Metadata code has changed since 2017.6).
HI Matt, Yes, the metadata file did get tweaked when I cut and pasted it into my reply to you. It does in fact have tabs in it. I just got the latest version of Qiime2 natively installed on a MacBook Pro so I will have a go at this command using this new set up and see how it goes…
I successfully validated my metadata file using the Keemei and it said that it was a valid Qiime2 mapping file. And yet when I run the command below using the latest version of Qiime2 (2017.8) I get a ValueError saying that, “%s is not a QIIME archive.” % filepath… metadataonlymesocosm2.txt is not a QIIME archive.It also states that Non-string Metadata index values detected. I don’t know how I could get this message if my data file was successfully validated with Keemei?
Please advise…
Hi @Lisa_Crummett - this is interesting --- can you please provide your metadata file so that we can figure out why things look okay to Keemei but not to QIIME 2? If you can do this, feel free to send to me in a direct message. Thank you!
Hi @Lisa_Crummett! Thanks for sharing your metadata, that was really helpful!
The problem is that your metadata file has several empty rows near the bottom. I took a screenshot of your file opened in my text editor, with some additional syntax highlighting turned on:
The error message you saw about non-string index values, while technically true, isn't a very helpful error message. I opened up an issue on the QIIME 2 issue tracker to start working on updating that error message to something a bit more helpful.
The reason this didn't show up as an error in Keemei has to do with the way Google Sheets treats data, and in particular, how it handles "empty cells." There isn't really a way to disambiguate between user-supplied empty cells, versus the empty cells that already are part of the spreadsheet as additional rows or columns.
Moving forward, if you delete the 4 extra lines at the bottom of the file and run that through QIIME 2 you should be in good shape (I tested that out locally and your metadata worked just fine in QIIME 2). Keep us posted!
Thank you for your handy detective work Matt. I am sorry that I am keeping you so busy with all of my questions!
I deleted those extra rows and resaved the metadata file and tried running the emperor plat command again and I got another error message saying that I had 17 values in the "Exper.Day" column that were non-numeric but this isn't true. This was true several steps back but I deleted those rows so that all of the data was numeric and saved the file as "metadataonlymesocosm2.txt". The bray_curtis_pcoa_results.qza file was not dependent on any previous metadata files right? So why I am I getting this error message:
All values in a custom axis must be numeric, this summary shows the
invalid values and the number of times they were found in column
'Exper.Day': NaN 17
Please advise... you should still have my metadata file if you want to look at it.
This is actually an error message coming from Emperor, not from QIIME 2, and what it is saying is that your bray_curtis_pcoa_results.qza file has 17 samples in it that aren't present in your sample metadata file. This make sense, since you mentioned:
this was true several steps back but I deleted those rows so that all of the data was numeric and saved the file as “metadataonlymesocosm2.txt”.
You have a few options:
Add the samples back to your metadata file and put some sentinel value in those fields (e.g. -1)
Filter the samples that don't have numeric values in their metadata out of your feature table and recompute the core-diversity step using that filtered table.
Maybe there is some other option: pinging @yoshiki, for an assist on emperor specifics.
Thanks Matt. I figured that the it was something like that holding it up. I went ahead and added back those missing 17 samples and gave them dummy values and saved that file as "metadata.txt" but after I ran the emperor plot command again, I got an error saying that the file did not exist even though it clearly does exist in the same directory as the rest of the data files:
This isn't really an error from QIIME 2, just a general command line issue - you told a command (QIIME 2 in this case) that a file (your metadata) was somewhere, with a certain name, but it isn't actually there.
It looks like you forgot to change working directories before running this command: in your first command block in this thread you appear to be working in a directory called Qiime2data. In your most recent post, you are working in your home directory, the shorthand of which is ~. The filepath you specified for your metadata (metadata.txt) doesn't give an absolute location, so your command line shell is going to look for that file in your working directory. Since you are in the wrong directory, it technically doesn't exist, as far as your computer can tell. Again, this isn't related to QIIME 2, but is more of a general computer situation.
Thanks, I feel silly... I had given the command "cd Qiime2data" at the start but left out the forward slash so it didn't work and I wasn't actually in the correct directory. Ugh. Okay so now I am in the correct directory and I am using a metadata file that has numerical values for every sample "metadata3" but I am still getting an error message that there are 16 values in the "Exper.Day" column that are not numeric. I don't see extra rows in my text file so I don't know what the problem is. I have attached the file here so that you can take a look at it. metadata3.txt (1.7 KB)
The set of Sample IDs in your metadata aren't the same as the set of Sample IDs in your feature table (which was used to produce the file bray_curtis_pcoa_results.qza).
One way to troubleshoot this is to generate a table summary of your feature table (check out this section in the moving pictures tutorial for some pointers). That viz summarizes samples and features in your feature table, and there is a little CSV download link in the first half of the viz:
In particular, "Frequency per sample detail" is what you are looking for.
Grab this CSV and fire up your favorite spreadsheet editor. Then start overlaying your sample metadata with this file, and make sure that your IDs match --- there are a few ways to do this (sorting comes to mind). Ultimately you are looking for some discrepancies: maybe a missing sample, maybe there is a typo in your metadata, maybe something else altogether.
I have opened another issue on the q2-emperor issue tracker to provide a more detailed error message when the Sample IDs aren't matching (like what you are experiencing here).
TOUCHDOWN! It worked! You were right in that some of the names in my metadata file were slightly different than what was in my feature table (not sure how I managed to do that?). Anyhow, I modified the names in my metadata file so that they would match up with my feature table and BAM the pcoa commands worked!
The release of QIIME 2 2017.9 includes improvements based on the issues encountered in this thread: when loading TSV-based metadata, QIIME 2 now strips out any rows made entirely of whitespace. As well, the error message raised by QIIME 2 when it has a hard time parsing TSV data has been improved so that it is a bit more concise, and has more specific help text on how to fix the problems observed. Thanks!
