Issue Running Codes

I hope this finds everyone having a great summer!

I am running Qiime2 (currently 2019.4) on Linux for microbiome data analysis. Last week, I started having issues running codes. Some of my codes would not fully paste into summit, resulting in an error output, then it would paste the rest of the code itself and give a bash error command not found (see attachment).

As a result, I thought it may be an issue with Qiime2 as I was running 2019.1 at the time and there is currently 2019.4 out. Therefore, I updated my Qimme2 to 2019.4. However, upon doing this I have had even more persistent issues running codes. I have trouble shot myself via Google and the Qiime2 forum. Based off of what I read, I uninstalled Miniconda3 and installed Miniconda2 with 2019.4, yet still I had the same issue.

Furthermore, I also attempted to go back to the 2019.1 version of Qiime2 with no success. Finally, based off of a slurm output, it seemed it could be an issue with the locale. However, the codes to change locale suggested in the slurm output export LC_ALL=C.UTF-8;
export LANG=C.UTF-8; OR export LC_ALL=US.UTF-8
export LANG=US.UTF-8 (found here under Traceback Runtime error: Python 3 Support — Click Documentation (7.x)) did not work either.

I attached screenshots of my slurm outputs for SBATCH attempts as well as a screenshot of codes not fully entering in the sinteractive. As I mentioned above, they paste partially, give an error saying I need the rest of the code, then it pastes the rest of the code itself.

Any help you can give is greatly appreciated as my data analysis is currently at a standstill.

  • Scott Wrigley
    Graduate Student
    Integrative Cardiovascular Physiology and Intestinal Health Labs
    Department of Food Science and Human Nutrition
    Colorado State University

Hey Scott!
Welcome to the forum. Your post is pretty broad - if it’s ok by you, I’d like to start small. :microbe: Based on the command you screencaptured, and the error message it produced, it looks like you might not be passing all of the required arguments to the qiime demux emp-paired command. We’ll troubleshoot that first, and if that doesn’t fix the issue, we can take it from there.

Take a look at your command, and be sure to read the red text in the error message. That should tell you what’s missing from what you tried to input.

Take your time, try that command again with all of the required arguments, and if you have any questions about which arguments are required, or what they mean, try reading the demux documentation. If it’s easier, you can get the same information by asking QIIME 2 for command help directly. E.g. enter qiime demux emp-paired --help from the command line.

If that doesn’t work, drop me a line here. I’d be happy to help.
:sun_with_face: Chris

p.s.
The structure of QIIME 2 commands is consistent, and understanding how they work may save you stress caused by copy-pasting in the future:

First, you are telling the command line what program you want to use: qiime
Then, you indicate the plugin you need: demux…
…and the command from that plugin you want to use: emp-paired.
The \ characters are escape characters that let you write one command on multiple lines. This doesn’t do anything special, but sometimes makes commands easier to read.
All of the --*-something lines tell your command what options you want it to use.

Please forgive me if this is too basic for you! Sometimes it’s hard to tell how much experience people have with command-line interfaces. :blush:

3 Likes

@WrigleyS try to nano the code into a bash file and then submit it to slurm. It looks like from the input your placed on the command prompt the code executed before you submitted all of your commands (as Keefe pointed out).

http://linuxcommand.org/lc3_wss0010.php

nano script.sh (put your bash script here)
chmod a+xwr script.sh
sbatch script.sh to slurm

1 Like

Chris,

I appreciate the reply!

No worries on starting broad. I totally understand.

I had everything in the code, including what was in red. “(1/1) Missing Option --o-error-correction-details”…The problem is, when I copy and paste the code from my text editor, it doesn’t all paste initially. Then, after giving me the error saying I didn’t have “(1/1) Missing Option --o-error-correction-details” as a part of my code, if pastes that portion of the code in without my input (as I already pasted the whole code in) and says “bash: --o-error-correction-details: command not found” So it is taking the whole code in, just in parts.

1 Like

Ben,

Thank you for the reply! I will give this a try. I normally nano and sbatch to .txt files. I’ll try .sh to see if that makes any difference!

by the way, it may not be recognizing the “” to tell it to go to the next line. Could this be because of some text issue? Copy/paste it into a text editor and then paste it into the command line? Ben

1 Like

Ben,

Thanks for the script suggestion. I ran it as follows with the following input, however I still get the same slurm error message that is also below.

nano beet-saliva.sh
chmod a+xwr beet-saliva.sh
sbatch beet-saliva.sh

With the following script:

#!/bin/sh
#SBATCH --job-name=demux_beet_saliva
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=05:00:00
#SBATCH --mail-type=ALL
#SBATCH [email protected]

source activate qiime2-2019.4

qiime demux emp-paired \
  --m-barcodes-file Beet_saliva_metadata.txt \
  --m-barcodes-column BarCodeSequence \
  --i-seqs emp-paired-end-sequences-beet-study.qza \
  --o-per-sample-sequences demux_beet_saliva.qza
  --o-error-correction-details demux-details.qza

@WrigleyS, if you read your command closely, you'll notice a \ character missing at the end of your second-to-last line.

Without that character, you are telling the command line to run two separate commands:

qiime demux emp-paired \
  --m-barcodes-file Beet_saliva_metadata.txt \
  --m-barcodes-column BarCodeSequence \
  --i-seqs emp-paired-end-sequences-beet-study.qza \
  --o-per-sample-sequences demux_beet_saliva.qza

AND

  --o-error-correction-details demux-details.qza

Whether you run these straight from the command line, or with an sbatch script, neither will produce results.

I make mistakes like this all the time, and they can be a little painful to troubleshoot at first. There's no way to prevent them all, but re-reading your commands and error messages carefully will often give you the insight you need to get things back on track.

Good luck with the rest of the analysis!
Chris

2 Likes

Chris,

Thank you again for your assistance! I really appreciate it!

I did correct that missing bit of code (), however, I still get the same exact error with my sbatch about the ASCII as encoding for the environment and the locales as I was before. :frowning:

 #!/bin/sh
#SBATCH --job-name=demux_beet_saliva
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=02:00:00
#SBATCH --mail-type=ALL
#SBATCH [email protected]

source activate qiime2-2019.4

qiime demux emp-paired \
  --m-barcodes-file Beet_saliva_metadata.txt \
  --m-barcodes-column BarCodeSequence \
  --i-seqs emp-paired-end-sequences-beet-study.qza \
  --o-per-sample-sequences demux_beet_saliva.qza \
  --o-error-correction-details demux-details.qza

1 Like

One down, one to go! :wink:
May be an OS issue. The small print on the click documentation page you found (good sleuthing, btw! :male_detective:) suggests some alternative formats for the export command you tried. Details here:

4 Likes

Chris,

Thank you so much for all your help! I’ve been at this 5 months so this is still pretty fresh to me.

Once the command line issue was corrected, I utilized LC_ALL=en_US.UTF8 and my code ran.

I don’t want to take up all your time, but I have one more question. Will I have to run the above locale code (LC_ALL=en_US.UTF8) before each command, or just at the beginning of the session similar to source activating Qiime?

2 Likes

Great question, @WrigleyS! I’ve never had this issue myself, so your best bet is probably to try it and find out. :face_with_monocle: If I had to guess, it will probably stick through the whole session.

If you get tired of setting this variable at login, you could probably add your locale-setting command to ~/.bashrc or ~/.bash_profile. This SO post isn’t perfectly relevant, but does describe which files are used with different shell ‘flavors’, which may help you decide where to put your commands.

1 Like

Chris,

I can't thank you enough. It was frustrating that after a week of trouble shooting, the main issue was the backtick! haha. We have a great community here with scientists working together to collectively make data analysis better. I will keep playing with the locale and see what I find.

2 Likes

Glad everything worked out so well, Scott. Happy :qiime2:-ing!

1 Like

A post was split to a new topic: Error when using q2-demux