New plugin (q2-srs) does not work for some users

Developer Discussion
1

Hi,

I have recently posted about a new QIIME2 plugin (q2-srs). It turns out some users are facing this issue, while others are not (eg, q2-srs runs normally from a Google colab). I am currently not able to reproduce that error, so it has been very hard to investigate what could be happening. So I am using my last resource that is to ask the QIIME2 team for help :wave:

For the ones facing the issue, it looks like the plugin is installed and all the files are there but somehow qiime is not looking for them where it should. So, is it something wrong with the plugin or the qiime env? Any help will be greatly appreciated.

Thank you,

Vitor

2 Likes
2

Just so the other :qiime2: devs are aware. I am also running into this issue.

I used the conda install -c vitorheidrich q2_srs approach.

When I try running the q2-srs tutorial I get:

(qiime2-2020.8) mike@microbiota:~/Documents/tmp/q2-srs> qiime srs SRScurve --i-table table-dada2.qza --p-step 100 --p-max-sample-size 3500 --p-rarefy-comparison --p-rarefy-comparison-legend --p-rarefy-repeats 100 --p-srs-color 'blue' --p-rarefy-color '#333333' --o-visualization SRScurve-plot.qzv --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: SRScurve.R /var/folders/ph/ms21wvc97pq3_cngx2zwlfy80000gn/T/tmpbit3_26b/table.tsv richness 100 0 3500 True 100 True blue #333333 solid longdash False /var/folders/ph/ms21wvc97pq3_cngx2zwlfy80000gn/T/qiime2-temp-xdbz5d1d

Traceback (most recent call last):
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in SRScurve
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRScurve.py", line 53, in SRScurve
run_commands([cmd])
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRScurve.py", line 29, in run_commands
subprocess.run(cmd, check=True)
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/subprocess.py", line 423, in run
with Popen(*popenargs, **kwargs) as process:
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/subprocess.py", line 1364, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'SRScurve.R': 'SRScurve.R'

Plugin error from srs:

[Errno 2] No such file or directory: 'SRScurve.R': 'SRScurve.R'

See above for debug info.

When I run which from within my qiime2-2020.8 environment I get:

(qiime2-2020.8) mike@microbiota: which SRScurve.R
/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/bin/SRScurve.R

It appears that SRScurve.R is there...

My qiime info is:

(qiime2-2020.8) mike@microbiota:~/Documents/tmp/q2-srs> qiime info
System versions
Python version: 3.6.11
QIIME 2 release: 2020.8
QIIME 2 version: 2020.8.0
q2cli version: 2020.8.0

Installed plugins
aldex2: 1.14.1
alignment: 2020.8.0
breakaway: 0+untagged.71.g503723a
clawback: 2019.7.0
composition: 2020.8.0
coremicrobiome: 1
corncob: 0.1.0
cutadapt: 2020.8.0
dada2: 2020.8.0
deblur: 2020.8.0
deicode: 0.2.4
demux: 2020.8.0
diversity: 2020.8.0
diversity-lib: 2020.8.0
emperor: 2020.8.0
empress: 1.0.0
feature-classifier: 2020.8.0
feature-table: 2020.8.0
fragment-insertion: 2020.8.0
gemelli: 0.0.5
gneiss: 2020.8.0
longitudinal: 2020.8.0
metadata: 2020.8.0
phylogeny: 2020.8.0
quality-control: 2020.8.0
quality-filter: 2020.8.0
qurro: 0.7.1
rescript: 2020.6.1+18.gf9be3c8
sample-classifier: 2020.8.0
songbird: 1.0.4
sourmash: 0.0.0
srs: 2020.8.0
taxa: 2020.8.0
types: 2020.8.0
vsearch: 2020.8.0

Application config directory
/Users/mike/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/var/q2cli

-I hope this helps!
-Mike

1 Like
3

The problem might be due to Python packaging – I’ve ran into finicky problems before where non-Python files would sometimes get downloaded and sometimes would be missing (causing problems like this).

It doesn’t look like q2-srs has a MANIFEST.in file – maybe try adding one of those to your repo and seeing if that helps? (Python docs on this.) We had to do this for q2-qurro in order to add in some files – the file we ended up with is https://github.com/biocore/qurro/blob/master/MANIFEST.in, for reference. (I think this is the sort of thing where ideally the setup.py file should tell pip to install some of the extra files, but sometimes that gets ignored… it might be dependent on the Python version, IIRC.)

I think you can test this by setting up different conda environments and running pip install git+https://github.com/vitorheidrich/q2-srs.git, or something like that – testing by running pip install -e . or something locally won’t reproduce the problem because the program will just be installed from the source directory you have set up, which will already include the files.

Hope this helps!

3 Likes
4

Hi Marcus,

Thank you for your suggestion!
At first, l was still not able to reproduce the problem, because even when using the alternative pip install command, all non-python files were being correctly downloaded and the plugin worked. Regarding the files being correctly downloaded, I believe that was also the case for @SoilRotifer and Pablo (the one that first reported me this issue), so that the problem is apparently occurring after this stage.

For comparison, when I ran which from within the qiime2-2020.8 (where q2-srs works) I also got:

(qiime2-2020.8) vitorheidrich@vheidrich:~$ which SRScurve.R
/home/vitorheidrich/miniconda3/envs/qiime2-2020.8/bin/SRScurve.R

Fortunately, when I tested the plugin with another machine I was able to reproduce the problem. Again, I ran which from within the qiime2-2020.8 (where q2-srs doesn’t work) and I got the same:

(qiime2-2020.8) vheidrich@pnd4:~$ which SRScurve.R
/home/tools/manual/miniconda3/envs/qiime2-2020.8/bin/SRScurve.R

Therefore, in summary, all files are always downloaded, but in some machines the R scripts are not found (even though they are there).

Even so, I tried your suggestion (including a MANIFEST.in file), but since the files were already being correctly downloaded - as expected - it didn’t work.

Any other suggestion? Thanks

1 Like
5

Hi @vheidrich, one thing I just realized is that the shebang line for SRScurve.R is:
#! /usr/bin/Rscript

Shouldn't this be altered based on the conda environment? That is, on my machine this should be:
#! /Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/bin/Rscript

When I edit this line and run the same command as I did before I get a new error:

qiime srs SRScurve --i-table table-dada2.qza --p-step 100 --p-max-sample-size 3500 --p-rarefy-comparison --p-rarefy-comparison-legend --p-rarefy-repeats 100 --p-srs-color 'blue' --p-rarefy-color '#333333' --o-visualization SRScurve-plot.qzv --verbose
Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: SRScurve.R /var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/tmp7mfpg8b2/table.tsv richness 100 0 3500 True 100 True blue #333333 solid longdash False /var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/qiime2-temp-mvwkcy_m

R version 3.5.1 (2018-07-02)
Error in contrib.url(repos, type) :
trying to use CRAN without setting a mirror
Calls: install.packages -> grep -> contrib.url
Execution halted
Traceback (most recent call last):
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2cli/commands.py", line 329, in call
results = action(**arguments)
File "", line 2, in SRScurve
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
output_types, provenance)
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 452, in callable_executor
ret_val = self._callable(output_dir=temp_dir, **view_args)
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRScurve.py", line 53, in SRScurve
run_commands([cmd])
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRScurve.py", line 29, in run_commands
subprocess.run(cmd, check=True)
File "/Users/robesonmichael/AnalysisSoftware/miniconda3/envs/qiime2-2020.8/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['SRScurve.R', '/var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/tmp7mfpg8b2/table.tsv', 'richness', '100', '0', '3500', 'True', '100', 'True', 'blue', '#333333', 'solid', 'longdash', 'False', '/var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/qiime2-temp-mvwkcy_m']' returned non-zero exit status 1.

Plugin error from srs:

Command '['SRScurve.R', '/var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/tmp7mfpg8b2/table.tsv', 'richness', '100', '0', '3500', 'True', '100', 'True', 'blue', '#333333', 'solid', 'longdash', 'False', '/var/folders/cc/rksyds7175786_2b0vhx4rrr0000gp/T/qiime2-temp-mvwkcy_m']' returned non-zero exit status 1.

See above for debug info.

I noticed this:

Error in contrib.url(repos, type) :
trying to use CRAN without setting a mirror
Calls: install.packages -> grep -> contrib.url
Execution halted

Any ideas?

1 Like
6

Good catch @SoilRotifer! I haven't followed this discussion too closely yet, so apologies if I missed this above, but I think the correct hash-pling is:

#!/usr/bin/env Rscript

this ensures that the script runs using the conda env R, versus a system-wide R.

2 Likes
7

Oh right! :man_facepalming:

2 Likes
8

I also tried to run install.packages("SRS") within R of my qiime2-2020.8 environment and it fails to build (Mac OS X 10.15.7).

If the install.packages issue above can be resolved, it might be a good idea to remove the install.packages commands from the SRScurve.R file, and just inform users to install the R portion manually, as done for other packages like ALDEx2.

1 Like
9

Hi @SoilRotifer,

Thank you for your suggestions! Changing the shebang line mostly solved the issue (thanks @thermokarst!). Regarding the second problem, it was just a matter of assigning a valid CRAN repo.

Now I believe the problem is fixed! Please let me know if you run into any other issue.

Cheers,

Vitor

2 Likes
10

Silly me forgot to change the shebang line in the other R file, which is why I was still unable to get it to work. But it works now after your update @vheidrich ! :raised_hands:

Glad we were able to help solve this! :man_mechanic:

2 Likes
11

Hi @vheidrich

And how do you assign a CRAN repo? Because I changed the first line of the SRS.R file into

#!/usr/bin/env Rscript

But now i get a different error:

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: SRS.R /tmp/tmpumsjjvqw/table.tsv 1000 True 1

R version 3.5.1 (2018-07-02) 
Error in contrib.url(repos, type) : 
  trying to use CRAN without setting a mirror
Calls: install.packages -> grep -> contrib.url
Execution halted
Traceback (most recent call last):
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2cli/commands.py", line 329, in __call__
    results = action(**arguments)
  File "<decorator-gen-121>", line 2, in SRS
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    output_types, provenance)
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/qiime2/sdk/action.py", line 390, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRS.py", line 45, in SRS
    run_commands([cmd])
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/site-packages/q2_srs/_SRS.py", line 28, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/sam/anaconda3/envs/qiime2-2020.8/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['SRS.R', '/tmp/tmpumsjjvqw/table.tsv', '1000', 'True', '1']' returned non-zero exit status 1.

Plugin error from srs:

  Command '['SRS.R', '/tmp/tmpumsjjvqw/table.tsv', '1000', 'True', '1']' returned non-zero exit status 1.

See above for debug info.

Cheers,
Pablo

12

Hi @Pablo_V,

That was a discussion on what I needed to do (as a developer) to update the plugin to get rid of those errors. Now that the plugin is updated, there is no need for users to manually change files. So please just reinstall q2-srs and you already will have the updated/fixed plugin.

Reinstallation instructions:

  1. deactivate your qiime env
  2. cd into your qiime env directory (in my case, located at /home/vitorheidrich/miniconda3/envs/qiime2-2020.8)
  3. run:
    rm -r bin/SRS* lib/python3.6/site-packages/*srs*
  1. follow the installation instructions provided here

Cheers,

Vitor

2 Likes
13

2 off-topic replies have been split into a new topic: q2-srs python incomptability

Please keep replies on-topic in the future.

14

An off-topic reply has been split into a new topic: I Need Help with q2-srs

Please keep replies on-topic in the future.