Dsfdr plugin question/s and support

Hello,

I am new to qiime and am in need of some help. I would like to run the q2_dsfdr plugin on my data. However, I have absolutely no idea how to do this. As of Feb. 5th 2018, @serenejiang stated that the plugin was now working. I installed the plugin and everything seemed to work fine. Thing is, I have no idea how to test if it is working correctly. In addition I have no idea what files I need to generate (original pipeline used was mothur). Does this mean I need to re-run my data through the qiime pipeline before I can use the plugin? Is there any additional detail that can be provided about the process? Again, I have no experience with qiime and limited experience with programming (using calour). Is there a more straightforward explanation that can be provided for novice users?

Thank you in advance,

Pinging @serenejiang to see if they can help.

You should be able to run --help to work out what types are needed and then import your data (depends on what files you specifically have) to avoid re-analyzing (or you can re-analyize).

I apologize for my ignorance, but how do I ping someone? Is it just @name?

Yep, and I already have in my response. The @ sign will send a notification to whoever is named.

When you say run ‘–help’ I assume you mean in the command terminal? I am using a virtualbox. If I type --help in the terminal I get the error ’ bash: --help: command not found '. Obviously I am doing something wrong here.

Ah yes, sorry run --help on the qiime2 dsfdr command created by the q2-dsfdr plugin. It should tell you a bit more about what it needs (I don’t have it installed myself, so I’m not certain what you’ll find).

Per gethub

source activate
pip install git+https://github.com/biocore/dsFDR.git
pip install git+https://github.com/serenejiang/q2_dsfdr.git
qiime dev refresh-cache

Here is what I get

(qiime2-2018.4) qiime2@qiime2core2018-4:~ source activate qiime2-2018.4 (qiime2-2018.4) qiime2@qiime2core2018-4:~ pip install git+https://github.com/biocore/dsFDR.git
Collecting git+https://github.com/biocore/dsFDR.git
Cloning https://github.com/biocore/dsFDR.git to /tmp/pip-req-build-db2t8gun
Requirement already satisfied (use --upgrade to upgrade): dsfdr==0.1.0.dev0 from git+https://github.com/biocore/dsFDR.git in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages
Requirement already satisfied: numpy in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages (from dsfdr==0.1.0.dev0) (1.12.1)
Requirement already satisfied: scipy in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages (from dsfdr==0.1.0.dev0) (0.19.1)
Building wheels for collected packages: dsfdr
Running setup.py bdist_wheel for dsfdr … done
Stored in directory: /tmp/pip-ephem-wheel-cache-zcxwbikm/wheels/20/b3/60/a91f8402ebae3ea3b5f08429fa3b8ef867d0410e49dd350789
Successfully built dsfdr
(qiime2-2018.4) qiime2@qiime2core2018-4:~ pip install git+https://github.com/serenejiang/q2_dsfdr.git Collecting git+https://github.com/serenejiang/q2_dsfdr.git Cloning https://github.com/serenejiang/q2_dsfdr.git to /tmp/pip-req-build-sn4ehg5u Requirement already satisfied (use --upgrade to upgrade): q2-dsfdr==0.0.1 from git+https://github.com/serenejiang/q2_dsfdr.git in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages Building wheels for collected packages: q2-dsfdr Running setup.py bdist_wheel for q2-dsfdr ... done Stored in directory: /tmp/pip-ephem-wheel-cache-es1rw2pq/wheels/df/9b/19/60d6402d1710ea89664012e92d31f233354e14d6a478b04fc6 Successfully built q2-dsfdr (qiime2-2018.4) qiime2@qiime2core2018-4:~ ^C
(qiime2-2018.4) qiime2@qiime2core2018-4:~$

So is the qiime2 dsfdr command created by the q2-dsfdr plugin?

Hi, Patrick. You could take a look at this dsfdr demo, which including datasets, notebooks and outpout to show you how to use dsfdr as standalone package. You could then use the datasets provided here as input into the q2_dsfdr and check whether you could get similar output. As long as you could get a biom table and a mapping file, regardless of which pipeline you used earlier, you should be able to use the q2_dsfdr plugin.

1 Like

I apologize, but I think I forgot a section in my previous comment.

source activate qiime2-2018.4
pip install git+https://github.com/biocore/dsFDR.git
pip install git+https://github.com/serenejiang/q2_dsfdr.git
qiime dev refresh-cache

(qiime2-2018.4) qiime2@qiime2core2018-4:~ source activate qiime2-2018.4 (qiime2-2018.4) qiime2@qiime2core2018-4:~ pip install git+https://github.com/biocore/dsFDR.git
Collecting git+https://github.com/biocore/dsFDR.git
Cloning https://github.com/biocore/dsFDR.git to /tmp/pip-req-build-db2t8gun
Requirement already satisfied (use --upgrade to upgrade): dsfdr==0.1.0.dev0 from git+https://github.com/biocore/dsFDR.git in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages
Requirement already satisfied: numpy in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages (from dsfdr==0.1.0.dev0) (1.12.1)
Requirement already satisfied: scipy in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages (from dsfdr==0.1.0.dev0) (0.19.1)
Building wheels for collected packages: dsfdr
Running setup.py bdist_wheel for dsfdr … done
Stored in directory: /tmp/pip-ephem-wheel-cache-zcxwbikm/wheels/20/b3/60/a91f8402ebae3ea3b5f08429fa3b8ef867d0410e49dd350789
Successfully built dsfdr
(qiime2-2018.4) qiime2@qiime2core2018-4:~ pip install git+https://github.com/serenejiang/q2_dsfdr.git Collecting git+https://github.com/serenejiang/q2_dsfdr.git Cloning https://github.com/serenejiang/q2_dsfdr.git to /tmp/pip-req-build-sn4ehg5u Requirement already satisfied (use --upgrade to upgrade): q2-dsfdr==0.0.1 from git+https://github.com/serenejiang/q2_dsfdr.git in ./miniconda/envs/qiime2-2018.4/lib/python3.5/site-packages Building wheels for collected packages: q2-dsfdr Running setup.py bdist_wheel for q2-dsfdr ... done Stored in directory: /tmp/pip-ephem-wheel-cache-es1rw2pq/wheels/df/9b/19/60d6402d1710ea89664012e92d31f233354e14d6a478b04fc6 Successfully built q2-dsfdr (qiime2-2018.4) qiime2@qiime2core2018-4:~

I am unsure how to get to the qiime2 dsfdr command created by the q2-dsfdr plugin.

Thank you so much! This is exactly what I was looking for (not sure how I over looked it)! Just to make sure, is the deblur-feature-table.biom file the output?

Hi Patrick, dblr_dsfdr.txt is the output file, deblur-feature-table.biom and metadata_rare2k.txt are the input files. In fact, I also have a demo for the usage for the q2-dsfdr. You could simply follow this step by step. Demonstrated data are available here (the same as used in the demo for standalone package).

1 Like

Hi @serenejiang,

I am also having an issue with the scikit-bio module. I keep getting an error that it is not compatible with phython 3.6. Do you know of an older version which is compatible?

**Update
If I create the environment using python 3.6 then it works, but subsequently the gneiss.util doesn’t (requires 3.5 as well).

Update and Fix- With a follow up question

Fix-
Replace ‘conda create -n dsfdr python=3.5 numpy scipy jupyter scikit-learn scikit-bio statsmodels’ with
‘conda create -n dsfdr python=3 numpy scipy jupyter scikit-learn scikit-bio statsmodels’. This resolves the issues for scikit-bio and gneiss.util being compatible.

Biom, gneiss.util, cython and h5py need to be installed prior to running
pip install git+git://github.com/biocore/dsFDR.git

Biom- https://pypi.org/project/biom-format/

gneiss.util- https://pypi.org/project/gneiss/

cython- https://pypi.org/project/Cython/

h5py- pip install h5py

Remove %matplotlib inline from In [1]:

Ran perfect (same results as demo) from there.

Hi @pmaes,
Sounds like everything is running okay — could you please confirm and/or clarify what the ongoing issue is?

Hi @Nicholas_Bokulich and @serenejiang,

Everything seems to working fine. I couldn’t figure out how to run the plugin in Qiime, but was able to run it in the stand alone python package (with the above mentioned steps). My question, whether being used in Qiime or the stand alone python package, is the role of the alpha= in the results = command. At first it appeared to be the threshold for the output file, that its when I left it at the default (0.01), the out put file contained only those OTUs with a p-value < 0.01, and when I set it at 0.05, only those OTUs with a p-value of < 0.05. I then decided I wanted to know the p-value for each OTU regardless of if it was significant or not so I set the alpha to 1.00 (assuming this would give a p-value for each OTU). Upon doing this I did in fact get a p-value for all OTUs, however it also returned p-values for some OTUs where p<0.05 that were not in the output when the alpha was set at 0.05. So I am confused as to how this portion of the command is influencing the p-value generation.
Thank you in advance

Hi, Patrick. The p-values you get from the output file are the raw p-values, i.e. the original p-values from the statistical tests without FDR correction. Note that our method does not provide FDR corrected p-value, because our way of controlling the FDR doesn’t required adjusted p-values, but based on estimated FDR and desired control level (cutoff alpha), it is guaranteed to control the FDR at the desired level. So all the OTUs you see in the output file were the OTUs found to be statistically significant after DS-FDR correction, but their p-values and test statistic were the uncorrected ones from the original tests.

1 Like

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.