Using Bash commands within Qiime2 commands?

[Daniel_Zagal]

Hello,
I have sample sets from different locations that I would like to process in the same way, but separately. I usually make a feature table summary and take the highest feature count for alpha rarefaction and the select the sampling depth that allows me to keep 95% of my samples. It all works great, but... what if I wanted to do this for 100 sites?
Is it possible to use bash commands with an qiime2 command? I think it is not, but I have not found confirmation of this.
I would like to be able to do something along the lines of:

for folder in Dir
do
qiime feature-table summarize
--i-table $folder/feature_table.qza
--m-sample-metadata-file sample_metadata.txt
--o-visualization $folder/$folder_table_summary.qzv
done

If this was possible it means that I could run my whole qiime2 script recursively instead of doing it manually folder by folder.

Maybe there are better ways to do this? But how would one go about doing it in cli if absolutely necessary?

UPDATE (10/17/22): I got something to work and I figured I would post it in case anybody is curious or has feedback as to how to improve this or point out any potential or incipient issues. Here is the script:

#Making Data Subsets and Getting Core Metrics Recursively using BASH

#Use metadata columns and their values to stratify samples
#To also make subsets including all the samples for each metadata column,
#"All" and "Samples" are added to each values list, respectively.

for metadata-col-1 in All value_x value_y value_z;
do for metadata-col-2 in Samples value_q value_r value_s;

#Make directories for each data subset

do mkdir Path/$metadata-col-1_$metadata-col-2;

#Make Feature Tables for data subsets. There are three possible cases:

if [ "$metadata-col-1" == "All" ] && [ "$metadata-col-2" == "Samples" ];
then echo "THIS IS ALL SAMPLES";

#Make All Samples Feature Table (You probably already have this table, so this step may be redundant)

qiime feature-table filter-samples
--i-table Input_Files/clean_feature_table.qza
--m-metadata-file Input_Files/sample_metadata.txt
--o-filtered-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza ;
elif [ "$metadata-col-1" == "All" ];
then echo "THIS IS ALL metadata-col-1 $metadata-col-2";

#Make All metadata-col-1 for each metadata-col-2 Feature Tables

qiime feature-table filter-samples
--i-table Input_Files/clean_feature_table.qza
--m-metadata-file Input_Files/sample_metadata.txt
--p-where "metadata-col-2 = '$metadata-col-2'"
--o-filtered-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza ;
elif [ "$metadata-col-2" == "Samples" ];
then echo "THIS IS $metadata-col-1 ALL metadata-col-2";

#Make All metadata-col-2 for each metadata-col-1 Feature Tables

qiime feature-table filter-samples
--i-table Input_Files/clean_feature_table.qza
--m-metadata-file Input_Files/sample_metadata.txt
--p-where "metadata-col-1 = '$metadata-col-1'"
--o-filtered-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza ;
else echo "THIS IS $metadata-col-1 $metadata-col-2";

#Make each metadata-col-1 for each metadata-col-2 Feature Tables

qiime feature-table filter-samples
--i-table Input_Files/clean_feature_table.qza
--m-metadata-file Input_Files/sample_metadata.txt
--p-where "metadata-col-1 = '$metadata-col-1' AND metadata-col-2 = '$metadata-col-2'"
--o-filtered-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza ;
fi;

#Make Feature Table Summaries

qiime feature-table summarize
--i-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza
--m-sample-metadata-file Input_Files/sample_metadata.txt
--o-visualization Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_table_summary.qzv ;

#Core Metrics: Alpha and Beta Diversity

qiime diversity core-metrics-phylometadata-col-1etic
--i-phylogeny Input_Files/rooted_tree.qza
--i-table Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_feature_table.qza
--p-sampling-depth 1000
--m-metadata-file Input_Files/sample_metadata.txt
--output-dir Path/$metadata-col-1_$metadata-col-2/$metadata-col-1_$metadata-col-2_core_metrics_results
done;
done

I hope this makes sense!

[llenzi (Luca)]

Hi @Daniel_Zagal,

I am not sure what would be the best way in your case, butyour solution should work!
To make it work, you have to create a bash script whith a forloop, within the loop you can call any qiime commands you would like.

Be sure to have the qiime environment already active in your terminal before running the script.
Let me know if you have problem and good luck.

Cheers
Luca

PS, I changed the category for your question, simply because the developer categories is only related to develop qiime2 plug ins really!

[gregcaporaso]

Hi @Daniel_Zagal,
That should be possible. However, note that QIIME 2 has a Python 3 API in addition to the command line interface.

You can see the usage of the Python 3 API discussed here and illustrated in the "Multiple Interface Edition" of the Moving Pictures tutorial or in the Cancer Microbiome Intervention tutorial. In both cases, select Python 3 API from the Interfaces dropdown box:

This is what I use when I try to do work like what you're describing - I find it really convenient for this, though I'm definitely a better Python programmer than a bash programmer. I have an example of using this in a for loop (to loop over different distance metrics that I'm using to create plots) in this notebook.

Hope this helps - good luck!

[jwdebelius (Justine Debelius)]

Hi @Daniel_Zagal,

Let me add onto @llenzi and @gregcaporaso's solution with another one!

Sometimes, I use snakemake to wrap my QIIME 2 commands, especially when I'm mixing with other types of scripts. I have an example on GitHub here as well: GitHub - jwdebelius/avengers-assemble

You do pay an IO penalty with this, since you read and write to file at each step, but depending on your system, its also theoretically possible to start these as jobs. (Please dont ask me how, though).

So, one more option to consider!

Best,
Justine

[Daniel_Zagal]

Hi @ Ilenzi,
Thanks for changing the category. I was not sure if I was posting in the right place.

[Daniel_Zagal]

Hi Greg,

Thanks for your thorough response. I am aware of the Python API and I know enough python to understand how powerful it can be. However, I am much more familiar with using qiime2 in bash and I don't think I want to got through the learning curve again at the moment. My programming skills are somewhat rudimentary and I do not have the time to switch gears right now. But I think I am close to getting my bash script to work! Will post it when it is ready.

Thanks again!

[Daniel_Zagal]

Hi @jwdebelius,

I didn't know about snakemake. That seems cool! I will definitely check it out at some point.

Thanks for the info!