Running The Moving Pictures Tutorial In Docker (2018.11)

Community Contributions Tutorials
1

Okay, so you have followed the official install guide for getting QIIME 2 up and running in Docker.... now what?!

Using the Moving Pictures Tutorial (2018.11) as a guide (we will just present commands here, please see the source tutorial for more detailed discussion) - we have two main options for running commands:

  • Option A - Single container
  • Option B - One container per command

Option A - Single Container

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        /bin/bash

After a moment, you should see something that looks like the following:

(qiime2-2018.11) I have no name!@aec08b34f1fb:/data$

Now you can start running the Moving Pictures commands one by one!

mkdir emp-single-end-sequences

wget \
    -O "sample-metadata.tsv" \
    "https://data.qiime2.org/2018.11/tutorials/moving-pictures/sample_metadata.tsv"

wget \
    -O "emp-single-end-sequences/sequences.fastq.gz" \
    "https://data.qiime2.org/2018.11/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz"

wget \
    -O "emp-single-end-sequences/barcodes.fastq.gz" \
    "https://data.qiime2.org/2018.11/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"

qiime tools import \
  --type EMPSingleEndSequences \
  --input-path emp-single-end-sequences \
  --output-path emp-single-end-sequences.qza

qiime demux emp-single \
  --i-seqs emp-single-end-sequences.qza \
  --m-barcodes-file sample-metadata.tsv \
  --m-barcodes-column BarcodeSequence \
  --o-per-sample-sequences demux.qza

qiime demux summarize \
  --i-data demux.qza \
  --o-visualization demux.qzv

qiime dada2 denoise-single \
  --i-demultiplexed-seqs demux.qza \
  --p-trim-left 0 \
  --p-trunc-len 120 \
  --o-representative-sequences rep-seqs-dada2.qza \
  --o-table table-dada2.qza \
  --o-denoising-stats stats-dada2.qza

qiime metadata tabulate \
  --m-input-file stats-dada2.qza \
  --o-visualization stats-dada2.qzv

mv rep-seqs-dada2.qza rep-seqs.qza &&  mv table-dada2.qza table.qza

qiime feature-table summarize \
  --i-table table.qza \
  --o-visualization table.qzv \
  --m-sample-metadata-file sample-metadata.tsv

qiime feature-table tabulate-seqs \
  --i-data rep-seqs.qza \
  --o-visualization rep-seqs.qzv

qiime phylogeny align-to-tree-mafft-fasttree \
  --i-sequences rep-seqs.qza \
  --o-alignment aligned-rep-seqs.qza \
  --o-masked-alignment masked-aligned-rep-seqs.qza \
  --o-tree unrooted-tree.qza \
  --o-rooted-tree rooted-tree.qza

qiime diversity core-metrics-phylogenetic \
  --i-phylogeny rooted-tree.qza \
  --i-table table.qza \
  --p-sampling-depth 1109 \
  --m-metadata-file sample-metadata.tsv \
  --output-dir core-metrics-results

qiime diversity alpha-group-significance \
  --i-alpha-diversity core-metrics-results/faith_pd_vector.qza \
  --m-metadata-file sample-metadata.tsv \
  --o-visualization core-metrics-results/faith-pd-group-significance.qzv

qiime diversity alpha-group-significance \
  --i-alpha-diversity core-metrics-results/evenness_vector.qza \
  --m-metadata-file sample-metadata.tsv \
  --o-visualization core-metrics-results/evenness-group-significance.qzv

qiime diversity beta-group-significance \
  --i-distance-matrix core-metrics-results/unweighted_unifrac_distance_matrix.qza \
  --m-metadata-file sample-metadata.tsv \
  --m-metadata-column BodySite \
  --o-visualization core-metrics-results/unweighted-unifrac-body-site-significance.qzv \
  --p-pairwise

qiime diversity beta-group-significance \
  --i-distance-matrix core-metrics-results/unweighted_unifrac_distance_matrix.qza \
  --m-metadata-file sample-metadata.tsv \
  --m-metadata-column Subject \
  --o-visualization core-metrics-results/unweighted-unifrac-subject-group-significance.qzv \
  --p-pairwise

qiime emperor plot \
  --i-pcoa core-metrics-results/unweighted_unifrac_pcoa_results.qza \
  --m-metadata-file sample-metadata.tsv \
  --p-custom-axes DaysSinceExperimentStart \
  --o-visualization core-metrics-results/unweighted-unifrac-emperor-DaysSinceExperimentStart.qzv

qiime emperor plot \
  --i-pcoa core-metrics-results/bray_curtis_pcoa_results.qza \
  --m-metadata-file sample-metadata.tsv \
  --p-custom-axes DaysSinceExperimentStart \
  --o-visualization core-metrics-results/bray-curtis-emperor-DaysSinceExperimentStart.qzv

qiime diversity alpha-rarefaction \
  --i-table table.qza \
  --i-phylogeny rooted-tree.qza \
  --p-max-depth 4000 \
  --m-metadata-file sample-metadata.tsv \
  --o-visualization alpha-rarefaction.qzv

wget \
    -O "gg-13-8-99-515-806-nb-classifier.qza" \
    "https://data.qiime2.org/2018.11/common/gg-13-8-99-515-806-nb-classifier.qza"

qiime feature-classifier classify-sklearn \
  --i-classifier gg-13-8-99-515-806-nb-classifier.qza \
  --i-reads rep-seqs.qza \
  --o-classification taxonomy.qza

qiime metadata tabulate \
  --m-input-file taxonomy.qza \
  --o-visualization taxonomy.qzv

qiime taxa barplot \
  --i-table table.qza \
  --i-taxonomy taxonomy.qza \
  --m-metadata-file sample-metadata.tsv \
  --o-visualization taxa-bar-plots.qzv

qiime feature-table filter-samples \
  --i-table table.qza \
  --m-metadata-file sample-metadata.tsv \
  --p-where "BodySite='gut'" \
  --o-filtered-table gut-table.qza

qiime composition add-pseudocount \
  --i-table gut-table.qza \
  --o-composition-table comp-gut-table.qza

qiime composition ancom \
  --i-table comp-gut-table.qza \
  --m-metadata-file sample-metadata.tsv \
  --m-metadata-column Subject \
  --o-visualization ancom-Subject.qzv

qiime taxa collapse \
  --i-table gut-table.qza \
  --i-taxonomy taxonomy.qza \
  --p-level 6 \
  --o-collapsed-table gut-table-l6.qza

qiime composition add-pseudocount \
  --i-table gut-table-l6.qza \
  --o-composition-table comp-gut-table-l6.qza

qiime composition ancom \
  --i-table comp-gut-table-l6.qza \
  --m-metadata-file sample-metadata.tsv \
  --m-metadata-column Subject \
  --o-visualization l6-ancom-Subject.qzv

When you're all done, simply type exit, hit Enter, and your container should exit and close. :whale2:

Option B - One container per command

This one looks similar to Option A, except here we prefix every command with the following:

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \

This will spin up a new container for each command, which is convenient for compartmentalizing each step.

Below are all of the commands from the Moving Pictures tutorial, with this docker prefix:

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        mkdir emp-single-end-sequences

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        wget \
            -O "sample-metadata.tsv" \
            "https://data.qiime2.org/2018.11/tutorials/moving-pictures/sample_metadata.tsv"

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        wget \
            -O "emp-single-end-sequences/sequences.fastq.gz" \
            "https://data.qiime2.org/2018.11/tutorials/moving-pictures/emp-single-end-sequences/sequences.fastq.gz"

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        wget \
            -O "emp-single-end-sequences/barcodes.fastq.gz" \
            "https://data.qiime2.org/2018.11/tutorials/moving-pictures/emp-single-end-sequences/barcodes.fastq.gz"

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime tools import \
          --type EMPSingleEndSequences \
          --input-path emp-single-end-sequences \
          --output-path emp-single-end-sequences.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime demux emp-single \
          --i-seqs emp-single-end-sequences.qza \
          --m-barcodes-file sample-metadata.tsv \
          --m-barcodes-column BarcodeSequence \
          --o-per-sample-sequences demux.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime demux summarize \
          --i-data demux.qza \
          --o-visualization demux.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime dada2 denoise-single \
          --i-demultiplexed-seqs demux.qza \
          --p-trim-left 0 \
          --p-trunc-len 120 \
          --o-representative-sequences rep-seqs-dada2.qza \
          --o-table table-dada2.qza \
          --o-denoising-stats stats-dada2.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime metadata tabulate \
          --m-input-file stats-dada2.qza \
          --o-visualization stats-dada2.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        mv rep-seqs-dada2.qza rep-seqs.qza &&  mv table-dada2.qza table.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime feature-table summarize \
          --i-table table.qza \
          --o-visualization table.qzv \
          --m-sample-metadata-file sample-metadata.tsv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime feature-table tabulate-seqs \
          --i-data rep-seqs.qza \
          --o-visualization rep-seqs.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime phylogeny align-to-tree-mafft-fasttree \
          --i-sequences rep-seqs.qza \
          --o-alignment aligned-rep-seqs.qza \
          --o-masked-alignment masked-aligned-rep-seqs.qza \
          --o-tree unrooted-tree.qza \
          --o-rooted-tree rooted-tree.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity core-metrics-phylogenetic \
          --i-phylogeny rooted-tree.qza \
          --i-table table.qza \
          --p-sampling-depth 1109 \
          --m-metadata-file sample-metadata.tsv \
          --output-dir core-metrics-results

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity alpha-group-significance \
          --i-alpha-diversity core-metrics-results/faith_pd_vector.qza \
          --m-metadata-file sample-metadata.tsv \
          --o-visualization core-metrics-results/faith-pd-group-significance.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity alpha-group-significance \
          --i-alpha-diversity core-metrics-results/evenness_vector.qza \
          --m-metadata-file sample-metadata.tsv \
          --o-visualization core-metrics-results/evenness-group-significance.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity beta-group-significance \
          --i-distance-matrix core-metrics-results/unweighted_unifrac_distance_matrix.qza \
          --m-metadata-file sample-metadata.tsv \
          --m-metadata-column BodySite \
          --o-visualization core-metrics-results/unweighted-unifrac-body-site-significance.qzv \
          --p-pairwise

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity beta-group-significance \
          --i-distance-matrix core-metrics-results/unweighted_unifrac_distance_matrix.qza \
          --m-metadata-file sample-metadata.tsv \
          --m-metadata-column Subject \
          --o-visualization core-metrics-results/unweighted-unifrac-subject-group-significance.qzv \
          --p-pairwise

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime emperor plot \
          --i-pcoa core-metrics-results/unweighted_unifrac_pcoa_results.qza \
          --m-metadata-file sample-metadata.tsv \
          --p-custom-axes DaysSinceExperimentStart \
          --o-visualization core-metrics-results/unweighted-unifrac-emperor-DaysSinceExperimentStart.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime emperor plot \
          --i-pcoa core-metrics-results/bray_curtis_pcoa_results.qza \
          --m-metadata-file sample-metadata.tsv \
          --p-custom-axes DaysSinceExperimentStart \
          --o-visualization core-metrics-results/bray-curtis-emperor-DaysSinceExperimentStart.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime diversity alpha-rarefaction \
          --i-table table.qza \
          --i-phylogeny rooted-tree.qza \
          --p-max-depth 4000 \
          --m-metadata-file sample-metadata.tsv \
          --o-visualization alpha-rarefaction.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        wget \
            -O "gg-13-8-99-515-806-nb-classifier.qza" \
            "https://data.qiime2.org/2018.11/common/gg-13-8-99-515-806-nb-classifier.qza"

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime feature-classifier classify-sklearn \
          --i-classifier gg-13-8-99-515-806-nb-classifier.qza \
          --i-reads rep-seqs.qza \
          --o-classification taxonomy.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime metadata tabulate \
          --m-input-file taxonomy.qza \
          --o-visualization taxonomy.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime taxa barplot \
          --i-table table.qza \
          --i-taxonomy taxonomy.qza \
          --m-metadata-file sample-metadata.tsv \
          --o-visualization taxa-bar-plots.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime feature-table filter-samples \
          --i-table table.qza \
          --m-metadata-file sample-metadata.tsv \
          --p-where "BodySite='gut'" \
          --o-filtered-table gut-table.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime composition add-pseudocount \
          --i-table gut-table.qza \
          --o-composition-table comp-gut-table.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime composition ancom \
          --i-table comp-gut-table.qza \
          --m-metadata-file sample-metadata.tsv \
          --m-metadata-column Subject \
          --o-visualization ancom-Subject.qzv

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime taxa collapse \
          --i-table gut-table.qza \
          --i-taxonomy taxonomy.qza \
          --p-level 6 \
          --o-collapsed-table gut-table-l6.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime composition add-pseudocount \
          --i-table gut-table-l6.qza \
          --o-composition-table comp-gut-table-l6.qza

docker run --rm -it \
    -u $(id -u):$(id -g) \
    -v $(pwd):/data \
    qiime2/core:2018.11 \
        qiime composition ancom \
          --i-table comp-gut-table-l6.qza \
          --m-metadata-file sample-metadata.tsv \
          --m-metadata-column Subject \
          --o-visualization l6-ancom-Subject.qzv

That is about all there is to it! Pretty convenient! :whale: :ship:

4 Likes
2

Using docker desktop for windows I got the error “invalid format” with the “docker run” code. This worked for me " docker run --interactive --tty qiime/core:2019.1 bash"

3

Thanks for sharing, but, there is something strange here:

qiime/core:2019.1 is not our docker image (but qiime2/core:2019.1 is). Also, the --interactive --tty portion of the command are the same flags we specified above (-it), just in their long form. Thanks!

4

Yes the image is qiime2/core:2019.1 sorry. It was this part of the code “u (id -u):(id -g)”. There was no flag -g option apparently in docker run. I’m new to docker so I apologise if I didn’t use the code correctly. But I found a solution and just wanted to share it

2 Likes