Large dataset analysis computational power

Hi,

I’m going to analyse a really large dataset of 570 samples, paired-end sequencing.
Considering the quite large computational power required to run QIIME2, I was wondering if any of you has tips or suggestion for it.
I have already started the pipeline on the HPC, but I’m stuck to the first step, importing the files. Using the following command line:

qiime tools import
–type ‘SampleData[PairedEndSequencesWithQuality]’
–input-path casava-18-paired-end-demultiplexed
–input-format CasavaOneEightSingleLanePerSampleDirFmt
–output-path demux-paired-end.qza

I get an disk quota exceeded error, it seems that the temp files fill all my account space (1TB).

I am sure that the DADA2 step will be a problem too.

Do you have any suggestions?

Thanks in advance

1 Like

Hey there @mefistofele82!

I don’t agree with that! There are some steps in a typical QIIME 2 analysis that need modest resources, but in general you can conduct most of an analysis on a modern laptop (YMMV).

What is the location of your tmp dir? On most HPC clusters it is configured to a shared space, not as part of your account’s allocation. Can you run the following and share here:

env

Thanks!

Hi,
thanks for your reply.

Considering that the samples are from 7 different sequencing, and DADA2 should analyse one sequencing per time, I was planning to analyse each sequencing separately and then merge the results. This should be also the right approach for the DADA2 denoising step, right?

Here following the env command result.

MKLROOT=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mkl
MANPATH=/usr/local/Cluster-Apps/intel/2017.4/documentation_2017/en/debugger//gdb-ia/man/:/usr/local/Cluster-                           Apps/intel/2017.4/documentation_2017/en/debugger//gdb-mic/man/:/usr/local/Cluster-Apps/intel/2017.4/document                           ation_2017/en/debugger//gdb-igfx/man/:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.19                           6/linux/mpi/man:/usr/local/Cluster-Apps/intel/2017.4/man/common:/usr/local/software/global/man:/usr/local/Cl                           uster-Apps/turbovnc/2.0.1/man:/usr/local/software/slurm/current/share/man:/usr/share/man:
VIRTUALENVWRAPPER_SCRIPT=/usr/bin/virtualenvwrapper.sh
GUESTFISH_INIT=\e[1;34m
HOSTNAME=login-e-9
I_MPI_F77=ifort
IPPROOT=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/ipp
[email protected]
TERM=xterm
SHELL=/bin/bash
SLURM_MPI_TYPE=pmi2
HISTSIZE=1000
SOCKS_PROXY_SERVER=10.143.100.11
GDBSERVER_MIC=/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/gdb/targets/mic/bin/gdbserver
SSH_CLIENT=82.10.154.124 54473 22
CONDA_SHLVL=1
LIBRARY_PATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/lib/intel64_                           lin:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/../tbb/lib/intel64_li                           n/gcc4.4:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/ipp/lib/intel64:/usr/                           local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64/gcc4.7:/usr/local/C                           luster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin/gcc4.7:/usr/local/Clus                           ter-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/usr/local/Cluster-A                           pps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin
PERL5LIB=/home/rs2012/perl5/lib/perl5:
CONDA_PROMPT_MODIFIER=(base)
VGL_BINDIR=/usr/local/Cluster-Apps/vgl/2.5.1/64/bin
QTDIR=/usr/lib64/qt-3.3
QTINC=/usr/lib64/qt-3.3/include
PERL_MB_OPT=--install_base /home/rs2012/perl5
MIC_LD_LIBRARY_PATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/ipp/lib/mi                           c:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/mic:/usr/local/Clust                           er-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin_mic:/usr/local/Cluster-App                           s/intel/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin_mic:/usr/local/Cluster-Apps                           /intel/2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin_mic:/usr/local/Cluster-Apps/intel                           /2017.4/compilers_and_libraries_2017.4.196/linux/mpi/mic/lib:/usr/local/Cluster-Apps/intel/2017.4/compilers_                           and_libraries_2017.4.196/linux/compiler/lib/mic
I_MPI_OFA_TRANSLATION_CACHE=0
SSH_TTY=/dev/pts/39
QT_GRAPHICSSYSTEM_CHECKED=1
SOCKS_PROXY_NETWORK=131.111.0.0/16
USER=rs2012
LD_LIBRARY_PATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/lib/intel                           64_lin:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/../tbb/lib/intel64                           _lin/gcc4.4:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/ipp/lib/intel64:/u                           sr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64/gcc4.7:/usr/loca                           l/Cluster-Apps/intel/2017.4/debugger_2017/iga/lib:/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/libipt/                           intel64/lib:/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib:/u                           sr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin/gcc4.7:/usr/                           local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin:/usr/local/Clus                           ter-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64:/usr/local/Cluster-Apps/                           intel/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin:/usr/local/software/global/li                           b:/usr/local/Cluster-Apps/vgl/2.5.1/64/lib:/usr/local/software/slurm/current/lib
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;                           05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:                           *.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31                           :*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=0                           1;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01                           ;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;                           31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01                           ;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=                           01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4                           v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.fl                           c=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=                           01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=                           01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=                           01;36:*.spx=01;36:*.xspf=01;36:
CONDA_EXE=/home/rs2012/miniconda3/bin/conda
MIC_LIBRARY_PATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/mic:/                           usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb/lib/intel64_lin_mic:/usr/lo                           cal/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64_lin_mic:/usr/loc                           al/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin_mic:/usr/local/Clu                           ster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mpi/mic/lib:/usr/local/Cluster-Apps/intel/20                           17.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/mic
VGL_COMPRESS=proxy
CPATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/include:/usr/local/                           Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/ipp/include:/usr/local/Cluster-Apps/intel                           /2017.4/compilers_and_libraries_2017.4.196/linux/tbb/include:/usr/local/Cluster-Apps/intel/2017.4/compilers_                           and_libraries_2017.4.196/linux/mkl/include
GUESTFISH_PS1=\[\e[1;32m\]><fs>\[\e[0;31m\]
NLSPATH=/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/gdb/intel64_mic/share/locale/%l_%t/%N:/usr/local/                           Cluster-Apps/intel/2017.4/debugger_2017/gdb/intel64/share/locale/%l_%t/%N:/usr/local/Cluster-Apps/intel/2017                           .4/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/usr/local/Cluster-Apps/inte                           l/2017.4/compilers_and_libraries_2017.4.196/linux/compiler/lib/intel64/locale/%l_%t/%N
MAIL=/var/spool/mail/rs2012
PATH=/home/rs2012/miniconda3/bin:/home/rs2012/miniconda3/condabin:/usr/local/software/master/cmake/latest/bi                           n:/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/gdb/intel64_mic/bin:/usr/local/Cluster-Apps/intel/2017.                           4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/bin:/usr/local/Cluster-Apps/intel/2017.4/compilers_an                           d_libraries_2017.4.196/linux/bin/intel64:/usr/local/software/global/bin:/opt/singularity/bin:/usr/local/Clus                           ter-Apps/singularity/images:/usr/local/Cluster-Apps/vgl/2.5.1/64/bin:/usr/local/Cluster-Apps/turbovnc/2.0.1/                           bin:/usr/local/software/slurm/current/sbin:/usr/local/software/slurm/current/bin:/usr/lib64/qt-3.3/bin:/home                           /rs2012/perl5/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/dell/srvadmin/bin:.:/home/rs2012/bi                           n
CONDA_PREFIX=/home/rs2012/miniconda3
TBBROOT=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/tbb
PWD=/home/rs2012
_LMFILES_=/usr/share/Modules/modulefiles/dot:/usr/local/software/modulefiles/slurm:/usr/local/Cluster-Config                           /modulefiles/turbovnc/2.0.1:/usr/local/Cluster-Config/modulefiles/vgl/2.5.1/64:/usr/local/Cluster-Config/mod                           ulefiles/singularity/current:/usr/local/Cluster-Config/modulefiles/rhel7/global:/usr/local/Cluster-Config/mo                           dulefiles/intel/compilers/2017.4:/usr/local/Cluster-Config/modulefiles/intel/mkl/2017.4:/usr/local/Cluster-C                           onfig/modulefiles/intel/impi/2017.4/intel:/usr/local/Cluster-Config/modulefiles/intel/libs/idb/2017.4:/usr/l                           ocal/Cluster-Config/modulefiles/intel/libs/tbb/2017.4:/usr/local/Cluster-Config/modulefiles/intel/libs/ipp/2                           017.4:/usr/local/Cluster-Config/modulefiles/intel/libs/daal/2017.4:/usr/local/Cluster-Config/modulefiles/int                           el/bundles/complib/2017.4:/usr/local/software/modulefiles/cmake/latest:/usr/local/Cluster-Config/modulefiles                           /rhel7/default-peta4
GDB_CROSS=/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/gdb/intel64_mic/bin/gdb-mic
LANG=en_GB.UTF-8
MODULEPATH=/usr/share/Modules/modulefiles:/etc/modulefiles:/usr/local/software/spack/current/share/spack/tcl                           /linux-rhel7-x86_64:/usr/local/software/modulefiles:/usr/local/Cluster-Config/modulefiles
LOADEDMODULES=dot:slurm:turbovnc/2.0.1:vgl/2.5.1/64:singularity/current:rhel7/global:intel/compilers/2017.4:                           intel/mkl/2017.4:intel/impi/2017.4/intel:intel/libs/idb/2017.4:intel/libs/tbb/2017.4:intel/libs/ipp/2017.4:i                           ntel/libs/daal/2017.4:intel/bundles/complib/2017.4:cmake/latest:rhel7/default-peta4
KDEDIRS=/usr
GUESTFISH_OUTPUT=\e[0m
I_MPI_F90=ifort
PYTHONDONTWRITEBYTECODE=1
I_MPI_CC=icc
DAALROOT=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal
MPM_LAUNCHER=/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/mpm/mic/bin/start_mpm.sh
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
HISTCONTROL=ignoredups
SOCKS_PROXY_PORT=1081
INTEL_PYTHONHOME=/usr/local/Cluster-Apps/intel/2017.4/debugger_2017/python/intel64/
SHLVL=1
HOME=/home/rs2012
I_MPI_CXX=icpc
_VIRTUALENVWRAPPER_API= mkvirtualenv rmvirtualenv lsvirtualenv showvirtualenv workon add2virtualenv cdsitepa                           ckages cdvirtualenv lssitepackages toggleglobalsitepackages cpvirtualenv setvirtualenvproject mkproject cdpr                           oject mktmpenv wipeenv allvirtualenv
I_MPI_FC=ifort
PERL_LOCAL_LIB_ROOT=:/home/rs2012/perl5
CONDA_PYTHON_EXE=/home/rs2012/miniconda3/bin/python
COSMOHOST=darwin
LOGNAME=rs2012
QTLIB=/usr/lib64/qt-3.3/lib
CVS_RSH=ssh
CLASSPATH=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/daal/lib/daal.jar:/u                           sr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mpi/intel64/lib/mpi.jar
XDG_DATA_DIRS=/home/rs2012/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/shar                           e:/usr/share
SSH_CONNECTION=82.10.154.124 54473 128.232.224.46 22
MODULESHOME=/usr/share/Modules
CONDA_DEFAULT_ENV=base
OMP_NUM_THREADS=1
LESSOPEN=||/usr/bin/lesspipe.sh %s
INFOPATH=/usr/local/Cluster-Apps/intel/2017.4/documentation_2017/en/debugger//gdb-ia/info/:/usr/local/Cluste                           r-Apps/intel/2017.4/documentation_2017/en/debugger//gdb-mic/info/:/usr/local/Cluster-Apps/intel/2017.4/docum                           entation_2017/en/debugger//gdb-igfx/info/
ACLOCAL_PATH=/usr/local/software/master/cmake/latest/share/aclocal
CMAKE_PREFIX_PATH=/usr/local/software/master/cmake/latest/
QT_PLUGIN_PATH=/usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
GUESTFISH_RESTORE=\e[0m
PERL_MM_OPT=INSTALL_BASE=/home/rs2012/perl5
I_MPI_ROOT=/usr/local/Cluster-Apps/intel/2017.4/compilers_and_libraries_2017.4.196/linux/mpi
BASH_FUNC_module()=() {  eval `/usr/bin/modulecmd bash $*`;
 if [[ $* == *"load "* ]]; then
 echo "$*:$USER:${SLURM_JOB_ID:-login}" | systemd-cat -t user_module_cmd;
 fi
}
_=/usr/bin/env

Yep!

Thanks! I don’t see a TMPDIR env var here - it is very likely that you’re filling up a shared tmp dir, rather than your home directory. Check with your sysadmin to see where your temp dir is, and what their expectation of usage is.