Quality Filtering using Deblur Error

irvinng98 · January 22, 2023, 10:05am

Hello,

I'm trying to do quality filtering using the deblur method. However, when I run the qiime quality-filter q-score function, I get an error (see below).

- Version of QIIME 2 you are running, and how it is installed (e.g. Virtualbox, conda, etc.)
2022.11 installed using conda on computing cluster
- What is the exact command or commands you ran?

#! /bin/bash -l

#PBS -l walltime=03:00:00,select=1:ncpus=12:mem=186gb
#PBS -N QC_Deblur_pipeline
#PBS -A <allocation>
######################################################################
# Load software environment
module load miniconda3

# Activate conda environment
source activate /home/<user>/miniconda3/envs/qiime2-2022.11

cd  /path/to/demux.qza

qiime quality-filter q-score \
 --i-demux demux.qza \
 --o-filtered-sequences filtered-sequences.qza \
 --o-filter-stats filter-stats.qza

- What is the exact error message? If you didn't run the command with the --verbose flag, please re-run and copy-and-paste the results.

Traceback (most recent call last):
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1054, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 920, in make_context
    self.parse_args(ctx, args)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/click/command.py", line 43, in parse_args
    return super().parse_args(ctx, args)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1610, in parse_args
    echo(ctx.get_help(), color=ctx.color)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 699, in get_help
    return self.command.get_help(self)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1298, in get_help
    self.format_help(ctx, formatter)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1331, in format_help
    self.format_options(ctx, formatter)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/click/command.py", line 157, in format_options
    for subcommand in self.list_commands(ctx):
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 92, in list_commands
    plugins = sorted(self._plugin_lookup)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
    import q2cli.core.cache
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 58, in __init__
    self._cache_dir = self._get_cache_dir()
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 85, in _get_cache_dir
    os.makedirs(cache_dir, exist_ok=True)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/irvinng/.config/q2cli'
Traceback (most recent call last):
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/bin/qiime", line 11, in <module>
    sys.exit(qiime())
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1651, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/click/core.py", line 1698, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 100, in get_command
    plugin = self._plugin_lookup[name]
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/commands.py", line 76, in _plugin_lookup
    import q2cli.core.cache
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 285, in <module>
    CACHE = DeploymentCache()
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 58, in __init__
    self._cache_dir = self._get_cache_dir()
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/site-packages/q2cli/core/cache.py", line 85, in _get_cache_dir
    os.makedirs(cache_dir, exist_ok=True)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/home/irvinng/miniconda3/envs/qiime2-2022.11/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/irvinng/.config/q2cli'

Thank you for your help!

jwdebelius · January 22, 2023, 4:38pm

Hi @irvinng98,

Welcome to the forum!

This is a new (to me error), but let's try to break it down together. With error messages, you start at the top for a general sense (usually there's a general failure but it looks like you didn't get one.) And then, we go down to the bottom and work form the bottom up.

So, if I start at the bottom, I get

Which tells me that there's an issue with reading/writing files. I'm not sure if teh issue is in your .config file, or if it's an issue in making other directories. (QIIME 2 writes temporary files during the archive building stage. Its a somewhat unique approach.)

So, how do you deal with it?

If it's your local machine, I'd modify your permissions. If it's a shared machine, I would still modify permissions for your local directory. I'd also talk to the sys admin about getting write permissions for the $TMPDIR variable (where ever that is on your system.)

Best,
Justine

irvinng98 · January 22, 2023, 9:24pm

I forgot to mention that when I try ' ls /home/irvinng/.config/q2cli' I see that this file doesn't exist. I read somewhere that q2cli is no longer stored in that directory? I've already changed the permissions on the config directory but to no avail.

jwdebelius · January 23, 2023, 2:26pm

Hi @irvinng98,

You'd need to run ls -al to see the directory, since it's hidden. (That's what that .config does. I'm not sure about it moving.

I again suspect it might be a tmp permissions issue, because I doubt your config file should need to change regularly. Does this happen when you just run

qiime

Have you recently updated your qiime config?

Best,
Justine

irvinng98 · January 24, 2023, 8:26pm

When I run

qiime

The following is returned (looks normal to me):

Usage: qiime [OPTIONS] COMMAND [ARGS]...

  QIIME 2 command-line interface (q2cli)
  --------------------------------------

  To get help with QIIME 2, visit https://qiime2.org.

  To enable tab completion in Bash, run the following command or add it to
  your .bashrc/.bash_profile:

      source tab-qiime

  To enable tab completion in ZSH, run the following commands or add them to
  your .zshrc:

      autoload -Uz compinit && compinit
      autoload bashcompinit && bashcompinit
      source tab-qiime

Options:
  --version   Show the version and exit.
  --help      Show this message and exit.

Commands:
  info                Display information about current deployment.
  tools               Tools for working with QIIME 2 files.
  dev                 Utilities for developers and advanced users.
  alignment           Plugin for generating and manipulating alignments.
  composition         Plugin for compositional data analysis.
  cutadapt            Plugin for removing adapter sequences, primers, and
                      other unwanted sequence from sequence data.
  dada2               Plugin for sequence quality control with DADA2.
  deblur              Plugin for sequence quality control with Deblur.
  demux               Plugin for demultiplexing & viewing sequence quality.
  diversity           Plugin for exploring community diversity.
  diversity-lib       Plugin for computing community diversity.
  emperor             Plugin for ordination plotting with Emperor.
  feature-classifier  Plugin for taxonomic classification.
  feature-table       Plugin for working with sample by feature tables.
  fragment-insertion  Plugin for extending phylogenies.
  gneiss              Plugin for building compositional models.
  longitudinal        Plugin for paired sample and time series analyses.
  metadata            Plugin for working with Metadata.
  phylogeny           Plugin for generating and manipulating phylogenies.
  quality-control     Plugin for quality control of feature and sequence data.
  quality-filter      Plugin for PHRED-based filtering and trimming.
  sample-classifier   Plugin for machine learning prediction of sample
                      metadata.
  taxa                Plugin for working with feature taxonomy annotations.
  vsearch             Plugin for clustering and dereplicating with vsearch.

When I run

ls -al /home/irvinng/.config/

I still don't see the q2cli file.

I also tried removing the entire environment and reinstalling qiime (using the instructions from the documentation). But to no avail.

jwdebelius · January 25, 2023, 3:02pm

Hi @irvinng98,

I'm wondering if part of the difference is that it looks like one was run in a torque job, and the other might have been run locally?

Just, as an experiment, could you try running a script to call QIIME in the torque system? I tried to set up a minimum job for it that should require like no resources.

#PBS -l walltime=00:15:00,select=1:ncpus=1:mem=4gb
#PBS -N QC_Deblur_pipeline
#PBS -A <allocation>
######################################################################
# Load software environment
module load miniconda3

# Activate conda environment
source activate /home/<user>/miniconda3/envs/qiime2-2022.11

qiime

Best,
Justine

irvinng98 · January 25, 2023, 6:37pm

You're absolutely right! I got the deblur stuff to work on local but when I submit it to the computing cluster, I get that error. Even just running

#PBS -l walltime=00:15:00,select=1:ncpus=1:mem=4gb
#PBS -N QC_Deblur_pipeline
#PBS -A <allocation>
######################################################################
# Load software environment
module load miniconda3

# Activate conda environment
source activate /home/<user>/miniconda3/envs/qiime2-2022.11

qiime

gave me the same error output. This is pretty worrying because I would like to run qiime in a torque job. Thoughts?

jwdebelius · January 25, 2023, 6:48pm

Hi @irvinng98,

This is going to depend on the system you're working with. My HPC has a module system, and I have to run source ~/.bashrc before I can get my conda environments to load.

You might also check the file system and see fi the config folder is present.

It might also make sense to reach out to others who use miniconda on your HPC if others have similar conda issues.

Best,
Justine

irvinng98 · January 25, 2023, 6:54pm

I've used miniconda on my HPC before and the current code is sufficient

#! /bin/bash -l

#PBS -l walltime=03:00:00,select=1:ncpus=1:mem=50gb
#PBS -N QC_Deblur_pipeline
#PBS -A <ALLOCATION>
#PBS -M <EMAIL>
#PBS -m abe


######################################################################

# Load software environment
module load miniconda3

# Activate conda environment
source activate /home/irvinng/miniconda3/envs/qiime2-2022.11

I do see the .config folder with the path: "/home/irvinng/.config" but the issue is I don't see the q2cli file/folder.

lizgehret · January 25, 2023, 10:06pm

Hey @irvinng98,

I suspect you're running into a permissions issue on your cluster, since you mentioned you're able to run this locally. If you haven't done so already, I'd recommend reaching out to your system admin who manages this cluster to see if they can modify these permissions for you (I'd share with them an overview of what you've tried to do on the cluster, the error message, and that it worked for you on your local machine). Hope this helps! If not, we'll keep digging

Cheers

system · February 26, 2023, 4:07am

This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.