Callable does not have parameter(s)

Developer Discussion
1

Hi everyone,
I’m writing a QIIME2 plugin for MetaPhlAn2, but when I do qiime dev refresh-cache I got the following error:

QIIME is caching your current deployment for improved performance. This may take a few moments and should only happen once per deployment.
Traceback (most recent call last):
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/bin/qiime", line 6, in <module>
    sys.exit(q2cli.__main__.qiime())
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/dev.py", line 27, in refresh_cache
    import q2cli.cache
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/cache.py", line 305, in <module>
    CACHE = DeploymentCache()
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/cache.py", line 61, in __init__
    self._state = self._get_cached_state(refresh=refresh)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/cache.py", line 107, in _get_cached_state
    self._cache_current_state(current_requirements)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/cache.py", line 200, in _cache_current_state
    state = self._get_current_state()
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/q2cli/cache.py", line 238, in _get_current_state
    plugin_manager = qiime2.sdk.PluginManager()
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py", line 44, in __new__
    self._init()
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/plugin_manager.py", line 58, in _init
    plugin = entry_point.load()
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg/pkg_resources/__init__.py", line 2258, in load
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/setuptools-27.2.0-py3.5.egg/pkg_resources/__init__.py", line 2264, in resolve
  File "/home/qiime2/Desktop/metaphlan2_test_plugin/lib/python3.5/site-packages/metaphlan2-2.6.0-py3.5.egg/metaphlan2/plugin_setup.py", line 52, in <module>
    description=('MetaPhlAn is a computational tool for profiling the composition of microbial communities (Bacteria, Archaea, Eukaryotes, and Viruses) from metagenomic shotgun sequencing data with species level resolution')
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/plugin/plugin.py", line 209, in register_function
    output_descriptions)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/sdk/action.py", line 344, in _init
    output_descriptions)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/type/signature.py", line 98, in __init__
    parameter_descriptions, output_descriptions)
  File "/home/qiime2/miniconda/envs/qiime2-2017.6/lib/python3.5/site-packages/qiime2/core/type/signature.py", line 183, in _parse_signature
    % (list(inputs) + list(parameters)))
TypeError: Callable does not have parameter(s): ['raw-data']

This is the code of the plugin:

from qiime2.plugin import Plugin
from q2_types.per_sample_sequences import SequencesWithQuality
from q2_types.feature_table import FeatureTable, Frequency
import metaphlan2


plugin = Plugin(
    name='metaphlan2',
    version=metaphlan2.__version__,
    website='http://segatalab.cibio.unitn.it/tools/metaphlan2/',
    user_support_text='[email protected]',
    package='metaphlan2',
    citation_text=('Truong DT, Franzosa EA, Tickle TL, Scholz M, Weingart G, Pasolli E, Tett A, Huttenhower C, Segata N. MetaPhlAn2 for enhanced metagenomic taxonomic profiling. Nature Methods. 2015 Oct 1;12(10):902-3'),
    description=('MetaPhlAn is a computational tool for profiling the composition of microbial communities (Bacteria, Archaea, Eukaryotes, and Viruses) from metagenomic shotgun sequencing data with species level resolution'),
    short_description='MetaPhlAn2 for enhanced metagenomic taxonomic profiling'
)

plugin.methods.register_function(
    function=metaphlan2.metaphlan2_function,

    inputs={'raw-data': SequencesWithQuality },
    input_descriptions={'raw-data': ('metagenomic shotgun sequencing data')},

    parameters={},
    parameter_descriptions={},

    outputs=[('tsv-table', FeatureTable[Frequency])],
    output_descriptions={'tsv-table': 'TAB-separated text file containing relative abundances of the species found in the input'},

    name='MetaPhlAn2 taxonomic profiling',
    description=('MetaPhlAn is a computational tool for profiling the composition of microbial communities (Bacteria, Archaea, Eukaryotes, and Viruses) from metagenomic shotgun sequencing data with species level resolution')
)

I searched and I couldn’t find anyone reporting the same issue, but I apologize in advance if this is a trivial question.
I’m not sure what I’m doing wrong here and any help will be much appreciated.

Many thanks,
Francesco

3 Likes
2

Hey @fasnicar!

Thanks for the PR (which is looks like you’re using based on that traceback).

I think the issue is the key to your inputs dictionary. Python doesn’t let you use - in an identifier, so likely what you need is raw_data. I checked out the bitbucket repo that was linked in your website argument, but I wasn’t able to find metaphlan2.metaphlan2_function, so if switching the “-” to a “_” doesn’t fix this, would you mind posting a link to that source?

Unrelated, but your SequencesWithQuality, which is set as the semantic type for raw-data, is typically found as SampleData[SequencesWithQuality] let us know if that doesn’t make sense for MetaPhlAn2, but I think that’s probably what you’ll want instead.

I’m super excited to see how this plugin develops, so keep in touch!

2 Likes
3

Hi @ebolyen!

No problem, it's really a minor thing that I encountered while testing the MetaPhlAn2 plugin.

Sorry about that, I didn't push the new code yet as I'm doing some other testing because MetaPhlAn2 was using Python2 originally, and I did some updates for using Python3 instead.

Many thanks for the suggestion. I'm still modifying the MetaPhlAn2 plugin code with some help from @mortonjt, I think I'll have some updates by the end of this week.

Many thanks for the help,
Francesco

1 Like
4

Hi all,
I have some updates and still some doubts that I hope you can help me sort out!

The code for MetaPhlAn2 that includes the new scripts for the QIIME2 plugin is in the qiime2 branch (https://bitbucket.org/biobakery/metaphlan2/overview).

So far I’m able to register the metaphlan2 plugin into QIIME2 with the “profile_single_fastq” and “profile_paired_fastq” functions.

I’m now testing the plugin, and I managed to run it, but I’m not sure I’m doing it in the right way!

I’m using a couple of .fastq files I have from a shotgun sample. I generated the .qza file and I noticed that when running the plugin the input seems to have folder-like structure that contains the .fastq compressed with gzip.

Is this what I should expect from a .qza input file? Will it be always a folder-like containing (potentially) more than one gzip compressed file or could it be organized in other ways?

Many thanks,
Francesco

6

Hi @fasnicar,

Sorry for the delayed response (and the very long reply)!

Awesome!

QZA (QIIME Zipped Artifact) files are really just zip files and they can contain virtually anything (while keeping track of provenance and other metadata). I would encourage you to unzip one of these and look at the insides if you haven't already (it isn't too complicated).

If you use qiime tools peek on a .qza file it will tell you what the format of the .qza is (you can also look at metadata.yaml in the root of the zip file).

To use your profile_single_fastq as an example:

def profile_single_fastq(raw_data: SingleLanePerSampleSingleEndFastqDirFmt, nproc: int=1) -> biom.Table:
    ...

SingleLanePerSampleSingleEndFastqDirFmt is a directory format (like what is found in the /data/ directory, and biom.Table of course a plain Python object. Both of these are what we call a view type. And QIIME 2 will invoke what are called transformers to convert what is found inside a .qza into what raw_data is asking for. It will also convert the biom.Table instance into BIOMV210DirFmt for storage in the /data/ directory of the zip file.

As an aside: You can take advantage of formats to avoid creating objects in memory, so in your case, you might try using BIOMV210Format which is just a filepath to a biom file (turning BIOMV210Format into BIOMV210DirFmt is nearly instant as it's just a matter of naming the file for the /data/ directory).

The reason the outputs get transformed as well is that semantic types (such as FeatureData[Frequency]) are associated with canonical formats. So QIIME 2 will convert the output of a method to an appropriate format for the type it is saving the data as. We expect these formats to change over time and so you should really think in terms of the semantic type of your input or output (as the particular details of what that means on disk may change over time). In any case, once QIIME 2 knows the format of your /data/ directory in the .qza it can invoke transformers to convert FormatA into FormatB arbitrarily (this is also what happens when you import data).

To very crudely outline what the framework does, imagine this scenario:

  • A user has a file input.qza with some type: TypeA, and format LegacyFormat
  • They provide this file as input to a method which accepts TypeA but uses GreatNewFormat to represent the data and produces TypeB using format ResultFormat
  • QIIME 2 checks:
    • that the types match (they do!)
    • if the format is correct (it isn't)
      • looks for a transformer to convert LegacyFormat to GreatNewFormat
      • invokes this transformer which produces GreatNewFormat
  • QIIME 2 hands the instance of GreatNewFormat to the method and waits for the method to return something
  • The output of ResultFormat is captured, and QIIME 2 checks to see if that is the cannonical format for TypeB
    • It is (but QIIME 2 could have invoked a transformer to convert it if it wasn't)!
  • The ResultFormat becomes the /data/ directory for the TypeB output .qza
  • The provenance directory from the original input.qza is appended to output.qza
  • The method and parameters are recorded in the provenance directory for the output.qza
  • The user receives some finished output.qza file.

I hope this helps without being too confusing, but please feel free to ask for clarification or followup!

1 Like
8

Many thanks @ebolyen!
I’m sorry I didn’t replied earlier. I’ll have a closer look to your suggestions in the next days and I’ll try to finalize the MetaPhlAn2 plugin.
I’ll let you know when I’ll have an updated version of the plugin.

1 Like