Getting a results artifact from a pipeline

jwdebelius · August 11, 2021, 12:04am

Hi all,

I'm struggling specific with a pipeline issue. I've got a ffunction that I think is similar to the core metrics function following the same syntax:

github.com

jwdebelius/q2-sidle/blob/memory-refactor/q2_sidle/_pipelines.py

def sidle_reconstruction(ctx, 
                         region, 
                         kmer_map, 
                         regional_alignment, 
                         regional_table, 
                         reference_taxonomy, 
                         min_counts=1000,
                         database='none', 
                         define_missing='merge',
                         block_size=10000,
                         debug=False, 
                         n_workers=1,
                         client_address=None,
                         ):
    """
    A pipeline to reconstruct the full data
    """
    reconstruct_database = ctx.get_action('sidle', 'reconstruct_database')
    reconstruct_counts = ctx.get_action('sidle', 'reconstruct_counts')
    reconstruct_taxonomy = ctx.get_action('sidle', 'reconstruct_taxonomy')

This file has been truncated. show original

The function gets registered as a plugin, theoretically.

github.com

jwdebelius/q2-sidle/blob/c41571706c217e46c7fbd07b1fcb5367339c23e1/q2_sidle/plugin_setup.py#L479-L570


# plugin.pipelines.register_function(
#     function=q2_sidle.sidle_reconstruction,
#     name="A pipeline to reconstruct the database, count table, and taxonomy",
#     description=("A pipeline to reconstruct a database, count table, and "
#                  "taxonomy"),
#     inputs={'kmer_map': List[FeatureData[KmerMap]],
#             'regional_alignment': List[FeatureData[KmerAlignment]],
#             'regional_table': List[FeatureTable[Frequency]],
#             'reference_taxonomy': FeatureData[Taxonomy],
#             },
#     outputs=[('database_map', FeatureData[SidleReconstruction]),
#              ('database_summary', FeatureData[ReconstructionSummary]),
#              ('reconstructed_taxonomy', FeatureData[Taxonomy]),
#              ],
#     parameters={'region': List[Str],
#                 'min_counts': Int % Range(0, None),
#                 'database': Str % Choices('none', 'greengenes', 'silva'),
#                 'define_missing': Str % Choices('merge', 'inherit', 'ignore'),
#                 'block_size': Int,
#                 'n_workers': Int % Range(1, None),

This file has been truncated. show original

I've got a test with synthetic data:

github.com

jwdebelius/q2-sidle/blob/c41571706c217e46c7fbd07b1fcb5367339c23e1/q2_sidle/tests/test_plugin_setup.py#L142-L167


# def test_sidle_reconstruction(self):
#     mapping, summary, counts, taxonomy = sidle.sidle_reconstruction(
#         region=['Bludhaven', 'Gotham'],
#         kmer_map=[self.kmer_map1, self.kmer_map2],
#         regional_alignment=[self.align1, self.align2],
#         regional_table=[self.table1, self.table2],
#         reference_taxonomy=self.taxonomy,
#         min_counts=10,
#         debug=True
#         )
#     pdt.assert_frame_equal(mapping.view(pd.DataFrame), 
#                            self.seq_map.view(pd.DataFrame))
#     pdt.assert_frame_equal(summary.view(pd.DataFrame),
#                            self.database_summary.view(pd.DataFrame))
#     pdt.assert_frame_equal(
#         count_table.view(pd.DataFrame),
#         pd.DataFrame( 
#             data=np.array([[100.,  50,   0,  50,  50, 50],
#                            [100.,  25, 100,  25,  25, 25],
#                            [  0., 100, 100,   0,  50, 50]]),

This file has been truncated. show original

But, when i run the test, i get a type error:

Traceback (most recent call last):
  File "test_plugin_setup.py", line 144, in test_sidle_reconstruction
    results = sidle.sidle_reconstruction(
  File "<decorator-gen-611>", line 2, in sidle_reconstruction
  File "/Users/jusdeb/miniconda3/envs/qiime2-2021.8-dev/lib/python3.8/site-packages/qiime2/sdk/action.py", line 245, in bound_callable
    outputs = self._callable_executor_(scope, callable_args,
  File "/Users/jusdeb/miniconda3/envs/qiime2-2021.8-dev/lib/python3.8/site-packages/qiime2/sdk/action.py", line 490, in _callable_executor_
    raise TypeError("Pipelines must return `Result` objects, "
TypeError: Pipelines must return `Result` objects, not <class 'qiime2.sdk.results.Results'>

Im probably missing something totally obvious, but Im not sure what it is.

Thanks!

thermokarst · August 11, 2021, 3:14pm

Hey @jwdebelius! This is a great question.

QIIME 2 Actions return Results objects, which is basically just a fancy tuple - this container has one or more Result objects - Result is an Artifact or a Visualization.

In your new pipeline you've got two separate lines that don't unpack that tuple:

github.com

jwdebelius/q2-sidle/blob/c41571706c217e46c7fbd07b1fcb5367339c23e1/q2_sidle/_pipelines.py#L35

      
        
                region=region,
                kmer_map=kmer_map,
                regional_alignment=regional_alignment,
                count_degenerates=True,
                block_size=block_size,
                n_workers=n_workers,
                client_address=client_address,
                debug=debug,
                )
            
            
counts = reconstruct_counts(
                region=region,
                regional_alignment=regional_alignment,
                regional_table=regional_table,
                database_map=db_map,
                database_summary=db_summary,
                region_normalize='average',
                min_counts=min_counts,
                block_size=block_size,
                n_workers=n_workers,
                client_address=client_address,

github.com

jwdebelius/q2-sidle/blob/c41571706c217e46c7fbd07b1fcb5367339c23e1/q2_sidle/_pipelines.py#L49

      
        
                database_map=db_map,
                database_summary=db_summary,
                region_normalize='average',
                min_counts=min_counts,
                block_size=block_size,
                n_workers=n_workers,
                client_address=client_address,
                debug=debug,
                )
            
            
taxonomy = reconstruct_taxonomy(
                reconstruction_map=db_map,
                taxonomy=reference_taxonomy,
                database=database,
                define_missing=define_missing,
                )
            
            
results = [db_map, db_summary, counts, taxonomy]
            
            
return tuple(results)

One really simple way to resolve the issue is to actually just append a , after each assignment variable, like here:

github.com

qiime2/q2-vsearch/blob/8a1f85f8cd3bde743c8386ad28742ec9ed156dbd/q2_vsearch/_cluster_features.py#L342-L343

      
        
            unmatched_table, = filter_features(table=table,
                                               metadata=unmatched_seqs_md)

I think once you do that you should be in good shape, but let me know! This is a common stumbling block, we should probably think about a new way to report the errors here, because this also crops up in the Artifact API pretty often, too (different error, same general idea).

jwdebelius · August 11, 2021, 6:08pm

Thanks @thermokarst! That worked perfectly