# Beta diversity API **URL:** https://forum.qiime2.org/t/beta-diversity-api/16288 **Category:** Technical Support **Tags:** python-api, group-significance, core-metrics **Created:** [August 28, 2020, 6:03pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288 "2020-08-28T18:03:54Z") **Posts on this page:** 9 **Page:** 1
### Author: ![sterrettJD](https://forum.qiime2.org/user_avatar/forum.qiime2.org/sterrettjd/32/20135_2.png) [@sterrettJD](https://forum.qiime2.org/u/sterrettJD) #### Post date: [August 28, 2020, 6:03pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/1 "2020-08-28T18:03:54Z")
Hi! I'm using the Artifact API to export all the results of beta diversity group significance testing to a csv. I've been able to do this with the alpha diversity significance and correlation tests, and I have been able to get this to work for most of the columns in my metadata, but there are a few problematic ones. I've included a simplified version of the code that still returns the error below. The actual code loops through the metrics and categorical metadata columns. I'm running QIIME2 2020.6 and python 3.6.10 Just to make sure it's clear, the general structure is to: - Run the core\_metrics\_phylogenetic pipeline - Export the visualization from group significance testing - Parse the resulting index.html file, extracting the relevant info - Create a data frame from the information - When looping through the columns, I concat each dataframe to make a main dataframe with the results of all columns for each specific metric - Export each metric's dataframe as a csv (not included) ```python feattable = Artifact.load('noMito_noChloro-filtered-table.qza') tree = Artifact.load('insertion-tree.qza') metadata = Metadata.load('meta-v3.tsv') core_metrics = diversity.pipelines.core_metrics_phylogenetic(table = feattable, phylogeny = tree, sampling_depth = 13100, metadata = metadata_API) metric = 'jaccard' column = 'Race_Ethnicity-cat' output_dir = f'{metric}_sig_{column}' matrix = getattr(core_metrics, f'{metric}_distance_matrix') df = pd.DataFrame() try: significance = beta_group_significance(distance_matrix=matrix, metadata=metadata_API.get_column(column)) significance.visualization.export_data(output_dir) with open(f'{output_dir}/index.html') as f: soup = BeautifulSoup(f, 'html.parser') keys = [key.string for key in soup.find_all('th')[2:]] #keys output is ['method name','test statistic name','sample size','number of groups', # 'test statistic','p-value','number of permutations'] values = np.array([value.string for value in soup.find_all('td')]) values.reshape((1,7)) #values is a vertical array with the corresponding values for each key col_index.append(column) #Make the data frame for the current column, concatenate it with the one for other columns #under the given metric #look into changing to df[f'{column}'] == values? Bigger fish to fry rn though dfn = pd.DataFrame(values,index=keys) df = pd.concat([df,dfn],axis=1) except ValueError: print(f'{column} could not be added, as each sample is in a different category.\n Moving to the next column.') ``` The error I'm met with is: ```python FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/t7/vhs8v0wd2xqb851ps1hzn0mm0000gn/T/qiime2-temp-ier4fo3c/Asian/Pacific%20Islander-boxplots.png'. ``` The specific line for the traceback is `metadata=metadata_API.get_column(column))` The whole traceback is: ```python `FileNotFoundError Traceback (most recent call last).` ` in ` ` 16 try:` `17 significance = beta_group_significance(distance_matrix=matrix,` ` ---> 18 metadata=metadata_API.get_column(column))` `19 significance.visualization.export_data(output_dir)` in beta_group_significance(distance_matrix, metadata, method, pairwise, permutations) /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py in bound_callable(*args, **kwargs) 243 # Execute 244 outputs = self._callable_executor_(scope, callable_args, --> 245 output_types, provenance) 246 247 if len(outputs) != len(self.signature.outputs): /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/qiime2/sdk/action.py in _callable_executor_(self, scope, view_args, output_types, provenance) 450 # will also need to be updated to support OutPath instead of str. 451 with tempfile.TemporaryDirectory(prefix='qiime2-temp-') as temp_dir: --> 452 ret_val = self._callable(output_dir=temp_dir, **view_args) 453 if ret_val is not None: 454 raise TypeError( /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/q2_diversity/_beta/_visualizer.py in beta_group_significance(output_dir, distance_matrix, metadata, method, pairwise, permutations) 194 fig = ax.get_figure() 195 fig.savefig(os.path.join(output_dir, '%s-boxplots.png' % --> 196 urllib.parse.quote(str(group_id)))) 197 fig.savefig(os.path.join(output_dir, '%s-boxplots.pdf' % 198 urllib.parse.quote(str(group_id)))) /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/matplotlib/figure.py in savefig(self, fname, transparent, **kwargs) 2201 self.patch.set_visible(frameon) 2202 -> 2203 self.canvas.print_figure(fname, **kwargs) 2204 2205 if frameon: /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/matplotlib/backend_bases.py in print_figure(self, filename, dpi, facecolor, edgecolor, orientation, format, bbox_inches, **kwargs) 2124 orientation=orientation, 2125 bbox_inches_restore=_bbox_inches_restore, -> 2126 **kwargs) 2127 finally: 2128 if bbox_inches and restore_bbox: /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py in print_png(self, filename_or_obj, metadata, pil_kwargs, *args, **kwargs) 533 else: 534 renderer = self.get_renderer() --> 535 with cbook.open_file_cm(filename_or_obj, "wb") as fh: 536 _png.write_png(renderer._renderer, fh, self.figure.dpi, 537 metadata={ **default_metadata,** metadata}) /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/contextlib.py in __enter__ (self) 79 def __enter__ (self): 80 try: ---> 81 return next(self.gen) 82 except StopIteration: 83 raise RuntimeError("generator didn't yield") from None /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/matplotlib/cbook/ __init__.py in open_file_cm(path_or_file, mode, encoding) 416 def open_file_cm(path_or_file, mode="r", encoding=None): 417 r"""Pass through file objects and context-manage `.PathLike`\s.""" --> 418 fh, opened = to_filehandle(path_or_file, mode, True, encoding) 419 if opened: 420 with fh: /opt/anaconda3/envs/qiime2-2020.6/lib/python3.6/site-packages/matplotlib/cbook/ __init__.py in to_filehandle(fname, flag, return_opened, encoding) 401 fh = bz2.BZ2File(fname, flag) 402 else: --> 403 fh = open(fname, flag, encoding=encoding) 404 opened = True 405 elif hasattr(fname, 'seek'): FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/t7/vhs8v0wd2xqb851ps1hzn0mm0000gn/T/qiime2-temp-ier4fo3c/Asian/Pacific%20Islander-boxplots.png' ``` Any ideas as to why I'm getting this error for this column? I also get it for my date collected column, which I'm meaning to convert to not be categorical, so that isn't quite as important. Is there a way to accomplish this without exporting the data and then parsing it? I couldn't find a way to pull it directly from the artifact in the API. (Also, I apologize if my code is a little clunky, I'm still somewhat new to python, and I haven't had a chance to clean this code up yet.) Thanks! šŸ˜€ ---
### Author: ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst) #### Post date: [August 28, 2020, 7:35pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/2 "2020-08-28T19:35:29Z")
Hey there @sterrettJD! > [@sterrettJD](#): > > Any ideas as to why Iā€™m getting this error for this column? Because there is a forward-slash (`/`) in the column name - it is being interpreted in this script as a directory. Are you able to change the name of the column? If so, removing the slash will get you moving most quickly here. If not, let us know and we can lend a hand with tweaking the script with you. ---
### Author: ![sterrettJD](https://forum.qiime2.org/user_avatar/forum.qiime2.org/sterrettjd/32/20135_2.png) [@sterrettJD](https://forum.qiime2.org/u/sterrettJD) #### Post date: [August 28, 2020, 10:41pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/3 "2020-08-28T22:41:21Z")
Thanks @thermokarst!! I replaced any "/" s in the the metadata contents, and that fixed it! I had been thinking that I just didn't understand what was going on with the temporary files, and I didn't realize it was being picked up as a new directory. I really appreciate the help! Also, am I reinventing the wheel with this? Is there any way to grab the stats results without exporting and parsing? I tried to find some resources to further describe the attributes and methods for using the API, but I couldn't find any documentation other than the the examples under the tutorials and [tkosciol's jupyter notebook](https://github.com/tkosciol/q2-notebooks/blob/master/q2-moving-pictures.ipynb). The docstrings in jupyter notebook were very helpful, but otherwise I wasn't sure where to look. ---
### Author: ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst) #### Post date: [August 28, 2020, 10:50pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/4 "2020-08-28T22:50:40Z")
> [@sterrettJD](#): > > I really appreciate the help! Glad to hear it! > [@sterrettJD](#): > > Also, am I reinventing the wheel with this? Not really - there isn't a programmatic way to interrogate this viz. Keep us posted! ---
### Author: ![sterrettJD](https://forum.qiime2.org/user_avatar/forum.qiime2.org/sterrettjd/32/20135_2.png) [@sterrettJD](https://forum.qiime2.org/u/sterrettJD) #### Post date: [August 29, 2020, 4:41am UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/5 "2020-08-29T04:41:47Z")
> [@thermokarst](#): > > Keep us posted! Will do! Would this be something worth contributing to the plugin? I've been keeping my eye out for a good first project, but I haven't seen many issues tagged as beginner/good first issue lately. I'd really like to learn more about the framework and how to contribute, as I'm trying to shift my career trajectory more toward development. I can see that it might not be entirely desirable to allow easy multiple hypothesis testing like that for beta diversity without false discovery rate correction, so I understand if there are concerns about statistical abuse with allowing people to test every column so easily. However, it can also be a pain to go through every column individually when your metadata contains \>200 columns. If it's best to have this discussion elsewhere, I'm happy to start a new topic or shoot you an email about it! Thanks! ---
### Author: ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst) #### Post date: [August 31, 2020, 3:05pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/6 "2020-08-31T15:05:53Z")
---
### Author: ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst) #### Post date: [September 1, 2020, 2:46pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/7 "2020-09-01T14:46:30Z")
Thanks @sterrettJD! Let's start a new topic over in "Developer Discussion," that way we can identify the goals and discuss a plan! Can you take point on that? Thanks! :qiime2: ---
### Author: ![thermokarst](https://forum.qiime2.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@thermokarst](https://forum.qiime2.org/u/thermokarst) #### Post date: [September 1, 2020, 2:46pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/8 "2020-09-01T14:46:32Z")
---
### Author: ![system](https://forum-qiime2-org.s3.dualstack.us-west-2.amazonaws.com/original/3X/2/1/21af5fe23cb6f4579467c66a9ed94e55274ca7bd.svg) [@system](https://forum.qiime2.org/u/system) #### Post date: [October 2, 2020, 8:46pm UTC](https://forum.qiime2.org/t/beta-diversity-api/16288/9 "2020-10-02T20:46:35Z")
This topic was automatically closed 31 days after the last reply. New replies are no longer allowed.