Difficulties in built a correlation network after using q2-SCNIC

joaomiranda · June 2, 2023, 3:23pm

Dear Qiimers,

I'm having some issues concerning q2-SCNIC and I need some help to execute a good pipeline that could give me a correlation network with positive and negative edges between my samples.

I'm using qiime 2022.2

1 - I've already runned the codes from q2-SCNIC according to this tutorial it is described below, with some observations:

I had errors when tried with the p-method sparcc, so according to the forum I used p-method pearson and worked.
I used a collapsed table with level 5, aiming to have a network with taxons and not "IDs"

 qiime SCNIC sparcc-filter \
  --i-table filtered_table_bacteria_level-5.qza \
  --o-table-filtered filtered_table_bacteria_level-5_scnic.qza

  qiime SCNIC calculate-correlations \
  --i-table filtered_table_bacteria_level-5_scnic.qza \
  --p-method pearson \
  --o-correlation-table bacteria_correls_level-5.qza

 qiime SCNIC make-modules-on-correlations \
  --i-correlation-table bacteria_correls_level-5.qza \
  --i-feature-table filtered_table_bacteria_level-5.qza \
  --p-min-r .35 \
  --o-collapsed-table bacteria_data_collapsed_level-5.qza \
  --o-correlation-network bacteria_net_modules_level-5.qza \
  --o-module-membership bacteria_membership_level-5.qza

 qiime tools extract \
  --input-path bacteria_net_modules_level-5.qza \
  --output-path extracted-bacteria-net-level-5_table

2 - Thus, I used my extracted network file .gml to import in Cytoscape v.3.9.1 and igraph in R v.4.3.0.

3 - My results are bit confused, I had imported the file and built networks in both interfaces, but it's kind messy and I want to built clearer and organized networks. Where are my mistakes? Could you help me?

Here are my files:
filtered_table_bacteria_level-5.qza (31.9 KB),
bacteria_net_modules_level-5.qza (62.0 KB),
Bacteria_taxonomy.qza (8.3 KB),
Metadata.txt (789 Bytes)

Here are my networks in Cytoscape and igraph respectively:

igraph

crusher083 · June 6, 2023, 12:53pm

Hello,

I never worked with Cytoscape and igraph (and overall I don't use software with GUI, because it is limiting). Try to find a button that will turn off lables.
It will be easy to visualize and correct labels using https://networkx.org/ in Python, and you can customize it to your wishes.

Cheers
V

BenKaehler · June 6, 2023, 8:47pm

Hi @joaomiranda,

Thanks very much for your question. I haven't worked on or with q2-SCNIC, but a few of us have been working on a new QIIME 2 plugin called q2-makarsa.

q2-makarsa provides access to SpiecEasi or FlashWeave for microbial network inference, and has visualisation and module detection built-in.

For instance, I ran your filtered_table_bacteria_level-5.qza table through FlashWeave (with default parameters) and got this:

The colouring is by Louvain communities, which were inferred from the network. The visualiser itself allows you to view more detailed statistics and the taxonomy information in your input.

I would also recommend a slightly different workflow, where rather than collapsing the network nodes by taxa, you could preserve the original ASVs but then colour the nodes according to their level-5 taxonomy.

For completeness, I produced the above plot by running.

qiime makarsa flashweave \
    --i-table filtered_table_bacteria_level-5.qza \
    --o-network flashweave.qza

qiime makarsa louvain-communities \
    --i-network flashweave.qza \
    --o-community flashweave-louvain.qza

qiime makarsa visualise-network \
    --i-network flashweave.qza  \
    --m-metadata-file flashweave-louvain.qza \
    --o-visualization flashweave.qzv

qiime tools view flashweave.qzv

If you instead wanted to use taxonomy data to colour your nodes, I would start with your ASV feature table at the flashweave command, then include the taxonomy file as a metadata input to the visualise-network command. If you would like to post the ASV file I would be happy to run you through that workflow.

Installation instructions and further information are available via the link above. We've only just released the plugin, so bug reports or feature requests are welcome!

Also please don't hesitate to follow up here if you would like any clarification.

Ben

joaomiranda · June 7, 2023, 2:46am

Hi @BenKaehler,

That's fantastic! I will run these workflow, it looks like that might gave me the results that I'm ehoping for.

Also I'm sending here my filtered-table without the taxonomy, and the taxonomy.qza file.

Thanks for your support!

Bacteria_taxonomy.qza (8.3 KB)
feature-bacteria-table-filtered.qza (17.8 KB)

BenKaehler · June 7, 2023, 8:55pm

Thanks @joaomiranda, I hope it works for you!

I ran your filtered table through FlashWeave. These are the commands I used:

qiime makarsa flashweave \
    --i-table feature-bacteria-table-filtered.qza \
    --o-network flashweave.qza

qiime makarsa louvain-communities \
    --i-network flashweave.qza \
    --o-community flashweave-louvain.qza

qiime makarsa visualise-network \
    --i-network flashweave.qza  \
    --m-metadata-file flashweave-louvain.qza \
    --m-metadata-file corrected_taxonomy.qza \
    --o-visualization flashweave.qzv

qiime tools view flashweave.qzv

This is the resulting network:

I have attached the resulting visualisation, so that you can play with the graph and read node details yourself. For instance you can change the number of taxonomic levels that used to colour the nodes. You don't need to install q2-makarsa to view it.

Also, I had to edit your taxonomy data to replace "|" delimiters with "; ", so I've attached that too.

It is possible to get a similar graph (and a few smaller subgraphs) using SpiecEasi, but I had to change the method to "mb" rather than the default "glasso".

Anyway, I hope that helps!

flashweave.qzv (220.5 KB)
corrected_taxonomy.qza (8.7 KB)

joaomiranda · June 12, 2023, 2:58am

Hey @BenKaehler,

Thank you so much for your support!

Unfortunately, when I was running my code, an error was encountered:

 Plugin error from makarsa:

  An error was encountered while running FlashWeave in Julia (return code 1), please inspect stdout and stderr to learn more.

Debug info has been saved to /tmp/qiime2-q2cli-err-p49_nhzi.log

I'm stucked now, I've my taxonomy.qza file corrected and my feature-table filtered according to my objectives.

Could you help me to solve this error?

joaomiranda · June 19, 2023, 9:20pm

Hi!

I think it's something related with the instalattion.

I'm using the qiime 2023.5 now, I've actualized my conda envinronment.

I had runned conda list and saw my plugins, the q2_makarsa looks like quite different:

q2-deblur                 2023.5.0                 py38_0    qiime2/label/r2023.5
q2-demux                  2023.5.0                 py38_0    qiime2/label/r2023.5
q2-diversity              2023.5.1                 py38_0    qiime2/label/r2023.5
q2-diversity-lib          2023.5.0                 py38_0    qiime2/label/r2023.5
q2-emperor                2023.5.0                 py38_0    qiime2/label/r2023.5
q2-feature-classifier     2023.5.0                 py38_0    qiime2/label/r2023.5
q2-feature-table          2023.5.0                 py38_0    qiime2/label/r2023.5
q2-fragment-insertion     2023.5.0                 py38_0    qiime2/label/r2023.5
q2-gneiss                 2023.5.0                 py38_0    qiime2/label/r2023.5
q2-longitudinal           2023.5.0                 py38_0    qiime2/label/r2023.5
q2-makarsa                0.0.0.dev0               pypi_0    pypi
q2-metadata               2023.5.0                 py38_0    qiime2/label/r2023.5
q2-mystery-stew           2023.5.0                 py38_0    qiime2/label/r2023.5
q2-phylogeny              2023.5.0                 py38_0    qiime2/label/r2023.5
q2-quality-control        2023.5.0                 py38_0    qiime2/label/r2023.5
q2-quality-filter         2023.5.0                 py38_0    qiime2/label/r2023.5
q2-sample-classifier      2023.5.0                 py38_0    qiime2/label/r2023.5
q2-taxa                   2023.5.0                 py38_0    qiime2/label/r2023.5
q2-types                  2023.5.0                 py38_0    qiime2/label/r2023.5
q2-vsearch                2023.5.0                 py38_0    qiime2/label/r2023.5
q2cli                     2023.5.1                 py38_0    qiime2/label/r2023.5
q2galaxy                  2023.5.0                 py38_0    qiime2/label/r2023.5
q2templates               2023.5.0                 py38_0    qiime2/label/r2023.5
qiime2                    2023.5.1                 py38_0    qiime2/label/r2023.5

So I decided to re-install q2_makarsa using the instructions described here.

And there is a new error:

conda install -c bioconda -c conda-forge r-spieceasi julia
Collecting package metadata (current_repodata.json): failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 748, in _update_chunk_length
        self.chunk_left = int(line, 16)
    ValueError: invalid literal for int() with base 16: b''

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 443, in _error_catcher
        yield
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 815, in read_chunked
        self._update_chunk_length()
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 752, in _update_chunk_length
        raise InvalidChunkLength(self, line)
    urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/models.py", line 816, in generate
        yield from self.raw.stream(chunk_size, decode_content=True)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 623, in stream
        for line in self.read_chunked(amt, decode_content=decode_content):
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 844, in read_chunked
        self._original_response.close()
      File "/home/joaomiranda/miniconda3/lib/python3.9/contextlib.py", line 135, in __exit__
        self.gen.throw(type, value, traceback)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/urllib3/response.py", line 460, in _error_catcher
        raise ProtocolError("Connection broken: %r" % e, e)
    urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1114, in __call__
        return func(*args, **kwargs)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
        exit_code = do_call(args, p)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 90, in do_call
        return getattr(module, func_name)(args, parser)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/cli/main_install.py", line 20, in execute
        install(args, parser, 'install')
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/cli/install.py", line 259, in install
        unlink_link_transaction = solver.solve_for_transaction(
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 152, in solve_for_transaction
        unlink_precs, link_precs = self.solve_for_diff(update_modifier, deps_modifier,
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 195, in solve_for_diff
        final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 300, in solve_final_state
        ssc = self._collect_all_metadata(ssc)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/common/io.py", line 86, in decorated
        return f(*args, **kwds)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 463, in _collect_all_metadata
        index, r = self._prepare(prepared_specs)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 1058, in _prepare
        reduced_index = get_reduced_index(self.prefix, self.channels,
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/index.py", line 287, in get_reduced_index
        new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 139, in query_all
        result = tuple(concat(executor.map(subdir_query, channel_urls)))
      File "/home/joaomiranda/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 608, in result_iterator
        yield fs.pop().result()
      File "/home/joaomiranda/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 438, in result
        return self.__get_result()
      File "/home/joaomiranda/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
        raise self._exception
      File "/home/joaomiranda/miniconda3/lib/python3.9/concurrent/futures/thread.py", line 52, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 131, in <lambda>
        subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 144, in query
        self.load()
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 209, in load
        _internal_state = self._load()
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 374, in _load
        raw_repodata_str = fetch_repodata_remote_request(
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 700, in fetch_repodata_remote_request
        resp = session.get(join_url(url, filename), headers=headers, proxies=session.proxies,
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/sessions.py", line 600, in get
        return self.request("GET", url, **kwargs)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
        resp = self.send(prep, **send_kwargs)
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/sessions.py", line 745, in send
        r.content
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/models.py", line 899, in content
        self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
      File "/home/joaomiranda/miniconda3/lib/python3.9/site-packages/requests/models.py", line 818, in generate
        raise ChunkedEncodingError(e)
    requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))

`$ /home/joaomiranda/miniconda3/bin/conda install -c bioconda -c conda-forge r-spieceasi julia`

  environment variables:
                 CIO_TEST=<not set>
        CONDA_DEFAULT_ENV=qiime2-2023.5
                CONDA_EXE=/home/joaomiranda/miniconda3/bin/conda
       CONDA_JL_CONDA_EXE=/home/joaomiranda/miniconda3/bin/conda
CONDA_JL_CONDA_EXE_BACKUP=
            CONDA_JL_HOME=/home/joaomiranda/miniconda3/envs/qiime2-2023.5
     CONDA_JL_HOME_BACKUP=
             CONDA_PREFIX=/home/joaomiranda/miniconda3/envs/qiime2-2023.5
           CONDA_PREFIX_1=/home/joaomiranda/miniconda3
    CONDA_PROMPT_MODIFIER=(qiime2-2023.5)
         CONDA_PYTHON_EXE=/home/joaomiranda/miniconda3/bin/python
               CONDA_ROOT=/home/joaomiranda/miniconda3
              CONDA_SHLVL=2
           CURL_CA_BUNDLE=<not set>
     JAVA_LD_LIBRARY_PATH=/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/jvm/lib/server
         JULIA_DEPOT_PATH=/home/joaomiranda/miniconda3/envs/qiime2-2023.5/share/julia:
          JULIA_LOAD_PATH=@:@qiime2-2023.5:@stdlib
  JULIA_SSL_CA_ROOTS_PATH=/home/joaomiranda/miniconda3/envs/qiime2-2023.5/ssl/cacert.pem
                     PATH=/home/joaomiranda/miniconda3/bin:/home/joaomiranda/miniconda3/envs/qii
                          me2-2023.5/bin:/home/joaomiranda/miniconda3/condabin:/usr/local/sbin:/
                          usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/game
                          s:/mnt/c/Program Files/WindowsApps/CanonicalGroupLimited.Ubuntu20.04on
                          Windows_2004.2022.8.0_x64__79rhkp1fndgsc:/mnt/c/Program Files
                          (x86)/Common Files/Oracle/Java/javapath:/mnt/c/Program Files
                          (x86)/Intel/iCLS Client:/mnt/c/Program Files/Intel/iCLS Client:/mnt/c/
                          WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WI
                          NDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/Program Files
                          (x86)/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program
                          Files/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program
                          Files (x86)/Intel/Intel(R) Management Engine
                          Components/IPT:/mnt/c/Program Files/Intel/Intel(R) Management Engine
                          Components/IPT:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/ME
                          GA11:/mnt/c/Users/Joao/AppData/Local/Microsoft/WindowsApps:/snap/bin
         PYTHONNOUSERSITE=/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python*/site-
                          packages/
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>

     active environment : qiime2-2023.5
    active env location : /home/joaomiranda/miniconda3/envs/qiime2-2023.5
            shell level : 2
       user config file : /home/joaomiranda/.condarc
 populated config files :
          conda version : 4.13.0
    conda-build version : not installed
         python version : 3.9.5.final.0
       virtual packages : __linux=4.4.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/joaomiranda/miniconda3  (writable)
      conda av data dir : /home/joaomiranda/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/bioconda/linux-64
                          https://conda.anaconda.org/bioconda/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/joaomiranda/miniconda3/pkgs
                          /home/joaomiranda/.conda/pkgs
       envs directories : /home/joaomiranda/miniconda3/envs
                          /home/joaomiranda/.conda/envs
               platform : linux-64
             user-agent : conda/4.13.0 requests/2.28.1 CPython/3.9.5 Linux/4.4.0-19041-Microsoft ubuntu/20.04.2 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

If submitted, this report will be used by core maintainers to improve
future releases of conda.
Would you like conda to send this report to the core maintainers?

BenKaehler · June 20, 2023, 9:01pm

Hi @joaomiranda, sorry for the slow response, I missed your question. I would be interested to know why qiime makarsa flashweave failed for you in this instance, but I'll move on to helping with your most recent post.

For future reference, you can get more information on what when wrong in cases like this by adding --verbose to the end of your command.

BenKaehler · June 20, 2023, 9:27pm

Hi again @joaomiranda, looking at your conda issue here I admit to being a bit flummoxed.

Did you follow the installation instructions here the first time you installed? Did it work that time?

Your output from conda list is normal. We're still installing q2-makarsa using pip. We may move to conda in future, but for the moment the output that you're seeing is fine. (We'll give it a more attractive version number if it gets published.)

Looking at the output from conda install -c bioconda -c conda-forge r-spieceasi julia, I suspect that your network connection might have had some issue while you were running it, or that it might have been some other intermittent conda issue. Googling that error message hasn't yielded anything useful to me. My best advice at this stage is to please try rerunning the installation instructions.

To test, I did a clean install on my machine, and the following list of commands worked for me.

wget https://data.qiime2.org/distro/core/qiime2-2023.5-py38-linux-conda.yml
conda env create -n qiime2-2023.5 --file qiime2-2023.5-py38-linux-conda.yml
conda activate qiime2-2023.5
conda install -c bioconda -c conda-forge r-spieceasi julia
julia -e 'using Pkg; Pkg.add(["FlashWeave", "ArgParse", "GraphIO"])'
pip install git+https://github.com/BenKaehler/q2-makarsa.git
qiime makarsa

Finally, I don't want to muddy the waters, but we've been using mamba as a replacement for conda, and it's been faster, at least. You could perhaps try using mamba instead of conda if the conda issues don't resolve.

Hopefully trying again will fix it, but please let me know how you go!

joaomiranda · June 20, 2023, 10:28pm

Hey @BenKaehler thank you for your response!

I've reinstalled q2_makarsa sucessfully.

So I've run again the code, now with --verbose, I have an error again:

  qiime makarsa flashweave \
    --i-table Ae_alb-filtered-bacteria-table.qza \
	--verbose \
    --o-network flashweave.qza

Running external command line application(s). This may print messages to stdout and/or stderr.
The command(s) being run are below. These commands cannot be manually re-run as they will depend on temporary files that no longer exist.

Command: run_FlashWeave.jl --datapath /tmp/tmpl7nkftvn/input-data.tsv --output /tmp/tmpl7nkftvn/network.gml --max_k 3 --alpha 0.01 --conv 0.01 --max_tests 1000000 --hps 5 --n_obs_min -1 --time_limit -1.0 --prec 64 --update_interval 30 --verbose --sensitive --feed_forward --FDR --normalize --make_sparse


### Loading data ###

### Normalizing ###

ERROR: LoadError: TypeError: in keyword argument header, expected Vector{String}, got a value of type Vector{Any}
Stacktrace:
 [1] learn_network(data::Matrix{Float64}; sensitive::Bool, heterogeneous::Bool, max_k::Int64, alpha::Float64, conv::Float64, header::Vector{Any}, meta_mask::BitVector, feed_forward::Bool, fast_elim::Bool, normalize::Bool, track_rejections::Bool, verbose::Bool, transposed::Bool, prec::Int64, make_sparse::Bool, make_onehot::Bool, max_tests::Int64, hps::Int64, FDR::Bool, n_obs_min::Int64, cache_pcor::Bool, time_limit::Float64, update_interval::Float64, parallel_mode::String, extra_data::Nothing, share_data::Bool, experimental_kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ FlashWeave ~/miniconda3/envs/qiime2-2023.5/share/julia/packages/FlashWeave/j91Ng/src/learning.jl:500
 [2] learn_network(data_path::String, meta_data_path::Nothing; otu_data_key::String, otu_header_key::String, meta_data_key::String, meta_header_key::String, verbose::Bool, transposed::Bool, kwargs::Base.Pairs{Symbol, Real, NTuple{16, Symbol}, NamedTuple{(:heterogeneous, :sensitive, :max_k, :alpha, :conv, :feed_forward, :max_tests, :hps, :FDR, :n_obs_min, :time_limit, :normalize, :track_rejections, :prec, :make_sparse, :update_interval), Tuple{Bool, Bool, Int64, Float64, Float64, Bool, Int64, Int64, Bool, Int64, Float64, Bool, Bool, Int64, Bool, Float64}}})
   @ FlashWeave ~/miniconda3/envs/qiime2-2023.5/share/julia/packages/FlashWeave/j91Ng/src/learning.jl:347
 [3] main()
   @ Main ~/miniconda3/envs/qiime2-2023.5/bin/run_FlashWeave.jl:110
 [4] top-level scope
   @ ~/miniconda3/envs/qiime2-2023.5/bin/run_FlashWeave.jl:136
in expression starting at /home/joaomiranda/miniconda3/envs/qiime2-2023.5/bin/run_FlashWeave.jl:136
Traceback (most recent call last):
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_makarsa/_flashweave.py", line 88, in flashweave
    run_commands([cmd])
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_makarsa/_run_commands.py", line 19, in run_commands
    subprocess.run(cmd, check=True)
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['run_FlashWeave.jl', '--datapath', '/tmp/tmpl7nkftvn/input-data.tsv', '--output', '/tmp/tmpl7nkftvn/network.gml', '--max_k', '3', '--alpha', '0.01', '--conv', '0.01', '--max_tests', '1000000', '--hps', '5', '--n_obs_min', '-1', '--time_limit', '-1.0', '--prec', '64', '--update_interval', '30', '--verbose', '--sensitive', '--feed_forward', '--FDR', '--normalize', '--make_sparse']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2cli/commands.py", line 468, in __call__
    results = action(**arguments)
  File "<decorator-gen-574>", line 2, in flashweave
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 274, in bound_callable
    outputs = self._callable_executor_(
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/qiime2/sdk/action.py", line 509, in _callable_executor_
    output_views = self._callable(**view_args)
  File "/home/joaomiranda/miniconda3/envs/qiime2-2023.5/lib/python3.8/site-packages/q2_makarsa/_flashweave.py", line 90, in flashweave
    raise Exception(
Exception: An error was encountered while running FlashWeave in Julia (return code 1), please inspect stdout and stderr to learn more.

Plugin error from makarsa:

  An error was encountered while running FlashWeave in Julia (return code 1), please inspect stdout and stderr to learn more.

See above for debug info.

BenKaehler · June 21, 2023, 11:30am

Thanks @joaomiranda, that looks like a proper bug.

If you’re happy to share your input file I would love to debug it for you, but no pressure.

joaomiranda · June 21, 2023, 11:49am

Surely!

Here is my feature-table .qza and .qzv files, and my taxonomy file that should be used in the last code of the pipeline.

Ae_alb-filtered-bacteria-table.qza (21.8 KB)
Ae_alb-filtered-bacteria-table.qzv (408.7 KB)
Bacteria_Taxonomy_Family.qza (6.1 KB)

BenKaehler · June 21, 2023, 9:21pm

Thanks @joaomiranda!

Strangely, I got a different error to you when I ran

qiime makarsa flashweave \
    --i-table Ae_alb-filtered-bacteria-table.qza \
    --o-network latest.qza \
    --verbose

The error that I got was related to the low number of samples in that table (8). Naively, I would guess that 8 samples isn't going to give you much power resolving an ecological network.

I could get FlashWeave to run by telling it to stop complaining about the low number of samples by running

qiime makarsa flashweave \
    --i-table Ae_alb-filtered-bacteria-table.qza \
    --o-network latest.qza \
    --p-n-obs-min 2

The result wasn't great, with 8 disjoint graphs of 8 or less features. Wow, if I thought 8 was a lucky number I would go buy a lottery ticket now.

I ran it through SpiecEasi, as well. SpiecEasi didn't find any networks at all.

There were also issues with the taxonomy file you sent. I think it was missing some of the feature ids that were in the feature table you sent. The corrected_taxonomy.qza taxonomy file I sent worked well, though. If your feature ids haven't changed since then, I would recommend using it instead.

So sorry, 8 samples might not be enough for this exercise, but feel free to run the command above to see what you get.

Also, I'm a bit disappointed that I couldn't fix the error that you saw, but I didn't see it. Please let me know if you see it again.

joaomiranda · June 21, 2023, 10:31pm

Hi @BenKaehler!

I think we are approaching to the correlation network that we desire.

I'll clarify you about my data, next I'll show to you my new network after ran the code according to your observations.

I have a dataset that includ three diferent groups (four, but one is an unique sample so I decided to not include in my analysis). These groups are different species of mosquitoes, whose gut microbiome were analyzed. My objective is to build correlation networks for each species separately. To understand the correlations between microbes in different species of mosquitoes.

Here is my metadata :
Metadata.txt (789 Bytes)

For this purpose I used the filter-samples to generate a table with just one species (Ae. albopictus). My new feature-table was filtered and I had 8 samples associated with Ae. albopictus.
Ae_alb-filtered-bacteria-table.qza (21.8 KB)

qiime feature-table filter-samples \
  --i-table q2_makarsa/filtered-bacteria-table.qza \
  --m-metadata-file Metadata.txt \
  --p-where "[species]='Ae. albopictus'" \
  --o-filtered-table q2_makarsa/Ae_alb-filtered-bacteria-table.qza

I've ran the code again with the --p-n-obs-min 2 and with your
corrected_taxonomy.qza (8.7 KB), it worked. But as you have mentioned, the result was 8 different networks

To compare, I've ran now with my filtered-table including all the species of mosquitoes that I'm analyzing. I have apparently a great result with an unique network, but I have some issues with this result, how could I figure out which bacteria are from which mosquito species? And from which sample? I have some diificulties in understand some terms as: Degree, Betweenness, Closeness centrality, Associativity and why there are 43 entries in my taxa list (But I think we can discuss this after solve the central issue).
filtered-bacteria-table.qza (19.0 KB)
bacteria_network.qzv (222.5 KB)

qiime makarsa flashweave \
    --i-table filtered-bacteria-table.qza \
    --o-network flashweave_2.qza \
	
	qiime makarsa louvain-communities \
    --i-network flashweave_2.qza \
    --o-community flashweave-louvain-2.qza
	
	qiime makarsa visualise-network \
    --i-network flashweave_2.qza  \
    --m-metadata-file flashweave-louvain-2.qza \
    --m-metadata-file corrected_taxonomy.qza \
    --o-visualization bacteria_network.qzv

My last try was to run with a --m-metada-file sample metadata with my metadata file, but I've received an error:

qiime makarsa visualise-network \
>     --i-network flashweave_2.qza  \
>     --m-metadata-file flashweave-louvain-2.qza \
>     --m-metadata-file corrected_taxonomy.qza \
>  --m-metadata-file Metadata.txt \
>     --o-visualization bacteria_network_2.qzv

There was an issue with merging QIIME 2 Metadata:

  Cannot merge because there are no IDs shared across metadata objects.

BenKaehler · June 23, 2023, 9:12pm

Hi @joaomiranda,

So some of these problems are easy and some of them are more ... open ended. I'll start with the easy ones.

Degree, Betweenness, Closeness, and Assortativity are statistics that can be calculated for each node, and say something about each node's position in the network. Intuitively they should correspond to some notion of node importance. If they're useful, that's great, if they're not, you can ignore them. If you want to read more about what they mean, they were all calculated using Networkx. Here is the documentation for degree, betweenness, closeness, and assortativity. (Also, it says "Associativity" in the visualisation, but it should say "Assortativity". If you download and reinstall the plug-in it should say Assortativity now.)

The taxonomy file you provided has 214 different taxa in it. If you ask the visualisation for, for example, "Taxon Level 3", it will truncate taxonomic classifications at Class.

Metadata.txt didn't work with visualise-network because it's metadata about samples. visualise-network needs metadata about features, like taxonomy. You can use sample metadata with FlashWeave, though, but let's address some of your bigger questions first.

First I'll say briefly what these networks mean. Both SpiecEasi and FlashWave try to infer undirected probabilistic graphical models. Their authors may or may not take issue with them being called "correlation networks", but I am not an author of either. Strictly, a graphical model tries to capture a dependence structure. Loosely, a graphical model says that if a node knows what its neighbours are doing, it doesn't care about the rest of the network.

Next, being a network inference and visualisation tool, q2-makarsa isn't really the right tool for asking questions like "which bacteria are from which mosquito species or sample?". In fact, most of these bacterial species appear in several or all of the mosquito species. A better question might be "which bacterial species are deferentially abundant in which mosquito species?". There are several tools for attempting to answer that question directly, like ANCOM, ANCOM-BC, or ALDEX2 (thanks @Nicholas_Bokulich).

However, being a network inference and visualisation tool, q2-makarsa might be well-equipped to answer the question "which bacterial species are directly affected by changing mosquito species, and which ones, even they might be differentially abundant, are actually bystanders?".

It turns out that FlashWeave gives us tools for answering that question by allowing you to introduce "meta-variables" into the network that correspond to mosquito species. That is, if the FlashWeave infers a positive edge between the meta-variable corresponding to, for example, Ae. albopictus and Candidatus Nanopelagicaceae, it means that it thinks that if the species is Ae. albopictus, then Candidatus Nanopelagicaceae counts are likely to be greater than they otherwise might be. Conversely, if it infers a negative edge between Ae. albopictus and Aphanothecaceae, then it thinks that if the species is Ae. albopictus, it is likely that we will see fewer Aphanothecaceae. I would encourage you to have a read through the FlashWeave paper to get a better understanding of meta-variables. As an aside, SpiecEasi is capable of something similar, but that functionality will require extra work (by me) to expose in q2-makarsa.

So, I tried it out on your data by running

qiime makarsa flashweave --i-table filtered-bacteria-table.qza --m-metadata-file Metadata.txt --o-network network-with-metadata.qza
qiime makarsa visualise-network --i-network network-with-metadata.qza --m-metadata-file corrected_taxonomy.qza --o-visualization network-with-metadata.qzv

The result follows. In this graph, which can be obtained by selecting color "mv" (for meta-variable), the dark blue nodes are bacterial species, and the light blue nodes are meta-variables. Three of them are the mosquito species for which you had multiple samples. The fourth is a location. If you care, you could eliminate the location node by removing the location column from your metadata. It doesn't affect the associations between bacterial and mosquito species. I know, the colours aren't great and I will try to make the contrast better in forthcoming releases.

Here is the visualisation so that you can view it yourself. Once you identify the mosquito species you can interrogate their neighbours by selecting them.

network-with-metadata.qzv (219.1 KB)

So, please take a look. I encourage you to also run a differential abundance analysis if you haven't yet, and have a look at which of the taxa that were associated with mosquito species by FlashWeave were inferred to be differentially abundant. For instance Psychromonadaceae and Chroococcidiopsidaceae might be significantly more abundant in Ae. aegypti, but the above network would tell you that in fact Chroococcidiopsidaceae levels are mostly just elevated because Psychromonadaceae levels are elevated, and Chroococcidiopsidaceae depends more on Psychromonadaceae than whether it's in Ae. aegypti.

Ok, that's a lot, so I'll stop here. Please follow up if you would like any clarification.

joaomiranda · June 27, 2023, 7:31pm

Hi @BenKaehler ,

Thank you so much for your attention and insights!

I'll go deep into the paper of Flashweave, and try to interpret better my network.

A last question, my meta variables, specifically the 'mosquito species'. Why they are connected with three bacterial taxa? It's a coincidence? Because I expected a bit more connections...

By the way, I have differencial abundance data, I used ANCOM. And with a quick look I saw some interesting microbial abundances associated with the interconnection of the nodes in my network.

BenKaehler · June 28, 2023, 9:29pm

Thanks @joaomiranda,

Do you mean is it a coincidence that Ae. aegypti and Ae. taeniorhynchus are both connected to exactly three bacterial species? Yes, that's a coincidence.

As for why they're not connected to more, FlashWeave tries to only associate nodes of the network that are directly dependent on one another. It doesn't mean that a mosquito node only affects its neighbours, it just suggests that the effects that it has on the broader community might be through its neighbours.

To try to visualise that, here is an illustration of where the mosquito nodes fall in their Louvain communities.

In this plot, the mosquito nodes are in an "undefined" community. This is a result of a QIIME 2 issue (but it's also convenient, because it means you can see where they are). They are not actually in undefined communities, they're in the communities that they're surrounded by, as you can see by visualising the Louvain communities directly (mozzies-louvain.qzv (1.2 MB)).

You might notice that this network looks slightly different to the last one. That's because I changed the FlashWeave parameters slightly to stop it ignoring features that didn't occur in at least 20 samples (which is almost all your samples). I also removed the location data from your metadata to focus on the mosquito species.

When performing any bioinformatics it is important to vary the input parameters to demonstrate that your conclusions are robust to parameter choices.

If you're seeing corroborating evidence in your ANCOM results, that's great. I would like to see the results if you publish them.

Here are the commands that I used to generate the above results.

Remove the locations from the metadata.

cut -f1,2 -d$'\t' Metadata.txt > mozzies-only.tsv

Generate the network with looser observations constraints and using only mosquito metadata.

qiime makarsa flashweave \
    --i-table filtered-bacteria-table.qza \
    --m-metadata-file mozzies-only.tsv \
    --p-n-obs-min 2 \
    --o-network mozzies-network.qza

Infer the Louvain communities.

qiime makarsa louvain-communities \
    --i-network mozzies-network.qza \
    --o-community mozzies-louvain.qza

Visualise the Louvain communities (as attached above).

qiime metadata tabulate \
    --m-input-file mozzies-louvain.qza \
    --o-visualization mozzies-louvain.qzv

Visualise the resulting network, with Louvain communities and taxonomic classifications

qiime makarsa visualise-network \
    --i-network mozzies-network.qza \
    --m-metadata-file corrected_taxonomy.qza \
    --m-metadata-file mozzies-louvain.qza \
    --o-visualization mozzies-network.qzv

Here is the resulting visualisation: mozzies-network.qzv (223.4 KB).

Hope that helps. Feel free to follow up if you have any more questions.