{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# NMODL Python Interface Tutorial\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Introduction\n", "\n", "NMODL is a code generation framework for NEURON Modeling Language. It is primarily designed to support optimised code generation backends for modern architectures including CPUs and GPUs. It provides high level Python interface that can be used for model introspection as well as performing various analysis on underlying model.\n", "\n", "This tutorial provides introduction to Python API with examples." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To get started, let's install nmodl via Python wheel as:" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%%capture\n", "! pip install nmodl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "> Note : Python wheel is only available for Linux and Mac OS. Windows version will be available in the future. Today you can use Windows Subsystem for Linux. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Parsing Model And Constructing AST\n", "\n", "Once the NMODL is setup properly we should be able to import `nmodl` module :" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import nmodl.dsl as nmodl" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you see any issues, check the [installation section](#installation). Lets take an example of a channel `CaDynamics` :" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "channel = \"\"\"\n", "NEURON {\n", " SUFFIX CaDynamics\n", " USEION ca READ ica WRITE cai\n", " RANGE decay, gamma, minCai, depth\n", "}\n", "\n", "UNITS {\n", " (mV) = (millivolt)\n", " (mA) = (milliamp)\n", " FARADAY = (faraday) (coulombs)\n", " (molar) = (1/liter)\n", " (mM) = (millimolar)\n", " (um) = (micron)\n", "}\n", "\n", "PARAMETER {\n", " gamma = 0.05 : percent of free calcium (not buffered)\n", " decay = 80 (ms) : rate of removal of calcium\n", " depth = 0.1 (um) : depth of shell\n", " minCai = 1e-4 (mM)\n", "}\n", "\n", "ASSIGNED {ica (mA/cm2)}\n", "\n", "INITIAL {\n", " cai = minCai\n", "}\n", "\n", "STATE {\n", " cai (mM)\n", "}\n", "\n", "BREAKPOINT { SOLVE states METHOD cnexp }\n", "\n", "DERIVATIVE states {\n", " cai' = -(10000)*(ica*gamma/(2*FARADAY*depth)) - (cai - minCai)/decay\n", "}\n", "\n", "FUNCTION foo() {\n", " LOCAL temp\n", " foo = 1.0 + gamma\n", "}\n", "\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can parse any valid NMODL constructs using NMODL's parsing interface. First, we have to create nmodl parser object using `nmodl::NmodlDriver` and then we can use `parse_string` method :" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "driver = nmodl.NmodlDriver()\n", "modast = driver.parse_string(channel)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `parse_string` method will throw an exception if input is invalid. Otherwise it returns [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) object." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we simply print AST object, we can see JSON representation :" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{\"Program\":[{\"NeuronBlock\":[{\"StatementBlock\":[{\"Suffix\":[{\"Name\":[{\"String\":[{\"name\":\"SUFFIX\"}]}]},{\"Name\":[{\"String\":[{\"name\":\"CaDynamics\"}]}]}]},{\"Useion\":[{\"Name\":[{\"String\":[{\"name\":\"ca\"}]}]},{\"R\n" ] } ], "source": [ "print(\"%.200s\" % modast) # only first 200 characters" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Querying AST object with Visitors\n", "\n", "One of the strength of NMODL python interface is access to inbuilt [Visitors](https://en.wikipedia.org/wiki/Visitor_pattern). One can perform different queries and analysis on AST using different visitors. Lets start with the examples of inbuilt visitors." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Lookup visitor\n", "\n", "As name suggest, lookup visitor allows to search different NMODL constructs in the AST. The `visitor` module provides access to inbuilt visitors. In order to use this visitor, we create an object of AstLookupVisitor :" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from nmodl import ast, visitor\n", "\n", "lookup_visitor = visitor.AstLookupVisitor()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Assuming we have created AST object (as shown [here](#create-ast)), we can search for any NMODL construct in the AST using AstLookupVisitor. For example, to find out `STATE` block in the mod file, we can simply do:" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "STATE {\n", " cai (mM)\n", "}\n" ] } ], "source": [ "states = lookup_visitor.lookup(modast, ast.AstNodeType.STATE_BLOCK)\n", "for state in states:\n", " print(nmodl.to_nmodl(state))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We have used `to_nmodl` helper function to convert AST object back to NMODL language. Note that the output of `to_nmodl` should be same as input except for comments (?, :, COMMENT). There are very few edge cases where the NMODL output could slightly differ and this is considered as bug. This is being addressed by testing entire [ModelDB](https://senselab.med.yale.edu/modeldb/) database.\n", "\n", "Using AstLookupVisitor we can introspect NMODL constructs at any level of details. Below are some examples to find out different constructs in the example mod file. All different kind of NMODL constructs are [listeed here](https://bluebrain.github.io/nmodl/html/doxygen/group__ast__type.html)." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1 differential equation(s) exist : \n", "\t cai' = -(10000)*(ica*gamma/(2*FARADAY*depth))-(cai-minCai)/decay \n", "1 prime variables exist : cai'\n", "4 range variables exist : decay gamma minCai depth\n", "4 parameter variables exist : decay gamma minCai depth\n", "17 units uses : (mV) (millivolt) (mA) (milliamp) (faraday) (coulombs) (molar) (1/liter) (mM) (millimolar) (um) (micron) (ms) (um) (mM) (mA/cm2) (mM)" ] } ], "source": [ "odes = lookup_visitor.lookup(modast, ast.AstNodeType.DIFF_EQ_EXPRESSION)\n", "primes = lookup_visitor.lookup(modast, ast.AstNodeType.PRIME_NAME)\n", "range_vars = lookup_visitor.lookup(modast, ast.AstNodeType.RANGE_VAR)\n", "parameters = lookup_visitor.lookup(modast, ast.AstNodeType.PARAM_ASSIGN)\n", "units = lookup_visitor.lookup(modast, ast.AstNodeType.UNIT)\n", "\n", "if odes:\n", " print(\"%d differential equation(s) exist : \" % len(odes))\n", " for ode in odes:\n", " print(\"\\t %s \" % nmodl.to_nmodl(ode))\n", "\n", "if primes:\n", " print(\"%d prime variables exist :\" % len(primes), end=\"\")\n", " for prime in primes:\n", " print(\" %s\" % nmodl.to_nmodl(prime), end=\"\")\n", " print()\n", "\n", "if range_vars:\n", " print(\"%d range variables exist :\" % len(range_vars), end=\"\")\n", " for range_var in range_vars:\n", " print(\" %s\" % nmodl.to_nmodl(range_var), end=\"\")\n", " print()\n", "\n", "if parameters:\n", " print(\"%d parameter variables exist :\" % len(parameters), end=\"\")\n", " for range_var in range_vars:\n", " print(\" %s\" % nmodl.to_nmodl(range_var), end=\"\")\n", " print()\n", "\n", "if units:\n", " print(\"%d units uses :\" % len(units), end=\"\")\n", " for unit in units:\n", " print(\" %s\" % nmodl.to_nmodl(unit), end=\"\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Apart from performing lookup on whole AST object (i.e. entire NMODL file), we can perform analysis on the specific construct. Lets take a synthetic example : say we want to find out all assignment statements in function `foo`. If we use lookup visitor on AST, it will returnn all statements in the mod file (e.g. including DERIVATIVE block). To avoid this, we can first find FUNCTION blocks and then search for statements within that block :" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "foo = 1+gamma\n" ] } ], "source": [ "functions = lookup_visitor.lookup(modast, ast.AstNodeType.FUNCTION_BLOCK)\n", "function = functions[0] # first function\n", "\n", "# expression statements include assignments\n", "new_lookup_visitor = visitor.AstLookupVisitor(ast.AstNodeType.EXPRESSION_STATEMENT)\n", "\n", "# using accept method of node we can visit it\n", "function.accept(new_lookup_visitor)\n", "statements = new_lookup_visitor.get_nodes()\n", "\n", "for statement in statements:\n", " print(nmodl.to_nmodl(statement))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Every AST node provides `accept` method that takes visitor object as parameter. In above example, we used `accept` method of `FunctionBock` node. This allows to run a given visitor on a specific node. `AstLookupVisitor` provides `get_nodes()` method that can be used to retrive the result of visitor. List of all `AstNodeType` can be found [here (todo)](####)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Symbol Table Visitor\n", "\n", "Symbol table visitor is used to find out all variables and their usage in mod file. To use this, first create a visitor object as: " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "from nmodl import symtab\n", "\n", "symv = symtab.SymtabVisitor()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the visitor object is created, we can run visitor on AST object to populate symbol table. Symbol table provides print method that can be used to print whole symbol table : " ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "------------------------------------------------------------------------------------------------------------------------------------\n", "| NMODL_GLOBAL [Program IN None] POSITION : UNKNOWN SCOPE : GLOBAL |\n", "------------------------------------------------------------------------------------------------------------------------------------\n", "| NAME | PROPERTIES | STATUS | LOCATION | VALUE | # READS | # WRITES | \n", "------------------------------------------------------------------------------------------------------------------------------------\n", "| ca | ion | | UNKNOWN | | 0 | 0 | \n", "| ica | assigned_definition read_ion | | UNKNOWN | | 0 | 0 | \n", "| cai | prime_name assigned_definition write_ion state_var | | UNKNOWN | | 0 | 0 | \n", "| decay | range parameter | | UNKNOWN | 80.000000 | 0 | 0 | \n", "| gamma | range parameter | | UNKNOWN | 0.050000 | 0 | 0 | \n", "| minCai | range parameter | | UNKNOWN | 0.000100 | 0 | 0 | \n", "| depth | range parameter | | UNKNOWN | 0.100000 | 0 | 0 | \n", "| mV | unit_def | | UNKNOWN | | 0 | 0 | \n", "| mA | unit_def | | UNKNOWN | | 0 | 0 | \n", "| FARADAY | factor_def | | 11.5-11 | | 0 | 0 | \n", "| molar | unit_def | | UNKNOWN | | 0 | 0 | \n", "| mM | unit_def | | UNKNOWN | | 0 | 0 | \n", "| um | unit_def | | UNKNOWN | | 0 | 0 | \n", "| states | derivative_block to_solve | | 36.1-10 | | 0 | 0 | \n", "| foo | function_block | | 40.1-8 | | 0 | 0 | \n", "------------------------------------------------------------------------------------------------------------------------------------\n", "\n", " --------------------------------------------------------------------------------------------------\n", " | StatementBlock4 [StatementBlock IN foo] POSITION : 40.16 SCOPE : LOCAL |\n", " --------------------------------------------------------------------------------------------------\n", " | NAME | PROPERTIES | STATUS | LOCATION | VALUE | # READS | # WRITES | \n", " --------------------------------------------------------------------------------------------------\n", " | temp | local | | UNKNOWN | | 0 | 0 | \n", " --------------------------------------------------------------------------------------------------\n", "\n" ] } ], "source": [ "symv.visit_program(modast)\n", "table = modast.get_symbol_table()\n", "table_s = str(table)\n", "\n", "print(table_s)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we can query for variables in the symbol table based on name of variable: " ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "cai [Properties : prime_name assigned_definition write_ion state_var]\n" ] } ], "source": [ "cai = table.lookup(\"cai\")\n", "print(cai)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "When we print the symbol, all it's properties get printed. For example, in above case the `cai` variable is used in :\n", "* differential equation (prime)\n", "* state block\n", "* assigned block\n", "* use ion statement\n", "\n", "We can also query based on the kind of variables. For example, to find out all range variables we can use `get_variables_with_properties` method with symbol property as an argument:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "decay [Properties : range parameter]\n", "gamma [Properties : range parameter]\n", "minCai [Properties : range parameter]\n", "depth [Properties : range parameter]\n" ] } ], "source": [ "range_vars = table.get_variables_with_properties(symtab.NmodlType.range_var)\n", "for var in range_vars:\n", " print(var)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can also query with multiple properties. For example, to find out read or write ion variables we can use (second argument `False` indicates any one property):" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "ica [Properties : assigned_definition read_ion]\n", "cai [Properties : prime_name assigned_definition write_ion state_var]\n" ] } ], "source": [ "ions_var = table.get_variables_with_properties(\n", " symtab.NmodlType.read_ion_var | symtab.NmodlType.write_ion_var, False\n", ")\n", "for var in ions_var:\n", " print(var)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Custom AST visitor\n", "\n", "If predefined visitors are limited, we can implement new visitor using AstVisitor interface. Lets say we want to implement a visitor that will print every floating point number in MOD file. Here is how we can do this: " ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "0.05\n", "0.1\n", "0.0001\n", "10000.0\n", "2.0\n", "1.0\n" ] } ], "source": [ "from nmodl import ast, visitor\n", "\n", "\n", "class DoubleVisitor(visitor.AstVisitor):\n", " def visit_double(self, node):\n", " print(node.eval()) # or, can use nmodl.to_nmodl(node)\n", "\n", "\n", "d_visitor = DoubleVisitor()\n", "modast.accept(d_visitor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The `AstVisitor` base class provides all necessary methods to traverse different ast nodes. New visitors will inherit from `AstVisitor` and implement only those method where we want different behaviour. For example, in the above case we want to visit ast nodes of type `Double` and print their value. To achieve this we implemented associated method of `Double` node i.e. `visit_double`. When we call `accept` method on the ast object, the entire AST tree will be visited (by `AstVisitor`). But whenever double node type will encounter in AST, the control will be handed back to `DoubleVisitor.visit_double` method. \n", "\n", "Lets implement the example of lookup visitor to print parameters with values :" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "gamma\n", "0.05\n", "decay\n", "80\n", "depth\n", "0.1\n", "minCai\n", "0.0001\n" ] } ], "source": [ "class ParameterVisitor(visitor.AstVisitor):\n", " def __init__(self):\n", " visitor.AstVisitor.__init__(self)\n", " self.in_parameter = False\n", "\n", " def visit_param_block(self, node):\n", " self.in_parameter = True\n", " node.visit_children(self)\n", " self.in_parameter = False\n", "\n", " def visit_name(self, node):\n", " if self.in_parameter:\n", " print(nmodl.to_nmodl(node))\n", "\n", " def visit_double(self, node):\n", " if self.in_parameter:\n", " print(node.eval())\n", "\n", " def visit_integer(self, node):\n", " if self.in_parameter:\n", " print(node.eval())\n", "\n", "\n", "param_visitor = ParameterVisitor()\n", "modast.accept(param_visitor)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Easy code generation using AST visitors\n", "\n", "With a little more code we can even create a code generator for python using a the visitor pattern." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "mfunc_src = \"\"\"FUNCTION myfunc(x, y) {\n", " if (x < y) {\n", " myfunc = x + y\n", " } else {\n", " myfunc = y\n", " }\n", "}\n", "\"\"\"\n", "import nmodl.dsl as nmodl\n", "from nmodl.dsl import ast\n", "\n", "driver = nmodl.NmodlDriver()\n", "mfunc_ast = driver.parse_string(mfunc_src)" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "from nmodl.dsl import ast, visitor\n", "\n", "\n", "class PyGenerator(visitor.AstVisitor):\n", " def __init__(self):\n", " visitor.AstVisitor.__init__(self)\n", " self.pycode = \"\"\n", " self.indent = 0\n", " self.func_name = \"\"\n", "\n", " def visit_function_block(self, node):\n", " params = []\n", " self.func_name = node.get_node_name()\n", " for p in node.parameters:\n", " params.append(p.get_node_name())\n", " params_str = \", \".join(params)\n", " self.pycode += f\"def {node.get_node_name()}({params_str}):\\n\"\n", " node.visit_children(self)\n", "\n", " def visit_statement_block(self, node):\n", " self.indent += 1\n", " node.visit_children(self)\n", " self.indent -= 1\n", "\n", " def visit_expression_statement(self, node):\n", " self.pycode += \" \" * 4 * self.indent\n", " expr = node.expression\n", " if type(expr) is ast.BinaryExpression and expr.op.eval() == \"=\":\n", " rhs = expr.rhs\n", " lhsn = expr.lhs.name.get_node_name()\n", " if lhsn == self.func_name:\n", " self.pycode += \"return \"\n", " rhs.accept(self)\n", " else:\n", " node.visit_children(self)\n", " else:\n", " node.visit_children(self)\n", " self.pycode += \"\\n\"\n", "\n", " def visit_if_statement(self, node):\n", " self.pycode += \" \" * 4 * self.indent + \"if \"\n", " node.condition.accept(self)\n", " self.pycode += \":\\n\"\n", " node.get_statement_block().accept(self)\n", " for n in node.elseifs:\n", " n.accept(self)\n", " if node.elses:\n", " node.elses.accept(self)\n", "\n", " def visit_else_statement(self, node):\n", " self.pycode += \" \" * 4 * self.indent + \"else:\\n\"\n", " node.get_statement_block().accept(self)\n", "\n", " def visit_wrapped_expression(self, node):\n", " self.pycode += \"(\"\n", " node.visit_children(self)\n", " self.pycode += \")\"\n", "\n", " def visit_binary_expression(self, node):\n", " lhs = node.lhs\n", " rhs = node.rhs\n", " op = node.op.eval()\n", " if op == \"^\":\n", " self.pycode += \"pow(\"\n", " lhs.accept(self)\n", " self.pycode += \", \"\n", " rhs.accept(self)\n", " self.pycode += \")\"\n", " else:\n", " lhs.accept(self)\n", " self.pycode += f\" {op} \"\n", " rhs.accept(self)\n", "\n", " def visit_var_name(self, node):\n", " self.pycode += node.name.get_node_name()\n", "\n", " def visit_integer(self, node):\n", " self.pycode += nmodl.to_nmodl(node)\n", "\n", " def visit_double(self, node):\n", " self.pycode += nmodl.to_nmodl(node)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "def myfunc(x, y):\n", " if x < y:\n", " return x + y\n", " else:\n", " return y\n", "\n" ] } ], "source": [ "pygen = PyGenerator()\n", "pygen.visit_program(mfunc_ast)\n", "print(pygen.pycode)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Example of CURIE information parsing\n", "\n", "In this example we show how ontology information from NEURON block can be parsed." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " REPRESENTS NCIT:C17145 (NCIT:C17145)\n", " REPRESENTS NCIT:C17008 (NCIT:C17008)\n", " USEION na READ ena WRITE ina REPRESENTS CHEBI:29101 (CHEBI:29101)\n", " USEION ca READ eca WRITE ina ()\n" ] } ], "source": [ "import nmodl.dsl as nmodl\n", "from nmodl.dsl import ast, visitor\n", "\n", "driver = nmodl.NmodlDriver()\n", "mod_string = \"\"\"\n", "NEURON {\n", " SUFFIX hx\n", " REPRESENTS NCIT:C17145 : sodium channel\n", " REPRESENTS NCIT:C17008 : potassium channel\n", " USEION na READ ena WRITE ina REPRESENTS CHEBI:29101\n", " USEION ca READ eca WRITE ina\n", " RANGE gnabar, gkbar, gl, el, gna, gk\n", "}\n", "\"\"\"\n", "modast = driver.parse_string(mod_string)\n", "\n", "lookup_visitor = visitor.AstLookupVisitor()\n", "\n", "ont_statements = lookup_visitor.lookup(modast, ast.AstNodeType.ONTOLOGY_STATEMENT)\n", "ions = lookup_visitor.lookup(modast, ast.AstNodeType.USEION)\n", "\n", "for statement in ont_statements:\n", " print(\n", " \" %s (%s)\" % (nmodl.to_nmodl(statement), nmodl.to_nmodl(statement.ontology_id))\n", " )\n", "\n", "for ion in ions:\n", " o_id = nmodl.to_nmodl(ion.ontology_id) if ion.ontology_id else \"\"\n", " print(\" %s (%s)\" % (nmodl.to_nmodl(ion), o_id))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.7" } }, "nbformat": 4, "nbformat_minor": 2 }