Pharo Tree-Sitter

This project provides Pharo bindings to Tree-sitter.
It allows you to analyze and highlight code using existing Tree-sitter parsers developed by the community.

• Pharo Tree-Sitter
  • Installation
  • Native Libraries Required
  • Automated Library Generation
    • How It Works
    • Required Tools for MacOS and Linux
    • Required Tools for Windows
  • Manual Installation
    • macOS
    • Building Tree-sitter Shared Library
    • Archlinux
  • Quick Example
  • Moose: TreeSitter to build FAST and Famix models
  • Utilities and Debugging
  • Pre-built Libraries (Temporary Solution)
• Create a new FASTLanguageMetamodel using PharoTreeSitter

Installation

To install it in Pharo, run the following script in a Playground:

Metacello new
  baseline: 'TreeSitter';
  repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src';
  load.

To add this project to your baseline you can copy paste:

spec
	baseline: 'TreeSitter'
	with: [ spec repository: 'github://Evref-BL/Pharo-Tree-Sitter:main/src' ]

Native Libraries Required

To function correctly, the following native libraries are required depending on your OS:

• libtree-sitter.{dll, so, dylib}: the core Tree-sitter runtime.
• tree-sitter-<language>.{dll, so, dylib}: the parser for the desired language.

For example, to use Tree-sitter with Python:

• libtree-sitter.dylib (core)
• tree-sitter-python.dylib (parser)

You can generate these manually or take advantage of our automated setup for some supported languages.

Automated Library Generation

We automated the generation and installation of libraries for a few selected languages (currently Python and TypeScript) to simplify the setup across platforms. The generated libraries are copied directly to the appropriate Pharo vm folders.

The automation covers:

• Tree-sitter core
• tree-sitter-python
• tree-sitter-typescript

How It Works

When a new parser or language is requested, the automation:

1. Checks if the required native library exists in the VM folder.
2. If not, looks in the local cloned repository.
3. If the repo isn’t cloned, it clones it.
4. Builds the required library using CMake + Ninja.
5. Copies it to the correct vm location.

This ensures libraries are only built when needed.

Note: Automation is only available for a limited set of languages for now.
Contributions are welcome! See the TreeSitter-Libraries package and create a new subclass like TreeSitter-NewLanguage.

Required Tools for MacOS and Linux

To make sure the automation process works, please ensure the following tools are installed on your system:

Tool	Role	Check Command
Git	Repository cloning	git --version
make	Build configuration	make --version

These are mandatory to clone the repos and generate the libraries on these operating systems. Also this automated process allows the tests on piplines to succeed.

However, it turns out that make command for TreeSitter is not supported on windows. But do not worry, we found an alternative to make, but you need to make sure that you have the correspondant tools on your machine, so the automation procces can function properly. Check the below section;

Required Tools for Windows

To make sure the automation process works, please ensure the following tools are installed on your system:

Tool	Role	Check Command
Git	Repository cloning	git --version
CMake	Build configuration	cmake --version
GCC	C/C++ compiler	gcc --version
Ninja	Build system	ninja --version

If any of these commands fail, install the corresponding tool before continuing.

Manual Installation

You can also manually install the required libraries and copy them to the appropriate Pharo VM path. Below is an example for macOS.

macOS

We recommand you to download the tree-sitter core library using HomeBrew.

brew install tree-sitter

Then, locate the libtree-sitter.dylib under the path /opt/homebrew/Cellar/tree-sitter/0.x.x/lib/libtree-sitter.dylib. Finally, check the possible lib location detected by Pharo using FFIMacLibraryFinder new paths in a playground. Both the core library and the parser library should be under any of these paths. Please move them there. For instance, move your python.dylib under /Pharo/vms/110-x64/Pharo.app/Contents/Resources/lib/ Pay attention !! Make sure that Pharo is closed when you move the libraries specially after modifications on the original project, for example adding a new api. Normally on Windows you cannot replace the existing dll (if you already generated it for the first time) unless Pharo is closed. But if you intend to make it on mac for example, you need to close it and reopen it so the new modifications are applied.

Building Tree-sitter Shared Library

To build the shared library for tree-sitter, you can follow the following script:

# assuming git, gcc, cmake, ninja installed via scoop
git clone https://github.com/tree-sitter/tree-sitter
git fetch origin v0.24.3
git checkout v0.24.3
cd tree-sitter
cd lib
mkdir build
cd build
cmake -G Ninja ..
ninja

Then move the libtree-sitter.dll file under the VM folder of Pharo.

Archlinux

The tree-sitter libs are available in the extra repository. So you only have to install tree-sitter and the grammar you wanna use with your package manager.

yay tree-sitter tree-sitter-grammars

Quick Example

Here is a basic usage example in Pharo:

parser := TSParser new.
pythonLanguage := TSLanguage python.
parser language: pythonLanguage.
string := 'print("Hello, I''m Python!") # comment'.
tree := parser parseString: string.
tree rootNode

Moose: TreeSitter to build FAST and Famix models

Mooseis a software analysis platform. It can manage two kinds of models:

• Famix been a dependency model
• FAST (Famix AST) been an AST representation of the language

It exist some utilities to help building a Famix model for a language supported by TreeSitter. You can find this project here: https://github.com/moosetechnology/TreeSitterFamixIntegration

And this project provides a way to easily create a FAST importer. You can find the documentation here: Documentation

Utilities and Debugging

This project provides a few utilities to help you debug a tree sitter parser or to help you with its manipulation.

Those utilises are documented here: Documentation

Pre-built Libraries (Temporary Solution)

If you face issues building the libraries, you can download pre-built libraries (for Windows and macOS) here: https://doi.org/10.5281/zenodo.15423234

Create a new FASTLanguageMetamodel using PharoTreeSitter

This library allows you to create a first version of any FASTLanguageMetamodel, in condition that this language is supported by treesitter and by PharoTreeSitter. To do that you can follow a detailed documentation here.