About this guide

日本語訳

繁體中文翻譯

简体中文翻译

This document is a guide to developing the Erg project. It describes how to make contributions, guidelines for development, and how to set up the development environment, etc.

An explanation of compiler internals is here.

Branch naming and operation policy

main

  • main development branch (trunk-based development)

  • The following conditions must be met

  • Compile succeeds

beta (not created at this time)

  • Latest beta release

  • The following conditions must be met

  • Compile succeeds

  • All tests passed

feature-(name)

  • A branch that develops one specific feature

  • No conditions

issue-(#issue)

  • branch that resolves a specific issue

  • No condition

fix-(#issue or bug name)

  • branch that fixes a specific bug (If the issue is a bug, please create this instead of issue-*)

  • No condition

erg build features

debug

Enter debug mode. As a result, the behavior inside Erg is sequentially displayed in the log. Also, enable backtrace_on_stack_overflow. Independent of Rust's debug_assertions flag.

backtrace

Enable only backtrace_on_stack_overflow.

japanese

Set the system language to Japanese. Erg internal options, help (help, copyright, license, etc.) and error display are guaranteed to be Japanese.

simplified_chinese

Set the system language to Simplified Chinese. Erg internal options, help (help, copyright, license, etc.) and errors are displayed in Simplified Chinese.

traditional_chinese

Set the system language to Traditional Chinese. Erg internal options, help (help, copyright, license, etc.) and errors are displayed in Traditional Chinese.

unicode/pretty

The compiler makes the display rich.

pre-commit

Used to run tests in pre-commit. It's a bug workaround.

large_thread

Increase the thread stack size. Used for Windows execution and test execution.

els

--language-server option becomes available. erg --language-server will start the Erg language server.

py_compatible

Enable Python-compatible mode, which makes parts of the APIs and syntax compatible with Python. Used for pylyzer.

Erg repository directory structure

  └─┬ assets: images, etc.
    ├─ CODE_OF_CONDUCT: Code of Conduct
    ├─┬ crates
    │ ├─ els: Erg Language Server
    │ ├─ erg_common: common utility
    │ ├─ erg_compiler: Compiler, **core of Erg**
    │ └─ erg_parser: Parser
    ├─┬doc
    │ ├─┬ EN
    │ │ ├─ API: Erg standard API
    │ │ ├─ compiler: About compiler implementation
    │ │ ├─ dev_guide: Guide for developers and contributors
    │ │ ├─ python: Python knowledge required for Erg development
    │ │ ├─ syntax: Erg syntax
    │ │ └─ tools: Erg command line tools
    │ └─┬ JA
    │ ...
    ├─ examples: sample code
    ├─ library: Erg script library
    ├─ src: directory where main.rs and driver are placed
    └─ tests: test code

format

Any document that does not follow the rules below is subject to correction.

  • Write code comments or internal documentation in a certain tone.
  • Documents to be shown to the outside (general users) should be written more and more.
  • Always include definitions, meanings, or links to terms that appear for the first time in the document.
  • Use parentheses as a proviso only for sentences that are supplementary but necessary for understanding the main text, and use footnotes for sentences that are not essential for understanding the main text1</ sup>.
  • If the content of the document is outdated, update it according to this method.
  • Files under syntax should have sequential numbers at the beginning of their file names, except for those not included in the document.
    • Scripts that automatically insert and replace files exist in doc/script.

1 See this for how to write footnotes. ↩

Embedding the Erg compiler in your application

It is easy to embed Erg in your application.

[dependencies]
erg = "0.5.12" # choose latest version

use erg::DummyVM;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut vm = DummyVM::default();
    let _res: String = vm.eval("print! \"Hello, world!\"")?;
    Ok(())
}

Python is required for execution.

There is also a stand-alone compiler version that is not connected to the runtime.

[dependencies]
erg_compiler = "0.5.12" # choose latest version

use erg_compiler::Compiler;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut compiler = Compiler::default();
    let code = compiler.compile("print!\"Hello, world!\"", "exec")?;
    code.dump_as_pyc("o.pyc", None)?;
    Ok(())
}

Compiler outputs a structure called CodeObj. This is generally not very useful, so you may want to use Transpiler, which outputs a Python script.

use erg_compiler::Transpiler;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut transpiler = Transpiler::default();
    let script = transpiler.transpile("print!\"Hello, world!\"", "exec")?;
    println!("{}", script.code);
    Ok(())
}

Other examples are HIRBuilder which outputs HIR (high-level intermediate representation) and ASTBuilder which outputs AST (abstract syntax trees).

use erg_compiler::HIRBuilder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = HIRBuilder::default();
    let artifact = builder.build("print!\"Hello, world!\"", "exec")?;
    println!("{}", artifact.hir);
    Ok(())
}

use erg_compiler::ASTBuilder;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut builder = ASTBuilder::default();
    let ast = builder.build("print! \"Hello, world!\")")?;
    println!("{}", ast);
    Ok(())
}

The structure that performs the semantic analysis implements a trait called ContextProvider. It can obtain information about variables in the module, etc.

use erg_compiler::Transpiler;
use erg_compiler::context::ContextProvider;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut transpiler = Transpiler::default();
    let script = transpiler.transpile("i = 0", "exec")?;
    println!("{}", script.code);
    let typ = transpiler.get_var_info("i").0.t;
    println!("{typ}");
    Ok(())
}

Development environment

What you need to install

  • Rust (installed with rustup)

    • ver >= 1.64.0
    • 2021 edition

  • pre-commit

We use pre-commit to have clippy check and test automatically. The checks may fail on the first run even if there are no bugs, in which case you should try committing again.

  • Python3 interpreter (3.7~3.11)

If you want to check the behavior of Erg in various versions, it is recommended to install such as pyenv.

Recommendation

  • Editor: Visual Studio Code
  • VSCode extensions: Rust-analyzer, GitLens, Git Graph, GitHub Pull Requests and Issues, Markdown All in One, markdownlint
  • OS: Windows 10/11 | Ubuntu 20.04/22.04 | macOS Monterey/Ventura

Multilingualization of Messages

Erg is making its messages (start, options, documentation, hints, warnings, error messages, etc.) multilingual. You don't need detailed knowledge of Rust or Erg to participate in this project. We appreciate your cooperation.

The method for multilingualization is explained below.

Look for switch_lang!

Find the entry switch_lang! in the Erg source code (use grep or your editor's search function). You should find something like this:


#![allow(unused)]
fn main() {
switch_lang!(
    "japanese" => format!("This feature ({name}) is not officially available yet"),
    "english" => format!("this feature({name}) is not implemented yet"),
),
}

This message is currently supported in Japanese and English only. Let's try adding a simplified Chinese message.

add a message

Add translated messages while viewing content in other languages. Don't forget the comma (,) at the end.


#![allow(unused)]
fn main() {
switch_lang!(
    "japanese" => format!("This feature ({name}) is not officially available yet"),
    "simplified_chinese" => format!("This function ({name}) has been officially provided"),
    "english" => format!("this feature({name}) is not implemented yet"),
),
}

Note that English is the default and should always come last. The {name} part is Rust's formatting feature that allows you to embed the contents of a variable (name) into a string.

Build

Now let's build with the --features simplified_chinese option.

screenshot_i18n_messages

You did it!

FAQs

Q: What does a specification like {RED}{foo}{RESET} mean? A: Everything after {RED} is displayed in red. {RESET} restores the color.

Q: If I want to add my own language, how do I replace the "simplified_chinese" => part? A: We currently support the following languages:

  • "english" (default)
  • "japanese" (Japanese)
  • "simplified_chinese" (Simplified Chinese)
  • "traditional_chinese" (Traditional Chinese)

If you would like to add languages ​​other than these, please make a request.

Guidelines for Rust code

local rules

  • Use log! for output for debugging (use println! etc. for output processing that is also necessary for release).
  • Unused or internal variables/methods (private and used only for specific functions) must be prefixed with _. If you want to avoid conflicts with reserved words, add one _ to the end.
  • Use clippy. However, some rules are not reasonable, so you can ignore rules other than deny by using #[allow(clippy::...)].
  • Define and use domain-specific Enums instead of numeric enumerations or bools.
  • Keep access modifiers to a minimum. Prioritize using pub(mod) or pub(crate) even when publishing.
  • Convert an iterable object in a for expression explicitly to an iterator (for i in x.iter() instead of for i in x).
  • Lazy evaluation. For example, if default is non-literal, use unwrap_or_else instead of unwrap_or.

Code not encouraged

  • Make heavy use of return type overloading. Specifically code that uses a lot of non-obvious .into. This is because type inference results can be counter-intuitive. In this case it is recommended to use from instead.
  • Make heavy use of Deref. This effectively poses the same problem as inheritance.

Code that makes decisions based on context

  • Define unused helper methods.
  • Use unwrap and clone a lot. In some cases there is nothing better than doing so.

Dependencies

Dependencies should be minimized as much as possible, and those that are necessary should be implemented by the Erg developing team. External dependencies are allowed only if they are extremely difficult to implement or hardware-dependent (e.g. libc, winapi), or crate without external dependencies may be used (e.g. unicode-xid). Otherwise, they may be allowed as optional dependencies (e.g. https client). In all cases, well-maintained and widely-used ones should be selected.

This rule applies only to the Erg compiler; Erg tools and libraries are free to add their own dependencies.

test

Testing is an important part of ensuring code quality.

Execute the test with the following command.

cargo test --features large_thread

Since cargo takes a small thread for running tests, we use the large_thread flag to avoid stack overflows.

Placement of tests

Arrange them according to the implemented features. Place parser tests under erg_parser/tests, compiler (type checker, etc.) tests under erg_compiler/tests, language feature tests that users can use directly under erg/tests (However, the tests are currently in development and are not necessarily arranged according to this convention).

How to write tests

There are two types of tests. A positive test and a negative test. A positive test is a test to check whether the compiler operates as intended, and a negative test is a test to check whether the compiler properly outputs an error for invalid input. Due to the nature of programming language processors, among all software, they are particularly susceptible to invalid input, and errors must always be presented to the user, so the latter must also be taken care of.

If you add a new feature to the language, you need to write at least one positive test. Also, please write negative tests if possible.

#[ignore] attribute

The Erg development team recommends pre-commit. This prevents bugs from getting into the code by running tests before each commit, but some tests are time-consuming and slow down the commit.

Therefore, tests that are heavy or have a low probability of failure are marked with the #[ignore] attribute. Tests with the #[ignore] attribute are not run by cargo test, but can be run with cargo test -- --include-ignored. These tests are run by CI and do not need to be run on the local PC.

Troubleshooting

Q: Local builds succeed, but GitHub Actions builds fail

A: The branch you are working on may not be following the changes in main.

Q: The pre-commit check fails

A: Try committing again. It may fail the first time. If it fails again and again, the code may contain a bug.

A: Make sure cargo is not running in another process.

Q: build.rs fails to run

A: Check for extra files/directories (such as __pychache__) on the directory where build.rs runs.

Versioning

The Erg compiler assigns version numbers according to semantic versioning. However, during version 0, different rules than usual are applied (following more detailed rules than semantic versioning). It is important to note that there are two types of compatibility in Erg. One is specification compatibility, which indicates compatibility with the language specification, and the other is internal compatibility, which indicates compatibility with (public) APIs such as compilers.

  • During version 0, specification and internal compatibility may be broken in minor releases. This is the same as normal semantic versioning.
  • During version 0, patch releases do not break specification compatibility, but there is no guarantee of internal compatibility.
  • New features are mainly added in minor releases, but can also be added in patch releases if they are trivial language features or compiler features.

Release Cycle

  • Patch releases are made about once every 1~2 weeks. However, if a serious bug is discovered, it may be moved up.
  • Minor releases are made about 10 times as often as patch releases, i.e., once every 3~6 months.
  • Major releases are made at indefinite intervals. The schedule for version 1 release is not planned at this time.

Nightly/Beta Releases

Erg will make nightly and beta releases on an irregular intervals. nightly releases are pre-releases of new patch releases, and beta releases are pre-releases of new minor/major releases. Nightly and beta versions are published on crates.io, and beta versions are also published on GitHub releases.

The format of the nightly version is 0.x.y-nightly.z. The same is true for beta versions.

Nightly releases are made almost every day (no release are made if no changes), while beta releases are made irregularly. However, once a beta release is released, a new beta release is released almost every day.