Skip to content

highlight: Fix crashes when resolving syntax includes - #4272

Open
Huge wants to merge 4 commits into
micro-editor:masterfrom
Huge:fix/syntax-include-crash
Open

Huge wants to merge 4 commits into
micro-editor:masterfrom
Huge:fix/syntax-include-crash

Conversation

@Huge

@Huge Huge commented Oct 20, 2026 •

Copy link
Copy Markdown

Two crashes in pkg/highlight while resolving include rules. Both can be hit with a custom syntax file in ~/.config/micro/syntax/.

1. An included syntax file fails to parse

resolveIncludesInDef() and resolveIncludesInRegion() ignore the error returned by ParseDef() and then use the Def it returned, which is nil on error. Example: a user css.yaml with an invalid regex. Opening a .css file shows the parse error as expected, but opening any .html file crashes micro, because html.yaml includes css:

Micro encountered an error: runtime.errorString runtime error: invalid memory address or nil pointer dereference
.../pkg/highlight/parser.go:328
.../pkg/highlight/parser.go:318
.../pkg/highlight/parser.go:304

With this change, includes are resolved one syntax file at a time, and the parse error of an included file is shown with TermMessage, as for any other syntax file that fails to parse. micro then opens the .html file normally.

2. A syntax file that includes itself (#1293, #1585)

When a region includes its own filetype, it gets a copy of that filetype's regions. Those copies include the filetype again when their includes are resolved, and so on forever. micro hangs while its memory grows until it dies with fatal error: out of memory.

With this change, the filetypes already included by the enclosing regions are passed down the recursion and are not included again. A self-including region now gets the rules of its own filetype once, one level deep. #2787 goes further and adds real support for circular rules. This PR only stops the crash, and every syntax file without such a loop resolves to the same rules as before.

Testing

  • New pkg/highlight/parser_test.go with one test per case. go test ./... passes and gofmt reports nothing.
  • I resolved the includes of all 12 built-in syntax files that use include, with and without the change, and compared the resulting rule trees. They are identical.
  • Manual check with micro -config-dir <dir> and the two syntax files below: master crashes on both. The patched build shows the parse error of the broken css.yaml and then opens t.html (ft:html), and opens t.js (ft:javascript) normally.
Syntax files used for the manual check

syntax/css.yaml, then open an .html file:

filetype: css

detect:
    filename: "\\.css$"

rules:
    - identifier: "("

syntax/javascript.yaml (from #1585), then open a .js file:

filetype: javascript

detect:
    filename: "\\.js$"

rules:
    - identifier: "foo"
    - special:
        start: "\\("
        end: "\\)"
        rules:
            - include: javascript

Fixes #1293
Fixes #1585

Disclaimer: Found the issue by test-driving an LM-powered vulnerability scanner I work on heavily.

Huge added 2 commits October 20, 2026 12:36
When a syntax file includes another one, resolveIncludesInDef() and
resolveIncludesInRegion() call ParseDef() on the included file and
ignore the error. ParseDef() returns a nil Def on error (for example an
invalid regex), so the following searchDef.rules access panics and
micro crashes as soon as a file of the including type is opened.

This happens e.g. with a user's ~/.config/micro/syntax/css.yaml that
contains a bad regex: opening the css file shows the parse error, but
opening any html file crashes micro, since html.yaml includes css.

Skip included files that fail to parse, like an include of a missing
filetype is skipped.

Reported-by: BS.ehlas.cz
A region that includes its own filetype gets a fresh copy of that
filetype's regions, and resolveIncludesInRegion() then resolves the
includes of those copies too, which include the filetype again, and so
on. micro hangs eating memory until the stack overflows.

Pass the filetypes already included by the enclosing regions down the
recursion and skip including them again. A self-including region now
gets the rules of its own filetype once, one level deep. Includes of
the built-in syntax files resolve to exactly the same rules as before.

Fixes micro-editor#1293
Fixes micro-editor#1585

Reported-by: BS.ehlas.cz
Comment thread pkg/highlight/parser.go Outdated
Comment on lines +312 to +314
if err != nil {
continue
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will prevent the crash, but it will also cover the issue present in the included syntax definition.

Maybe more like this:

diff --git a/internal/buffer/buffer.go b/internal/buffer/buffer.go
index f033ca24..2f23cc5d 100644
--- a/internal/buffer/buffer.go
+++ b/internal/buffer/buffer.go
@@ -806,6 +806,11 @@ func resolveIncludes(syndef *highlight.Def) {
 					screen.TermMessage("Error parsing syntax file " + f.Name() + ": " + err.Error())
 					continue
 				}
+				err = highlight.ResolveInclude(syndef, file)
+				if err != nil {
+					screen.TermMessage("Error parsing syntax file " + f.Name() + ": " + err.Error())
+					continue
+				}
 				files = append(files, file)
 				break
 			}
@@ -814,8 +819,6 @@ func resolveIncludes(syndef *highlight.Def) {
 			break
 		}
 	}
-
-	highlight.ResolveIncludes(syndef, files)
 }
 
 // UpdateRules updates the syntax rules and filetype for this buffer
diff --git a/pkg/highlight/parser.go b/pkg/highlight/parser.go
index 804e6bc0..b08828cc 100644
--- a/pkg/highlight/parser.go
+++ b/pkg/highlight/parser.go
@@ -298,42 +298,51 @@ func getIncludesInRegion(region *region) []string {
 	return includes
 }
 
-// ResolveIncludes will sort out the rules for including other filetypes
-// You should call this after parsing all the Defs
-func ResolveIncludes(def *Def, files []*File) {
-	resolveIncludesInDef(files, def)
+// ResolveInclude will combine the rules with included filetypes
+func ResolveInclude(def *Def, file *File) error {
+	return resolveIncludeInDef(file, def)
 }
 
-func resolveIncludesInDef(files []*File, d *Def) {
+func resolveIncludeInDef(file *File, d *Def) error {
 	for _, lang := range d.rules.includes {
-		for _, searchFile := range files {
-			if lang == searchFile.FileType {
-				searchDef, _ := ParseDef(searchFile, nil)
-				d.rules.patterns = append(d.rules.patterns, searchDef.rules.patterns...)
-				d.rules.regions = append(d.rules.regions, searchDef.rules.regions...)
+		if lang == file.FileType {
+			searchDef, err := ParseDef(file, nil)
+			if err != nil {
+				return err
 			}
+			d.rules.patterns = append(d.rules.patterns, searchDef.rules.patterns...)
+			d.rules.regions = append(d.rules.regions, searchDef.rules.regions...)
 		}
 	}
 	for _, r := range d.rules.regions {
-		resolveIncludesInRegion(files, r)
+		err := resolveIncludeInRegion(file, r)
+		if err != nil {
+			return err
+		}
 		r.parent = nil
 	}
+	return nil
 }
 
-func resolveIncludesInRegion(files []*File, region *region) {
+func resolveIncludeInRegion(file *File, region *region) error {
 	for _, lang := range region.rules.includes {
-		for _, searchFile := range files {
-			if lang == searchFile.FileType {
-				searchDef, _ := ParseDef(searchFile, nil)
-				region.rules.patterns = append(region.rules.patterns, searchDef.rules.patterns...)
-				region.rules.regions = append(region.rules.regions, searchDef.rules.regions...)
+		if lang == file.FileType {
+			searchDef, err := ParseDef(file, nil)
+			if err != nil {
+				return err
 			}
+			region.rules.patterns = append(region.rules.patterns, searchDef.rules.patterns...)
+			region.rules.regions = append(region.rules.regions, searchDef.rules.regions...)
 		}
 	}
 	for _, r := range region.rules.regions {
-		resolveIncludesInRegion(files, r)
+		err := resolveIncludeInRegion(file, r)
+		if err != nil {
+			return err
+		}
 		r.parent = region
 	}
+	return nil
 }
 
 func parseRules(input []any, curRegion *region) (ru *rules, err error) {

While we might change the include logic in buffer.go already:
This whole files = append(files, file) plus later check if len(files) >= len(includes) (and last break) looks ugly. We should make sure that the includes are collected only once (highlight.GetIncludes() might require a rework for this); then we don't need to check if files exceed includes.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2a859b8, along the lines of your patch: ResolveInclude() handles one file and returns its parse error, and resolveIncludes() in buffer.go shows the error with TermMessage and moves on to the next syntax file. With the broken css.yaml from the description, opening an .html file now shows Error parsing syntax file css: error parsing regexp: missing closing ): `(` and then opens the file with ft:html.

I kept the self-include guard from the second commit, now in resolveIncludeInRegion(). GetIncludes() returns each filetype only once, so buffer.go drops a filetype from the list once it is resolved and stops when the list is empty, without the files/includes length check.

The resolved rules of the 12 built-in syntax files that use include are the same as on master.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 2a859b8,

Yes, but unfortunately two different things have been done in this one commit.
Please split the two logical changes into two separate commits.

@Huge Huge Oct 29, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split into two separate commits as needed:

  1. 3804588 highlight: Report an included syntax file that fails to parse (resolves each include one at a time and surfaces parse errors via TermMessage).
  2. 9e7dc71 highlight: Return each filetype only once from GetIncludes (deduplicates included filetypes so each is returned once, replacing the file-count break check).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmaluka
What do you think, should we include 3804588 in e769e52 directly or keep it separate?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd need to get a bit familiar with this whole PR first. I'll try to find time for that soon (feel free to ping me if I forget to).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we include 3804588 in e769e52 directly or keep it separate?

It seems better to keep them separate. So far it's been easier for me to review them separately. They solve different problems, right?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They solve different problems, right?

More or less.
In the first step, we silently ignore definitions with errors, just to not crash.
In the second step, we exchange this silent drop again with error messages to the user.

So far it's been easier for me to review them separately.

OK, that's reason enough. 👍

Comment thread pkg/highlight/parser.go Resolved
Huge added 2 commits October 29, 2026 09:30
Resolve includes one syntax file at a time, as suggested in review, so
that an included file that fails to parse shows its error instead of
being skipped without a word.
Returning each included filetype only once replaces the check that
compared the number of found files with the total number of includes.
@Huge
Huge force-pushed the fix/syntax-include-crash branch from 2a859b8 to 9e7dc71 Compare October 29, 2026 14:14
Comment thread pkg/highlight/parser.go
if isIncluded(lang, included) {
continue
}
included = append(included[:len(included):len(included)], lang)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's up with this? Why not just append(included, lang)?

// The region gets the included rules once; the region copied in by
// that include does not include the same file again.
inner := def.rules.regions[0]
if assert.Len(t, inner.rules.patterns, 1) && assert.Len(t, inner.rules.regions, 1) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why if?

Comment thread pkg/highlight/parser.go
}

// GetIncludes returns a list of filetypes that are included by this syntax def
// GetIncludes returns the filetypes that the Def includes, each one once

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does "each one once" mean?

What is this whole commit for? What problem does it solve?

Comment thread pkg/highlight/parser.go Outdated
Comment on lines +312 to +314
if err != nil {
continue
}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we include 3804588 in e769e52 directly or keep it separate?

It seems better to keep them separate. So far it's been easier for me to review them separately. They solve different problems, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recursive include in syntax file causes Micro to hang Crashes on lauch if syntax includes itself

3 participants