Repository navigation
Conversation
When a syntax file includes another one, resolveIncludesInDef() and resolveIncludesInRegion() call ParseDef() on the included file and ignore the error. ParseDef() returns a nil Def on error (for example an invalid regex), so the following searchDef.rules access panics and micro crashes as soon as a file of the including type is opened. This happens e.g. with a user's ~/.config/micro/syntax/css.yaml that contains a bad regex: opening the css file shows the parse error, but opening any html file crashes micro, since html.yaml includes css. Skip included files that fail to parse, like an include of a missing filetype is skipped. Reported-by: BS.ehlas.cz
A region that includes its own filetype gets a fresh copy of that filetype's regions, and resolveIncludesInRegion() then resolves the includes of those copies too, which include the filetype again, and so on. micro hangs eating memory until the stack overflows. Pass the filetypes already included by the enclosing regions down the recursion and skip including them again. A self-including region now gets the rules of its own filetype once, one level deep. Includes of the built-in syntax files resolve to exactly the same rules as before. Fixes micro-editor#1293 Fixes micro-editor#1585 Reported-by: BS.ehlas.cz
| if err != nil { | ||
| continue | ||
| } |
There was a problem hiding this comment.
This will prevent the crash, but it will also cover the issue present in the included syntax definition.
Maybe more like this:
diff --git a/internal/buffer/buffer.go b/internal/buffer/buffer.go
index f033ca24..2f23cc5d 100644
--- a/internal/buffer/buffer.go
+++ b/internal/buffer/buffer.go
@@ -806,6 +806,11 @@ func resolveIncludes(syndef *highlight.Def) {
screen.TermMessage("Error parsing syntax file " + f.Name() + ": " + err.Error())
continue
}
+ err = highlight.ResolveInclude(syndef, file)
+ if err != nil {
+ screen.TermMessage("Error parsing syntax file " + f.Name() + ": " + err.Error())
+ continue
+ }
files = append(files, file)
break
}
@@ -814,8 +819,6 @@ func resolveIncludes(syndef *highlight.Def) {
break
}
}
-
- highlight.ResolveIncludes(syndef, files)
}
// UpdateRules updates the syntax rules and filetype for this buffer
diff --git a/pkg/highlight/parser.go b/pkg/highlight/parser.go
index 804e6bc0..b08828cc 100644
--- a/pkg/highlight/parser.go
+++ b/pkg/highlight/parser.go
@@ -298,42 +298,51 @@ func getIncludesInRegion(region *region) []string {
return includes
}
-// ResolveIncludes will sort out the rules for including other filetypes
-// You should call this after parsing all the Defs
-func ResolveIncludes(def *Def, files []*File) {
- resolveIncludesInDef(files, def)
+// ResolveInclude will combine the rules with included filetypes
+func ResolveInclude(def *Def, file *File) error {
+ return resolveIncludeInDef(file, def)
}
-func resolveIncludesInDef(files []*File, d *Def) {
+func resolveIncludeInDef(file *File, d *Def) error {
for _, lang := range d.rules.includes {
- for _, searchFile := range files {
- if lang == searchFile.FileType {
- searchDef, _ := ParseDef(searchFile, nil)
- d.rules.patterns = append(d.rules.patterns, searchDef.rules.patterns...)
- d.rules.regions = append(d.rules.regions, searchDef.rules.regions...)
+ if lang == file.FileType {
+ searchDef, err := ParseDef(file, nil)
+ if err != nil {
+ return err
}
+ d.rules.patterns = append(d.rules.patterns, searchDef.rules.patterns...)
+ d.rules.regions = append(d.rules.regions, searchDef.rules.regions...)
}
}
for _, r := range d.rules.regions {
- resolveIncludesInRegion(files, r)
+ err := resolveIncludeInRegion(file, r)
+ if err != nil {
+ return err
+ }
r.parent = nil
}
+ return nil
}
-func resolveIncludesInRegion(files []*File, region *region) {
+func resolveIncludeInRegion(file *File, region *region) error {
for _, lang := range region.rules.includes {
- for _, searchFile := range files {
- if lang == searchFile.FileType {
- searchDef, _ := ParseDef(searchFile, nil)
- region.rules.patterns = append(region.rules.patterns, searchDef.rules.patterns...)
- region.rules.regions = append(region.rules.regions, searchDef.rules.regions...)
+ if lang == file.FileType {
+ searchDef, err := ParseDef(file, nil)
+ if err != nil {
+ return err
}
+ region.rules.patterns = append(region.rules.patterns, searchDef.rules.patterns...)
+ region.rules.regions = append(region.rules.regions, searchDef.rules.regions...)
}
}
for _, r := range region.rules.regions {
- resolveIncludesInRegion(files, r)
+ err := resolveIncludeInRegion(file, r)
+ if err != nil {
+ return err
+ }
r.parent = region
}
+ return nil
}
func parseRules(input []any, curRegion *region) (ru *rules, err error) {
While we might change the include logic in buffer.go already:
This whole files = append(files, file) plus later check if len(files) >= len(includes) (and last break) looks ugly. We should make sure that the includes are collected only once (highlight.GetIncludes() might require a rework for this); then we don't need to check if files exceed includes.
There was a problem hiding this comment.
Done in 2a859b8, along the lines of your patch: ResolveInclude() handles one file and returns its parse error, and resolveIncludes() in buffer.go shows the error with TermMessage and moves on to the next syntax file. With the broken css.yaml from the description, opening an .html file now shows Error parsing syntax file css: error parsing regexp: missing closing ): `(` and then opens the file with ft:html.
I kept the self-include guard from the second commit, now in resolveIncludeInRegion(). GetIncludes() returns each filetype only once, so buffer.go drops a filetype from the list once it is resolved and stops when the list is empty, without the files/includes length check.
The resolved rules of the 12 built-in syntax files that use include are the same as on master.
There was a problem hiding this comment.
Done in 2a859b8,
Yes, but unfortunately two different things have been done in this one commit.
Please split the two logical changes into two separate commits.
There was a problem hiding this comment.
Split into two separate commits as needed:
3804588highlight: Report an included syntax file that fails to parse(resolves each include one at a time and surfaces parse errors viaTermMessage).9e7dc71highlight: Return each filetype only once from GetIncludes(deduplicates included filetypes so each is returned once, replacing the file-count break check).
There was a problem hiding this comment.
I'd need to get a bit familiar with this whole PR first. I'll try to find time for that soon (feel free to ping me if I forget to).
There was a problem hiding this comment.
They solve different problems, right?
More or less.
In the first step, we silently ignore definitions with errors, just to not crash.
In the second step, we exchange this silent drop again with error messages to the user.
So far it's been easier for me to review them separately.
OK, that's reason enough. 👍
Resolve includes one syntax file at a time, as suggested in review, so that an included file that fails to parse shows its error instead of being skipped without a word.
Returning each included filetype only once replaces the check that compared the number of found files with the total number of includes.
2a859b8 to
9e7dc71
Compare
| if isIncluded(lang, included) { | ||
| continue | ||
| } | ||
| included = append(included[:len(included):len(included)], lang) |
There was a problem hiding this comment.
What's up with this? Why not just append(included, lang)?
| // The region gets the included rules once; the region copied in by | ||
| // that include does not include the same file again. | ||
| inner := def.rules.regions[0] | ||
| if assert.Len(t, inner.rules.patterns, 1) && assert.Len(t, inner.rules.regions, 1) { |
| } | ||
|
|
||
| // GetIncludes returns a list of filetypes that are included by this syntax def | ||
| // GetIncludes returns the filetypes that the Def includes, each one once |
There was a problem hiding this comment.
What does "each one once" mean?
What is this whole commit for? What problem does it solve?
| if err != nil { | ||
| continue | ||
| } |
Two crashes in
pkg/highlightwhile resolvingincluderules. Both can be hit with a custom syntax file in~/.config/micro/syntax/.1. An included syntax file fails to parse
resolveIncludesInDef()andresolveIncludesInRegion()ignore the error returned byParseDef()and then use theDefit returned, which is nil on error. Example: a usercss.yamlwith an invalid regex. Opening a.cssfile shows the parse error as expected, but opening any.htmlfile crashes micro, becausehtml.yamlincludes css:With this change, includes are resolved one syntax file at a time, and the parse error of an included file is shown with
TermMessage, as for any other syntax file that fails to parse. micro then opens the.htmlfile normally.2. A syntax file that includes itself (#1293, #1585)
When a region includes its own filetype, it gets a copy of that filetype's regions. Those copies include the filetype again when their includes are resolved, and so on forever. micro hangs while its memory grows until it dies with
fatal error: out of memory.With this change, the filetypes already included by the enclosing regions are passed down the recursion and are not included again. A self-including region now gets the rules of its own filetype once, one level deep. #2787 goes further and adds real support for circular rules. This PR only stops the crash, and every syntax file without such a loop resolves to the same rules as before.
Testing
pkg/highlight/parser_test.gowith one test per case.go test ./...passes andgofmtreports nothing.include, with and without the change, and compared the resulting rule trees. They are identical.micro -config-dir <dir>and the two syntax files below: master crashes on both. The patched build shows the parse error of the brokencss.yamland then openst.html(ft:html), and openst.js(ft:javascript) normally.Syntax files used for the manual check
syntax/css.yaml, then open an.htmlfile:syntax/javascript.yaml(from #1585), then open a.jsfile:Fixes #1293
Fixes #1585
Disclaimer: Found the issue by test-driving an LM-powered vulnerability scanner I work on heavily.