detached 5f9df8b

fix(expr): tokenize and parse like the runner (whitespace, errors, positions, depth), built-ins win over extensions

commit
5f9df8b2555896813a127bd6b39e85ce307f9c83
author
brice · · 2026-10-12 18:18 UTC
parent
3b23706
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
18 files · +1372 -586
Split
UnifiedSplit
M expr/PORTING.md +194 -92
@@ -1,26 +1,54 @@
11 # Porting notes for package expr
22
3-The package is a port of `@actions/expressions` 0.3.61 (MIT, GitHub,
4-actions/languageservices). The JavaScript is the specification wherever this
5-package could be checked against it: `testdata/oracle.json.gz` holds what the
6-reference answers for 33 575 expressions (tokens with positions, error
7-messages, result kinds, string coercions, `toJSON` output), and
8-`TestOracle` replays it. `gen/oracle.mjs` regenerates it, `gen/golden.mjs` the
9-`testdata/tojson` goldens, and `gen/casing_dump.mjs` + `gen/casing.go` the
10-casing tables (`go generate ./expr`; Node 25 / V8 14.1, Unicode 16). The
11-tables are relative to the Go toolchain's own Unicode tables
12-(`unicode.Version`); `TestCasingTablesMatchGoUnicodeVersion` fails with the
13-regeneration command when the toolchain changes. The oracle records the
14-npm integrity (sha512) of the reference tarball and the Node/V8/Unicode
15-versions in its `reference` field, and `TestOracle` logs them.
3+The package implements the expression language of GitHub Actions as **the C#
4+runner** (actions/runner, `src/Sdk/DTExpressions2`) runs it on github.com. It
5+started as a port of `@actions/expressions` 0.3.61 (MIT, GitHub,
6+actions/languageservices), the JavaScript used by the editors, and was then
7+checked against the runner. Where the two differ the runner wins, because it
8+is what users see at run time; the differences are listed below, with the C#
9+source that decides each of them. The JavaScript is still the reference for
10+the token stream and positions of `Lex`, which `Parse` no longer uses.
1611
17-The C# runner (actions/runner, `DistributedTask/Expressions2`) is the
18-implementation that runs on github.com. It was not available while this was
19-written. Nothing below has been checked against it.
12+Three kinds of evidence back the tests:
2013
21-## Limits added by this port (not in the JavaScript)
14+- `testdata/oracle.json.gz`: what the JavaScript answers on 34 765 expressions
15+ (tokens, error messages, result kinds, string forms, `toJSON`), generated by
16+ `gen/oracle.mjs`; `TestOracle` replays it, and excuses a mismatch only where
17+ the runner gives this package's answer.
18+- `testdata/runner.json.gz`: what the runner's own code answers on the same
19+ expressions. `gen/runner` compiles `src/Sdk/DTExpressions2` (plus stubs for
20+ the few assemblies it needs, see `gen/runner/README.md`) and runs it;
21+ `TestRunnerOracle` holds the package to it: 34 765 cases, the only
22+ differences are negative zero (below) and error texts of Json.NET.
23+ The same driver recorded `testdata/tojson/*.runner`, `ordinal_cases.txt`
24+ and `number_cases.txt`, and checked the number formatting on 200 000 random
25+ doubles.
26+- `testdata/languageservices/*.json`: the 1 020 cross-language fixtures of
27+ actions/languageservices (see `xlang_test.go`): 1 018 pass, 2 are known
28+ differences with a reason: `case/27` (the runner reports an even `case()` on
29+ the `(`, not the name) and `syntax-errors/memory-errors/0` (the runner's
30+ memory accounting). `testdata/languageservices/SOURCE` has the commit.
2231
23-The reference has no bound on what an evaluation may build. These are
32+The runner's own expression tests (`ExpressionParserL0`, `ConditionFunctionsL0`)
33+are ported in `runner_l0_test.go`. The casing tables of the JavaScript
34+(`casing.go`, `gen/casing*`) remain for `Lex`/oracle support only.
35+
36+Regenerating: `dotnet build -c Release expr/gen/runner` then the `*.mjs` scripts
37+in `expr/gen/runner` (they need a clone of actions/runner and the .NET SDK, not
38+`go test`). Runner commit used: `5f0b3ddcc6ee9fe9441e00d55e85b94ea4aad798`,
39+.NET 9.0.20.
40+
41+Caveat for all of the above: the runner has two copies of the engine,
42+`src/Sdk/DTExpressions2` (used by the legacy template evaluator) and
43+`src/Sdk/Expressions` (used by the newer `WorkflowTemplateEvaluator`, which the
44+runner currently runs next to the legacy one and compares). They are almost the
45+same code; the differences found are noted where they matter (negative zero,
46+`StrictJsonParsing`, `AlwaysTraceRealized`). The differential run uses
47+`DTExpressions2`.
48+
49+## Limits added by this port (not in the JavaScript or the runner)
50+
51+The JavaScript has no bound on what an evaluation may build. These are
2452 additions, each reported as an `*EvalError` (or `*Error` for the parser) and
2553 none changes the result of an evaluation that stays within them.
2654
@@ -38,8 +66,8 @@ none changes the result of an evaluation that stays within them.
3866 loops of filters, `join` and `contains`, and passes `ctx` to extension
3967 functions.
4068 - **Evaluation depth** of 1000 for hand-built trees (parsed trees stay far
41- below it), **JSON depth** `MaxJSONDepth` = 1000 for `FromJSON` and for the
42- encoder (cyclic values built with `Array.Append` yield an error from
69+ below it), **JSON depth** `MaxJSONDepth` = 1000 for the
70+ encoder (`FromJSON` is limited to 64 levels like Json.NET) (cyclic values built with `Array.Append` yield an error from
4371 `EncodeJSON`, `"null"` from `ToJSON`, an `*EvalError` from `toJSON()`).
4472 The JS recursion limit is engine-dependent, around 5 000 to 10 000.
4573 - **Built-in arity is re-checked at evaluation**, so a hand-built AST gets an
@@ -47,8 +75,8 @@ none changes the result of an evaluation that stays within them.
4775 - **Input length** is checked before the input is converted, without
4876 allocating.
4977
50-Other limits that exist in the JS and are ported: expression length 21000
51-UTF-16 units and parser depth 50 (see below).
78+Limits that the runner has and this package follows: expression length 21000
79+UTF-16 units and tree depth 50 (see below).
5280
5381 ## Performance notes (deferred)
5482
@@ -58,11 +86,10 @@ UTF-16 units and parser depth 50 (see below).
5886 same contexts, add a prepared `Environment` (fold once, reuse) and cache
5987 folded names in the AST.
6088
61-## Where the JavaScript is believed to differ from the C# runner
89+## Where the JavaScript differs from the C# runner (and what this package does)
6290
63-Each item is implemented as the JavaScript does. Check the runner's tests
64-(`src/Test/L0/Expressions*`, `Sdk/Expressions2/...`) and decide for each one
65-whether to keep the JS behaviour or follow the runner.
91+Each item names the runner source that decides it. All are resolved; the
92+package follows the runner.
6693
6794 - [x] **String escapes: every `''` is unescaped.** Resolved: follows the C#.
6895 `LexicalAnalyzer.ReadStringToken` (`Tokens/LexicalAnalyzer.cs`, the
@@ -204,41 +231,88 @@ whether to keep the JS behaviour or follow the runner.
204231 `join(success(), format('{2}'))` is `true` in the runner and an error in
205232 the JS. Built-ins now receive the call and evaluate what they need;
206233 extension functions still get evaluated arguments.
207-- [ ] **Index access**: `a[b]` where `b` is a number uses `floor(b)` and
208- rejects NaN, negatives and infinities; where `b` is a string it is a key
209- or a decimal index (`'1'`, `' 1 '`, `'1.9'` all index 1 of an array).
210- Booleans and null index as `1`/`0` on arrays and as `"true"`/`""` keys on
211- dictionaries.
212-- [ ] **Filtered arrays** (`a.*`): a dictionary filtered by `*` yields its
213- values; `.*.b` skips elements without `b`; `.*.*` flattens one level;
214- arrays produced by `*` are fresh, so `a.* == a.*` is false.
215-- [ ] **Comparison**: `null` is only ordered against numbers/booleans (it
216- becomes 0); arrays and dictionaries are never ordered and `==` is
217- identity; `NaN` is unordered and unequal to itself; `true > false`.
218-- [ ] **`&&`/`||` return an operand**, not a boolean (`'' || 'x'` is `'x'`).
219-- [ ] **Limits**: max expression length 21000 (UTF-16 units; the JS throws a
220- bare `Error("ErrorExceededMaxLength")`), max parse depth 50. The depth
221- counting (including `index()` and the flat logical chains) is ported
222- literally; C# `MaxDepth` is 50 with its own counting. Cross-check with
223- deep-nesting tests. There is no node-count or evaluation-memory limit
224- here; the C# runner has `MaxMemory` (1 MB) and a result-size limit.
225-- [ ] **Trailing comma in calls**: `contains('a', 'b',)` parses (the JS loop
226- only requires a comma when the next token is not `)`). `f(,)` does not.
227-- [ ] **Whitespace**: only space, `\t`, `\r`, `\n` separate tokens. U+00A0 and
228- other Unicode spaces end a word but are not skipped, so `1 \u00a0 2`
229- reports an unexpected symbol. `\v`, `\f` likewise.
230-- [ ] **Empty expression**: the JS parser returns `undefined` for no tokens.
231- `Parse` returns a nil Node and no error; `Evaluate(nil)` errors.
232-- [ ] **Unknown contexts at evaluation time** are null (the JS `Dictionary.get`
233- returns `undefined` and `visitContextAccess` then crashes later); here
234- declared-but-absent contexts are null, never a crash.
235-- [ ] **Error positions**: the JS EOF token's start column is negative after a
236- trailing newline (stale `start`); ported as is
237- (`TestEOFTokenPosition`).
238-- [ ] **Function names and context names are case-insensitive**; the JS
239- lower-cases with `toLowerCase`, as does this port (`foldKey`). Extension
240- functions override built-ins at evaluation (`functions?.get(...) ||
241- wellKnownFunctions[...]`) but built-ins win at validation time (arity).
234+- [x] **Large integers in `fromJSON`**: found by the differential run. Json.NET
235+ reads an integer that does not fit in 64 bits as a `BigInteger`, and
236+ `(Double)BigInteger` truncates toward zero to 53 bits instead of rounding
237+ to nearest, so `fromJSON('18446744073709551615')` is 2^64-2048, not 2^64,
238+ and does not equal the literal `18446744073709551615` of an expression
239+ (which `Double.TryParse` rounds to nearest). Reproduced in
240+ `jsonParser.number` (`TestFromJSONBigIntegers`); checked on 57 values.
241+- [x] **Index access**: resolved, same in the C# (`Sdk/Operators/Index.cs`,
242+ `IndexHelper`): a number is floored, NaN, negatives and numbers above
243+ Int32.MaxValue give no index; a primitive also has a string form for
244+ dictionary keys, so booleans and null are `"true"`/`""` there and 1/0 on
245+ arrays; `' 1 '` and `'1.9'` index element 1. Differences found: the
246+ index is only evaluated when the object is an array or a dictionary, and
247+ its errors are not wrapped (see the lazy evaluation item above).
248+ `TestRunnerSemantics`.
249+- [x] **Filtered arrays** (`a.*`): resolved, same in the C#
250+ (`Index.HandleFilteredArray`): `*` on a dictionary yields its values,
251+ `.*.b` skips elements without `b`, `.*.*` flattens one level, and every
252+ `*` builds a new array, so `a.* == a.*` is false. `TestRunnerSemantics`.
253+- [x] **Comparison**: resolved, same in the C# (`EvaluationResult.cs`
254+ `CoerceTypes`/`AbstractEqual`/`AbstractGreaterThan`): null and booleans
255+ become numbers when compared with another kind, a string compared with a
256+ number is parsed with `ParseNumber`, arrays and dictionaries compare by
257+ reference and are never ordered, NaN is unequal to itself and unordered.
258+ Strings compare with `OrdinalIgnoreCase` (see above). `TestRunnerSemantics`.
259+- [x] **`&&`/`||` return an operand**: resolved, same in the C# (`And.cs`,
260+ `Or.cs`: they return `result.Value` of the deciding operand).
261+- [x] **Limits**: resolved, follows the C#. Length 21000 UTF-16 units
262+ (`ExpressionConstants.MaxLength`): "Exceeded max expression length 21000",
263+ without a token (the JS throws a bare `ErrorExceededMaxLength`). Depth 50
264+ is checked on the finished tree (`ExpressionParser.CheckMaxDepth`): the
265+ root is depth 1, an operand of an operator, an argument of a call and the
266+ two sides of an index are one deeper, **parentheses add nothing**, and a
267+ chain of `&&` or `||` is one node, nested ones included, so
268+ `((((...1...))))` with 500 parentheses and `1 || (1 || (1 ...))` parse;
269+ `"!" * 50 + "true"` and 50 comparisons chained with `==` do not. The error
270+ has no token: "Exceeded max expression depth 50". The JS limit counted
271+ parser recursion (parentheses included). Memory: the runner has
272+ `EvaluationOptions.MaxMemory` (1 MiB by default) and fails with "The
273+ maximum allowed memory size was exceeded while evaluating the following
274+ expression: ...", counting strings at 26 bytes + 2 per UTF-16 unit and 24
275+ per object, per call depth (`EvaluationMemory`). This package keeps its
276+ own output budget (see above), which is not byte-for-byte the same, so
277+ the fixture `syntax-errors.json` "memory-errors" is a known difference.
278+- [x] **Trailing comma in calls**: resolved, follows the C#. `contains('a', 'b',)`
279+ is "Unexpected symbol: ')'" (the tokenizer rejects a `)` after a `,`). The JS
280+ accepted it. `f(,)` and `contains(,'a')` fail as before.
281+- [x] **Whitespace**: resolved, follows the C#. `Char.IsWhiteSpace` separates
282+ tokens: U+0009..U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000..U+200A,
283+ U+2028, U+2029, U+202F, U+205F, U+3000. So `1 \\u00a0 2` is "Unexpected
284+ symbol: '2'" (not the NBSP), and `\\v1` is 1. The JS skipped only space,
285+ tab, CR and LF. U+FEFF is not white space.
286+- [x] **Empty expression**: same in the C# (`CreateTree` returns null when there
287+ is no token). `Parse` returns a nil Node and no error; `Evaluate(nil)`
288+ errors; callers (the template layer) decide.
289+- [x] **Unknown contexts at evaluation time** are null: in the runner a
290+ declared named value is evaluated by the caller's node; here an absent
291+ key of `EvalContext.Contexts` is null.
292+- [x] **Error positions**: resolved, follows the C#. A `ParseException` names
293+ the token and its **absolute 1-based index in the expression**
294+ (`Located at position N within expression: <whole expression>`);
295+ `Error.Pos` is the same place as a zero-based line and column.
296+ Arity and even-argument errors name the `(` of the call (the token
297+ `FlushTopEndParameters` holds), not the function: `contains()` is
298+ "Too few parameters supplied: '('. Located at position 9". The
299+ end-of-input error is "Unexpected end of expression: '<last token>'"
300+ (the JS said "Unexpected symbol: 'EOF'"), and the JS's negative EOF column
301+ is gone with the JS lexer. Unexpected-symbol tokens are the
302+ boundary-delimited run (`ReadKeywordToken`): `github..event` reports `.`
303+ where the JS reported `.event`. Lexing and parsing are interleaved, so the
304+ first problem in reading order wins (the JS lexed everything first).
305+ `TestParseErrorPositions`, and `TestRunnerOracle` on 1 700 error cases.
306+ One fixture still differs, see `xlangKnownDifferences`: `case/27` expects
307+ the name for an even `case()`, the runner passes the `(`.
308+- [x] **Function names and context names are case-insensitive**: same in the
309+ C# (`StringComparer.OrdinalIgnoreCase` dictionaries; see the folding
310+ item). Built-ins win over extension functions of the same name at
311+ validation **and at evaluation** (`ExpressionParser.TryGetFunctionInfo`
312+ looks in `ExpressionConstants.WellKnownFunctions` first and builds the
313+ node from it); the JS let an extension override a built-in at
314+ evaluation. An extension supplied under a built-in's name is now never
315+ called.
242316
243317 ## JavaScript behaviours deliberately NOT ported
244318
@@ -257,36 +331,64 @@ a bug.
257331 evaluating e.g. `github.true` with an empty context: the reference does not
258332 handle a missing root context. Here it is null.
259333 - Recursion limits: JS recursion depth for `fromJSON` input is engine
260- dependent (about 5 000 to 10 000 levels). `FromJSON` accepts `MaxJSONDepth`
261- (1000) levels.
262-- Lone surrogates: JS strings are UTF-16 and can hold them; Go strings are
263- UTF-8. They are kept as WTF-8 (`0xED 0xA0-0xBF xx`) so that `fromJSON` /
264- `toJSON` round-trip, `==`/`<`/`contains`/`startsWith`/`endsWith` compare
265- them like code units, case mapping copies them, and `format`/`join` rejoin a
266- high surrogate followed by a low one into the character they form. This is
267- covered by oracle cases. Two surrogate strings concatenated by other means
268- (an extension function) are not rejoined.
334+ dependent. `FromJSON` follows Json.NET's `MaxDepth` of 64.
335+- Lone surrogates: JS strings are UTF-16 and can hold them. The runner cannot
336+ (Json.NET reads an unpaired `\ud800` as U+FFFD), and neither can Go strings;
337+ the WTF-8 machinery of the first version (`wtf8Surrogate`, `normalizeWTF8`)
338+ is only used by the case-folding and comparison helpers for robustness and
339+ by the oracle tests.
340+
341+## Still open
342+
343+- [ ] **Memory accounting.** The runner caps an evaluation at `MaxMemory` (1 MiB)
344+ counted by `EvaluationMemory`/`MemoryCounter` (26 bytes + 2 per UTF-16 unit
345+ per string, 24 per object, trimmed per call depth); this package caps
346+ `MaxResultBytes` (1 MiB of output, different units) and accepts
347+ `format(...)` chains the runner rejects. Fixture
348+ `syntax-errors/memory-errors/0` is the known difference. Reproducing
349+ `EvaluationMemory` exactly would need the node depth of every value;
350+ worth doing only if the error text matters ("The maximum allowed memory
351+ size was exceeded while evaluating the following expression: ...", which
352+ needs `ConvertToExpression` of the root, also not ported).
353+- [ ] **Negative zero.** `-0` is `0` here. `DTExpressions2/EvaluationResult.cs`
354+ (the legacy engine) prints `-0` for `format('{0}', -0)` and `toJSON(-0)`,
355+ because .NET Core prints `-0`; `Expressions/EvaluationResult.cs` (the engine of
356+ `WorkflowTemplateEvaluator`) maps it to `0`, and the cross-language fixture
357+ `coerce_string.json` expects `0`. If the legacy engine is what a user sees,
358+ `-0` would be right in `format()`; the choice was made for the fixture.
359+ `toJSON` goes through `ConvertToString` in both and so follows the same
360+ choice.
361+- [ ] **Comments and constructors in `fromJSON`.** Json.NET accepts `/* */` and
362+ `//` comments and `new Date(1)` constructors; the first is read in a
363+ surprising way (`/*x*/1` is the string `x`). They fail here.
364+- [ ] **`fromJSON` error texts.** Json.NET messages ("Unexpected character
365+ encountered while parsing value: a. Path '', line 0, position 0.") are
366+ not reproduced; the outer message "Error parsing JSON when evaluating
367+ fromJson" is the JS one, the runner's exception message is the Json.NET
368+ one. Only the failure is checked.
369+- [ ] **`StrictJsonParsing`.** The newer engine has an option that switches
370+ `fromJSON` to `System.Text.Json` (strict, 64 levels, different
371+ messages); default off, "temporary for telemetry". Not implemented.
372+- [ ] **Unicode version of the folding table** is that of the .NET 9 runtime that
373+ generated `fold_tables.go`; a runner on another runtime may fold newly
374+ assigned letters differently.
375+- [ ] **Tracing and secret masking** (`ITraceWriter`, `ISecretMasker`,
376+ `EvaluationResult` tracing) are runner features outside this package.
269377
270-## Other things to check once the runner is available
378+## Left to the callers
271379
272-- [ ] Run the runner's expression test vectors (`ExpressionParserL0`,
273- `ExpressionTests`, `ExpressionValuesL0`, `ExpressionTestUtil`) through a
274- table adapter; list every disagreement here before deciding.
275-- [ ] Decide whether `EvalContext` needs a hook for the functions the runner
276- implements natively with special semantics: `success()`, `failure()`,
277- `always()`, `cancelled()` (take no arguments; read job status) and
278- `hashFiles()` (variadic glob patterns, string result). The API supports
279- them through `FuncInfo` + `Func` without changes; arity is declared by
280- the caller.
281-- [ ] Context names the runner allows per location (`github`, `env`, `vars`,
282- `job`, `jobs`, `steps`, `runner`, `secrets`, `strategy`, `matrix`,
283- `needs`, `inputs`) and which functions each location may call
284- (`success`... only in `if`/`jobs.*.if`...). This belongs to the template
285- layer, not to package expr.
286-- [ ] `format()` argument limit is 255 in the JS (`MAX_ARGUMENTS`); the C#
287- limit is `ExpressionConstants.MaxArguments` (check).
288-- [x] Case-insensitive comparison and `contains`: `OrdinalIgnoreCase` is the
289- truth, see above.
290-- [ ] Secret masking and the `${{ }}` splitting of a larger string are not
291- part of this package (they belong to the template layer): `Parse`
292- takes the text between the braces only.
380+- Run the runner's expression test vectors: done for `ExpressionParserL0` and
381+ `ConditionFunctionsL0` (`runner_l0_test.go`). The runner has no other
382+ expression tests in `src/Test/L0` (the larger suites are in the closed
383+ service); the cross-language fixtures cover the rest.
384+- The functions the runner implements natively with special semantics:
385+ `success()`, `failure()`, `always()`, `cancelled()` (no arguments; read job
386+ status, or `action_status` in a composite action's main step) and
387+ `hashFiles()` (variadic, string result) are supplied by the caller through
388+ `FuncInfo` + `Func`; `runner_l0_test.go` shows the declaration the runner
389+ uses (`FunctionInfo(name, 0, 0)`).
390+- Context names per location and which functions each location may call belong
391+ to the template layer.
392+- The `format()` argument limit is 255 in both (`Byte.MaxValue`).
393+- Secret masking and the `${{ }}` splitting of a larger string are not part of
394+ this package: `Parse` takes the text between the braces only.
@@ -1,26 +1,54 @@
1 # Porting notes for package expr1 # Porting notes for package expr
2 2
3-The package is a port of `@actions/expressions` 0.3.61 (MIT, GitHub,3+The package implements the expression language of GitHub Actions as **the C#
4-actions/languageservices). The JavaScript is the specification wherever this4+runner** (actions/runner, `src/Sdk/DTExpressions2`) runs it on github.com. It
5-package could be checked against it: `testdata/oracle.json.gz` holds what the5+started as a port of `@actions/expressions` 0.3.61 (MIT, GitHub,
6-reference answers for 33 575 expressions (tokens with positions, error6+actions/languageservices), the JavaScript used by the editors, and was then
7-messages, result kinds, string coercions, `toJSON` output), and7+checked against the runner. Where the two differ the runner wins, because it
8-`TestOracle` replays it. `gen/oracle.mjs` regenerates it, `gen/golden.mjs` the8+is what users see at run time; the differences are listed below, with the C#
9-`testdata/tojson` goldens, and `gen/casing_dump.mjs` + `gen/casing.go` the9+source that decides each of them. The JavaScript is still the reference for
10-casing tables (`go generate ./expr`; Node 25 / V8 14.1, Unicode 16). The10+the token stream and positions of `Lex`, which `Parse` no longer uses.
11-tables are relative to the Go toolchain's own Unicode tables
12-(`unicode.Version`); `TestCasingTablesMatchGoUnicodeVersion` fails with the
13-regeneration command when the toolchain changes. The oracle records the
14-npm integrity (sha512) of the reference tarball and the Node/V8/Unicode
15-versions in its `reference` field, and `TestOracle` logs them.
16 11
17-The C# runner (actions/runner, `DistributedTask/Expressions2`) is the12+Three kinds of evidence back the tests:
18-implementation that runs on github.com. It was not available while this was
19-written. Nothing below has been checked against it.
20 13
21-## Limits added by this port (not in the JavaScript)14+- `testdata/oracle.json.gz`: what the JavaScript answers on 34 765 expressions
15+ (tokens, error messages, result kinds, string forms, `toJSON`), generated by
16+ `gen/oracle.mjs`; `TestOracle` replays it, and excuses a mismatch only where
17+ the runner gives this package's answer.
18+- `testdata/runner.json.gz`: what the runner's own code answers on the same
19+ expressions. `gen/runner` compiles `src/Sdk/DTExpressions2` (plus stubs for
20+ the few assemblies it needs, see `gen/runner/README.md`) and runs it;
21+ `TestRunnerOracle` holds the package to it: 34 765 cases, the only
22+ differences are negative zero (below) and error texts of Json.NET.
23+ The same driver recorded `testdata/tojson/*.runner`, `ordinal_cases.txt`
24+ and `number_cases.txt`, and checked the number formatting on 200 000 random
25+ doubles.
26+- `testdata/languageservices/*.json`: the 1 020 cross-language fixtures of
27+ actions/languageservices (see `xlang_test.go`): 1 018 pass, 2 are known
28+ differences with a reason: `case/27` (the runner reports an even `case()` on
29+ the `(`, not the name) and `syntax-errors/memory-errors/0` (the runner's
30+ memory accounting). `testdata/languageservices/SOURCE` has the commit.
22 31
23-The reference has no bound on what an evaluation may build. These are32+The runner's own expression tests (`ExpressionParserL0`, `ConditionFunctionsL0`)
33+are ported in `runner_l0_test.go`. The casing tables of the JavaScript
34+(`casing.go`, `gen/casing*`) remain for `Lex`/oracle support only.
35+
36+Regenerating: `dotnet build -c Release expr/gen/runner` then the `*.mjs` scripts
37+in `expr/gen/runner` (they need a clone of actions/runner and the .NET SDK, not
38+`go test`). Runner commit used: `5f0b3ddcc6ee9fe9441e00d55e85b94ea4aad798`,
39+.NET 9.0.20.
40+
41+Caveat for all of the above: the runner has two copies of the engine,
42+`src/Sdk/DTExpressions2` (used by the legacy template evaluator) and
43+`src/Sdk/Expressions` (used by the newer `WorkflowTemplateEvaluator`, which the
44+runner currently runs next to the legacy one and compares). They are almost the
45+same code; the differences found are noted where they matter (negative zero,
46+`StrictJsonParsing`, `AlwaysTraceRealized`). The differential run uses
47+`DTExpressions2`.
48+
49+## Limits added by this port (not in the JavaScript or the runner)
50+
51+The JavaScript has no bound on what an evaluation may build. These are
24 additions, each reported as an `*EvalError` (or `*Error` for the parser) and52 additions, each reported as an `*EvalError` (or `*Error` for the parser) and
25 none changes the result of an evaluation that stays within them.53 none changes the result of an evaluation that stays within them.
26 54
@@ -38,8 +66,8 @@ none changes the result of an evaluation that stays within them.
38 loops of filters, `join` and `contains`, and passes `ctx` to extension66 loops of filters, `join` and `contains`, and passes `ctx` to extension
39 functions.67 functions.
40 - **Evaluation depth** of 1000 for hand-built trees (parsed trees stay far68 - **Evaluation depth** of 1000 for hand-built trees (parsed trees stay far
41- below it), **JSON depth** `MaxJSONDepth` = 1000 for `FromJSON` and for the69+ below it), **JSON depth** `MaxJSONDepth` = 1000 for the
42- encoder (cyclic values built with `Array.Append` yield an error from70+ encoder (`FromJSON` is limited to 64 levels like Json.NET) (cyclic values built with `Array.Append` yield an error from
43 `EncodeJSON`, `"null"` from `ToJSON`, an `*EvalError` from `toJSON()`).71 `EncodeJSON`, `"null"` from `ToJSON`, an `*EvalError` from `toJSON()`).
44 The JS recursion limit is engine-dependent, around 5 000 to 10 000.72 The JS recursion limit is engine-dependent, around 5 000 to 10 000.
45 - **Built-in arity is re-checked at evaluation**, so a hand-built AST gets an73 - **Built-in arity is re-checked at evaluation**, so a hand-built AST gets an
@@ -47,8 +75,8 @@ none changes the result of an evaluation that stays within them.
47 - **Input length** is checked before the input is converted, without75 - **Input length** is checked before the input is converted, without
48 allocating.76 allocating.
49 77
50-Other limits that exist in the JS and are ported: expression length 2100078+Limits that the runner has and this package follows: expression length 21000
51-UTF-16 units and parser depth 50 (see below).79+UTF-16 units and tree depth 50 (see below).
52 80
53 ## Performance notes (deferred)81 ## Performance notes (deferred)
54 82
@@ -58,11 +86,10 @@ UTF-16 units and parser depth 50 (see below).
58 same contexts, add a prepared `Environment` (fold once, reuse) and cache86 same contexts, add a prepared `Environment` (fold once, reuse) and cache
59 folded names in the AST.87 folded names in the AST.
60 88
61-## Where the JavaScript is believed to differ from the C# runner89+## Where the JavaScript differs from the C# runner (and what this package does)
62 90
63-Each item is implemented as the JavaScript does. Check the runner's tests91+Each item names the runner source that decides it. All are resolved; the
64-(`src/Test/L0/Expressions*`, `Sdk/Expressions2/...`) and decide for each one92+package follows the runner.
65-whether to keep the JS behaviour or follow the runner.
66 93
67 - [x] **String escapes: every `''` is unescaped.** Resolved: follows the C#.94 - [x] **String escapes: every `''` is unescaped.** Resolved: follows the C#.
68 `LexicalAnalyzer.ReadStringToken` (`Tokens/LexicalAnalyzer.cs`, the95 `LexicalAnalyzer.ReadStringToken` (`Tokens/LexicalAnalyzer.cs`, the
@@ -204,41 +231,88 @@ whether to keep the JS behaviour or follow the runner.
204 `join(success(), format('{2}'))` is `true` in the runner and an error in231 `join(success(), format('{2}'))` is `true` in the runner and an error in
205 the JS. Built-ins now receive the call and evaluate what they need;232 the JS. Built-ins now receive the call and evaluate what they need;
206 extension functions still get evaluated arguments.233 extension functions still get evaluated arguments.
207-- [ ] **Index access**: `a[b]` where `b` is a number uses `floor(b)` and234+- [x] **Large integers in `fromJSON`**: found by the differential run. Json.NET
208- rejects NaN, negatives and infinities; where `b` is a string it is a key235+ reads an integer that does not fit in 64 bits as a `BigInteger`, and
209- or a decimal index (`'1'`, `' 1 '`, `'1.9'` all index 1 of an array).236+ `(Double)BigInteger` truncates toward zero to 53 bits instead of rounding
210- Booleans and null index as `1`/`0` on arrays and as `"true"`/`""` keys on237+ to nearest, so `fromJSON('18446744073709551615')` is 2^64-2048, not 2^64,
211- dictionaries.238+ and does not equal the literal `18446744073709551615` of an expression
212-- [ ] **Filtered arrays** (`a.*`): a dictionary filtered by `*` yields its239+ (which `Double.TryParse` rounds to nearest). Reproduced in
213- values; `.*.b` skips elements without `b`; `.*.*` flattens one level;240+ `jsonParser.number` (`TestFromJSONBigIntegers`); checked on 57 values.
214- arrays produced by `*` are fresh, so `a.* == a.*` is false.241+- [x] **Index access**: resolved, same in the C# (`Sdk/Operators/Index.cs`,
215-- [ ] **Comparison**: `null` is only ordered against numbers/booleans (it242+ `IndexHelper`): a number is floored, NaN, negatives and numbers above
216- becomes 0); arrays and dictionaries are never ordered and `==` is243+ Int32.MaxValue give no index; a primitive also has a string form for
217- identity; `NaN` is unordered and unequal to itself; `true > false`.244+ dictionary keys, so booleans and null are `"true"`/`""` there and 1/0 on
218-- [ ] **`&&`/`||` return an operand**, not a boolean (`'' || 'x'` is `'x'`).245+ arrays; `' 1 '` and `'1.9'` index element 1. Differences found: the
219-- [ ] **Limits**: max expression length 21000 (UTF-16 units; the JS throws a246+ index is only evaluated when the object is an array or a dictionary, and
220- bare `Error("ErrorExceededMaxLength")`), max parse depth 50. The depth247+ its errors are not wrapped (see the lazy evaluation item above).
221- counting (including `index()` and the flat logical chains) is ported248+ `TestRunnerSemantics`.
222- literally; C# `MaxDepth` is 50 with its own counting. Cross-check with249+- [x] **Filtered arrays** (`a.*`): resolved, same in the C#
223- deep-nesting tests. There is no node-count or evaluation-memory limit250+ (`Index.HandleFilteredArray`): `*` on a dictionary yields its values,
224- here; the C# runner has `MaxMemory` (1 MB) and a result-size limit.251+ `.*.b` skips elements without `b`, `.*.*` flattens one level, and every
225-- [ ] **Trailing comma in calls**: `contains('a', 'b',)` parses (the JS loop252+ `*` builds a new array, so `a.* == a.*` is false. `TestRunnerSemantics`.
226- only requires a comma when the next token is not `)`). `f(,)` does not.253+- [x] **Comparison**: resolved, same in the C# (`EvaluationResult.cs`
227-- [ ] **Whitespace**: only space, `\t`, `\r`, `\n` separate tokens. U+00A0 and254+ `CoerceTypes`/`AbstractEqual`/`AbstractGreaterThan`): null and booleans
228- other Unicode spaces end a word but are not skipped, so `1 \u00a0 2`255+ become numbers when compared with another kind, a string compared with a
229- reports an unexpected symbol. `\v`, `\f` likewise.256+ number is parsed with `ParseNumber`, arrays and dictionaries compare by
230-- [ ] **Empty expression**: the JS parser returns `undefined` for no tokens.257+ reference and are never ordered, NaN is unequal to itself and unordered.
231- `Parse` returns a nil Node and no error; `Evaluate(nil)` errors.258+ Strings compare with `OrdinalIgnoreCase` (see above). `TestRunnerSemantics`.
232-- [ ] **Unknown contexts at evaluation time** are null (the JS `Dictionary.get`259+- [x] **`&&`/`||` return an operand**: resolved, same in the C# (`And.cs`,
233- returns `undefined` and `visitContextAccess` then crashes later); here260+ `Or.cs`: they return `result.Value` of the deciding operand).
234- declared-but-absent contexts are null, never a crash.261+- [x] **Limits**: resolved, follows the C#. Length 21000 UTF-16 units
235-- [ ] **Error positions**: the JS EOF token's start column is negative after a262+ (`ExpressionConstants.MaxLength`): "Exceeded max expression length 21000",
236- trailing newline (stale `start`); ported as is263+ without a token (the JS throws a bare `ErrorExceededMaxLength`). Depth 50
237- (`TestEOFTokenPosition`).264+ is checked on the finished tree (`ExpressionParser.CheckMaxDepth`): the
238-- [ ] **Function names and context names are case-insensitive**; the JS265+ root is depth 1, an operand of an operator, an argument of a call and the
239- lower-cases with `toLowerCase`, as does this port (`foldKey`). Extension266+ two sides of an index are one deeper, **parentheses add nothing**, and a
240- functions override built-ins at evaluation (`functions?.get(...) ||267+ chain of `&&` or `||` is one node, nested ones included, so
241- wellKnownFunctions[...]`) but built-ins win at validation time (arity).268+ `((((...1...))))` with 500 parentheses and `1 || (1 || (1 ...))` parse;
269+ `"!" * 50 + "true"` and 50 comparisons chained with `==` do not. The error
270+ has no token: "Exceeded max expression depth 50". The JS limit counted
271+ parser recursion (parentheses included). Memory: the runner has
272+ `EvaluationOptions.MaxMemory` (1 MiB by default) and fails with "The
273+ maximum allowed memory size was exceeded while evaluating the following
274+ expression: ...", counting strings at 26 bytes + 2 per UTF-16 unit and 24
275+ per object, per call depth (`EvaluationMemory`). This package keeps its
276+ own output budget (see above), which is not byte-for-byte the same, so
277+ the fixture `syntax-errors.json` "memory-errors" is a known difference.
278+- [x] **Trailing comma in calls**: resolved, follows the C#. `contains('a', 'b',)`
279+ is "Unexpected symbol: ')'" (the tokenizer rejects a `)` after a `,`). The JS
280+ accepted it. `f(,)` and `contains(,'a')` fail as before.
281+- [x] **Whitespace**: resolved, follows the C#. `Char.IsWhiteSpace` separates
282+ tokens: U+0009..U+000D, U+0020, U+0085, U+00A0, U+1680, U+2000..U+200A,
283+ U+2028, U+2029, U+202F, U+205F, U+3000. So `1 \\u00a0 2` is "Unexpected
284+ symbol: '2'" (not the NBSP), and `\\v1` is 1. The JS skipped only space,
285+ tab, CR and LF. U+FEFF is not white space.
286+- [x] **Empty expression**: same in the C# (`CreateTree` returns null when there
287+ is no token). `Parse` returns a nil Node and no error; `Evaluate(nil)`
288+ errors; callers (the template layer) decide.
289+- [x] **Unknown contexts at evaluation time** are null: in the runner a
290+ declared named value is evaluated by the caller's node; here an absent
291+ key of `EvalContext.Contexts` is null.
292+- [x] **Error positions**: resolved, follows the C#. A `ParseException` names
293+ the token and its **absolute 1-based index in the expression**
294+ (`Located at position N within expression: <whole expression>`);
295+ `Error.Pos` is the same place as a zero-based line and column.
296+ Arity and even-argument errors name the `(` of the call (the token
297+ `FlushTopEndParameters` holds), not the function: `contains()` is
298+ "Too few parameters supplied: '('. Located at position 9". The
299+ end-of-input error is "Unexpected end of expression: '<last token>'"
300+ (the JS said "Unexpected symbol: 'EOF'"), and the JS's negative EOF column
301+ is gone with the JS lexer. Unexpected-symbol tokens are the
302+ boundary-delimited run (`ReadKeywordToken`): `github..event` reports `.`
303+ where the JS reported `.event`. Lexing and parsing are interleaved, so the
304+ first problem in reading order wins (the JS lexed everything first).
305+ `TestParseErrorPositions`, and `TestRunnerOracle` on 1 700 error cases.
306+ One fixture still differs, see `xlangKnownDifferences`: `case/27` expects
307+ the name for an even `case()`, the runner passes the `(`.
308+- [x] **Function names and context names are case-insensitive**: same in the
309+ C# (`StringComparer.OrdinalIgnoreCase` dictionaries; see the folding
310+ item). Built-ins win over extension functions of the same name at
311+ validation **and at evaluation** (`ExpressionParser.TryGetFunctionInfo`
312+ looks in `ExpressionConstants.WellKnownFunctions` first and builds the
313+ node from it); the JS let an extension override a built-in at
314+ evaluation. An extension supplied under a built-in's name is now never
315+ called.
242 316
243 ## JavaScript behaviours deliberately NOT ported317 ## JavaScript behaviours deliberately NOT ported
244 318
@@ -257,36 +331,64 @@ a bug.
257 evaluating e.g. `github.true` with an empty context: the reference does not331 evaluating e.g. `github.true` with an empty context: the reference does not
258 handle a missing root context. Here it is null.332 handle a missing root context. Here it is null.
259 - Recursion limits: JS recursion depth for `fromJSON` input is engine333 - Recursion limits: JS recursion depth for `fromJSON` input is engine
260- dependent (about 5 000 to 10 000 levels). `FromJSON` accepts `MaxJSONDepth`334+ dependent. `FromJSON` follows Json.NET's `MaxDepth` of 64.
261- (1000) levels.335+- Lone surrogates: JS strings are UTF-16 and can hold them. The runner cannot
262-- Lone surrogates: JS strings are UTF-16 and can hold them; Go strings are336+ (Json.NET reads an unpaired `\ud800` as U+FFFD), and neither can Go strings;
263- UTF-8. They are kept as WTF-8 (`0xED 0xA0-0xBF xx`) so that `fromJSON` /337+ the WTF-8 machinery of the first version (`wtf8Surrogate`, `normalizeWTF8`)
264- `toJSON` round-trip, `==`/`<`/`contains`/`startsWith`/`endsWith` compare338+ is only used by the case-folding and comparison helpers for robustness and
265- them like code units, case mapping copies them, and `format`/`join` rejoin a339+ by the oracle tests.
266- high surrogate followed by a low one into the character they form. This is340+
267- covered by oracle cases. Two surrogate strings concatenated by other means341+## Still open
268- (an extension function) are not rejoined.342+
343+- [ ] **Memory accounting.** The runner caps an evaluation at `MaxMemory` (1 MiB)
344+ counted by `EvaluationMemory`/`MemoryCounter` (26 bytes + 2 per UTF-16 unit
345+ per string, 24 per object, trimmed per call depth); this package caps
346+ `MaxResultBytes` (1 MiB of output, different units) and accepts
347+ `format(...)` chains the runner rejects. Fixture
348+ `syntax-errors/memory-errors/0` is the known difference. Reproducing
349+ `EvaluationMemory` exactly would need the node depth of every value;
350+ worth doing only if the error text matters ("The maximum allowed memory
351+ size was exceeded while evaluating the following expression: ...", which
352+ needs `ConvertToExpression` of the root, also not ported).
353+- [ ] **Negative zero.** `-0` is `0` here. `DTExpressions2/EvaluationResult.cs`
354+ (the legacy engine) prints `-0` for `format('{0}', -0)` and `toJSON(-0)`,
355+ because .NET Core prints `-0`; `Expressions/EvaluationResult.cs` (the engine of
356+ `WorkflowTemplateEvaluator`) maps it to `0`, and the cross-language fixture
357+ `coerce_string.json` expects `0`. If the legacy engine is what a user sees,
358+ `-0` would be right in `format()`; the choice was made for the fixture.
359+ `toJSON` goes through `ConvertToString` in both and so follows the same
360+ choice.
361+- [ ] **Comments and constructors in `fromJSON`.** Json.NET accepts `/* */` and
362+ `//` comments and `new Date(1)` constructors; the first is read in a
363+ surprising way (`/*x*/1` is the string `x`). They fail here.
364+- [ ] **`fromJSON` error texts.** Json.NET messages ("Unexpected character
365+ encountered while parsing value: a. Path '', line 0, position 0.") are
366+ not reproduced; the outer message "Error parsing JSON when evaluating
367+ fromJson" is the JS one, the runner's exception message is the Json.NET
368+ one. Only the failure is checked.
369+- [ ] **`StrictJsonParsing`.** The newer engine has an option that switches
370+ `fromJSON` to `System.Text.Json` (strict, 64 levels, different
371+ messages); default off, "temporary for telemetry". Not implemented.
372+- [ ] **Unicode version of the folding table** is that of the .NET 9 runtime that
373+ generated `fold_tables.go`; a runner on another runtime may fold newly
374+ assigned letters differently.
375+- [ ] **Tracing and secret masking** (`ITraceWriter`, `ISecretMasker`,
376+ `EvaluationResult` tracing) are runner features outside this package.
269 377
270-## Other things to check once the runner is available378+## Left to the callers
271 379
272-- [ ] Run the runner's expression test vectors (`ExpressionParserL0`,380+- Run the runner's expression test vectors: done for `ExpressionParserL0` and
273- `ExpressionTests`, `ExpressionValuesL0`, `ExpressionTestUtil`) through a381+ `ConditionFunctionsL0` (`runner_l0_test.go`). The runner has no other
274- table adapter; list every disagreement here before deciding.382+ expression tests in `src/Test/L0` (the larger suites are in the closed
275-- [ ] Decide whether `EvalContext` needs a hook for the functions the runner383+ service); the cross-language fixtures cover the rest.
276- implements natively with special semantics: `success()`, `failure()`,384+- The functions the runner implements natively with special semantics:
277- `always()`, `cancelled()` (take no arguments; read job status) and385+ `success()`, `failure()`, `always()`, `cancelled()` (no arguments; read job
278- `hashFiles()` (variadic glob patterns, string result). The API supports386+ status, or `action_status` in a composite action's main step) and
279- them through `FuncInfo` + `Func` without changes; arity is declared by387+ `hashFiles()` (variadic, string result) are supplied by the caller through
280- the caller.388+ `FuncInfo` + `Func`; `runner_l0_test.go` shows the declaration the runner
281-- [ ] Context names the runner allows per location (`github`, `env`, `vars`,389+ uses (`FunctionInfo(name, 0, 0)`).
282- `job`, `jobs`, `steps`, `runner`, `secrets`, `strategy`, `matrix`,390+- Context names per location and which functions each location may call belong
283- `needs`, `inputs`) and which functions each location may call391+ to the template layer.
284- (`success`... only in `if`/`jobs.*.if`...). This belongs to the template392+- The `format()` argument limit is 255 in both (`Byte.MaxValue`).
285- layer, not to package expr.393+- Secret masking and the `${{ }}` splitting of a larger string are not part of
286-- [ ] `format()` argument limit is 255 in the JS (`MAX_ARGUMENTS`); the C#394+ this package: `Parse` takes the text between the braces only.
287- limit is `ExpressionConstants.MaxArguments` (check).
288-- [x] Case-insensitive comparison and `contains`: `OrdinalIgnoreCase` is the
289- truth, see above.
290-- [ ] Secret masking and the `${{ }}` splitting of a larger string are not
291- part of this package (they belong to the template layer): `Parse`
292- takes the text between the braces only.
M expr/doc.go +9 -6
@@ -6,12 +6,15 @@
66 // Package expr implements GitHub Actions expressions: the language found
77 // between the braces of ${{ }} in workflow files.
88 //
9-// The behaviour is a faithful port of GitHub's own implementation published
10-// as the npm package @actions/expressions (MIT, actions/languageservices):
11-// same tokens, precedence, error messages, coercions and built-in functions.
12-// Where that implementation is believed to differ from the hosted service
13-// (the C# runner in actions/runner), the Go code follows the JavaScript and
14-// says so in a comment; PORTING.md lists what remains to be checked.
9+// The behaviour is that of the C# runner (actions/runner, MIT), the
10+// implementation that evaluates expressions on github.com: its tokenizer and
11+// parser, error messages, coercions, case-insensitive comparison, number
12+// formatting, JSON reader and writer, and built-in functions. It was first
13+// ported from GitHub's editor library, the npm package @actions/expressions
14+// (MIT, actions/languageservices), then checked against the runner on 34 765
15+// expressions and against the shared cross-language fixtures; where the two
16+// differ, the runner wins. PORTING.md lists every difference with the C#
17+// source that decides it.
1518 //
1619 // The entry points are [Parse], which turns the text of one expression into a
1720 // [Node], and [Evaluate], which computes its [Value] against the named
@@ -6,12 +6,15 @@
6 // Package expr implements GitHub Actions expressions: the language found6 // Package expr implements GitHub Actions expressions: the language found
7 // between the braces of ${{ }} in workflow files.7 // between the braces of ${{ }} in workflow files.
8 //8 //
9-// The behaviour is a faithful port of GitHub's own implementation published9+// The behaviour is that of the C# runner (actions/runner, MIT), the
10-// as the npm package @actions/expressions (MIT, actions/languageservices):10+// implementation that evaluates expressions on github.com: its tokenizer and
11-// same tokens, precedence, error messages, coercions and built-in functions.11+// parser, error messages, coercions, case-insensitive comparison, number
12-// Where that implementation is believed to differ from the hosted service12+// formatting, JSON reader and writer, and built-in functions. It was first
13-// (the C# runner in actions/runner), the Go code follows the JavaScript and13+// ported from GitHub's editor library, the npm package @actions/expressions
14-// says so in a comment; PORTING.md lists what remains to be checked.14+// (MIT, actions/languageservices), then checked against the runner on 34 765
15+// expressions and against the shared cross-language fixtures; where the two
16+// differ, the runner wins. PORTING.md lists every difference with the C#
17+// source that decides it.
15 //18 //
16 // The entry points are [Parse], which turns the text of one expression into a19 // The entry points are [Parse], which turns the text of one expression into a
17 // [Node], and [Evaluate], which computes its [Value] against the named20 // [Node], and [Evaluate], which computes its [Value] against the named
M expr/eval.go +21 -26
@@ -42,8 +42,8 @@ type EvalContext struct {
4242 Contexts map[string]Value
4343 // Funcs implements the extra functions declared to [Parse], such as
4444 // hashFiles, success, always, cancelled and failure. Names are matched
45- // case-insensitively. As in the reference implementation these take
46- // precedence over the built-in functions.
45+ // case-insensitively. The built-in functions cannot be
46+ // replaced: as in the runner, they win over a function of the same name.
4747 Funcs map[string]Func
4848 // MaxResultBytes is the output budget of the evaluation, see
4949 // [MaxResultBytes]. Zero means the default; a negative value removes the
@@ -233,25 +233,22 @@ func (e *evaluator) logical(l *Logical) (Value, error) {
233233
234234 func (e *evaluator) call(c *FunctionCall) (Value, error) {
235235 name := foldKey(c.Name.Lexeme)
236- ext, isExt := e.funcs[name]
237- isExt = isExt && ext != nil
238- var impl builtinFunc
239- if !isExt {
240- // Arity of the built-ins is checked by the parser, and again here
241- // so that a hand-built tree cannot reach an implementation with the
242- // wrong number of arguments.
243- f, ok := builtinFuncs[name]
244- if !ok {
245- return nil, evalErrorf("function %s is not implemented", c.Name.Lexeme)
246- }
236+ // Built-ins win over extension functions of the same name, as in the
237+ // runner (ExpressionParser.TryGetFunctionInfo looks in WellKnownFunctions
238+ // first); the JavaScript reference let an extension override them here.
239+ if f, ok := builtinFuncs[name]; ok {
240+ // Arity is checked by the parser, and again here so that a hand-built
241+ // tree cannot reach an implementation with the wrong number of
242+ // arguments.
247243 if t, bad := arityError(name, f, len(c.Args)); bad {
248244 return nil, &EvalError{Message: t.description() + ": '" + c.Name.Lexeme + "'"}
249245 }
250- impl = builtinImpls[name]
251- }
252- if !isExt {
253246 // Built-ins evaluate their own arguments, lazily.
254- return impl(e, c)
247+ return builtinImpls[name](e, c)
248+ }
249+ ext, ok := e.funcs[name]
250+ if !ok || ext == nil {
251+ return nil, evalErrorf("function %s is not implemented", c.Name.Lexeme)
255252 }
256253 args := make([]Value, len(c.Args))
257254 for i, a := range c.Args {
@@ -261,17 +258,15 @@ func (e *evaluator) call(c *FunctionCall) (Value, error) {
261258 }
262259 args[i] = v
263260 }
264- {
265- v, err := ext(e.ctx, args)
266- if err != nil {
267- var ee *EvalError
268- if errors.As(err, &ee) {
269- return nil, err
270- }
271- return nil, &EvalError{Message: err.Error(), Err: err}
261+ v, err := ext(e.ctx, args)
262+ if err != nil {
263+ var ee *EvalError
264+ if errors.As(err, &ee) {
265+ return nil, err
272266 }
273- return v, nil
267+ return nil, &EvalError{Message: err.Error(), Err: err}
274268 }
269+ return v, nil
275270 }
276271
277272 // indexer is idxHelper of the reference: how an index value is read, as a
@@ -42,8 +42,8 @@ type EvalContext struct {
42 Contexts map[string]Value42 Contexts map[string]Value
43 // Funcs implements the extra functions declared to [Parse], such as43 // Funcs implements the extra functions declared to [Parse], such as
44 // hashFiles, success, always, cancelled and failure. Names are matched44 // hashFiles, success, always, cancelled and failure. Names are matched
45- // case-insensitively. As in the reference implementation these take45+ // case-insensitively. The built-in functions cannot be
46- // precedence over the built-in functions.46+ // replaced: as in the runner, they win over a function of the same name.
47 Funcs map[string]Func47 Funcs map[string]Func
48 // MaxResultBytes is the output budget of the evaluation, see48 // MaxResultBytes is the output budget of the evaluation, see
49 // [MaxResultBytes]. Zero means the default; a negative value removes the49 // [MaxResultBytes]. Zero means the default; a negative value removes the
@@ -233,25 +233,22 @@ func (e *evaluator) logical(l *Logical) (Value, error) {
233 233
234 func (e *evaluator) call(c *FunctionCall) (Value, error) {234 func (e *evaluator) call(c *FunctionCall) (Value, error) {
235 name := foldKey(c.Name.Lexeme)235 name := foldKey(c.Name.Lexeme)
236- ext, isExt := e.funcs[name]236+ // Built-ins win over extension functions of the same name, as in the
237- isExt = isExt && ext != nil237+ // runner (ExpressionParser.TryGetFunctionInfo looks in WellKnownFunctions
238- var impl builtinFunc238+ // first); the JavaScript reference let an extension override them here.
239- if !isExt {239+ if f, ok := builtinFuncs[name]; ok {
240- // Arity of the built-ins is checked by the parser, and again here240+ // Arity is checked by the parser, and again here so that a hand-built
241- // so that a hand-built tree cannot reach an implementation with the241+ // tree cannot reach an implementation with the wrong number of
242- // wrong number of arguments.242+ // arguments.
243- f, ok := builtinFuncs[name]
244- if !ok {
245- return nil, evalErrorf("function %s is not implemented", c.Name.Lexeme)
246- }
247 if t, bad := arityError(name, f, len(c.Args)); bad {243 if t, bad := arityError(name, f, len(c.Args)); bad {
248 return nil, &EvalError{Message: t.description() + ": '" + c.Name.Lexeme + "'"}244 return nil, &EvalError{Message: t.description() + ": '" + c.Name.Lexeme + "'"}
249 }245 }
250- impl = builtinImpls[name]
251- }
252- if !isExt {
253 // Built-ins evaluate their own arguments, lazily.246 // Built-ins evaluate their own arguments, lazily.
254- return impl(e, c)247+ return builtinImpls[name](e, c)
248+ }
249+ ext, ok := e.funcs[name]
250+ if !ok || ext == nil {
251+ return nil, evalErrorf("function %s is not implemented", c.Name.Lexeme)
255 }252 }
256 args := make([]Value, len(c.Args))253 args := make([]Value, len(c.Args))
257 for i, a := range c.Args {254 for i, a := range c.Args {
@@ -261,17 +258,15 @@ func (e *evaluator) call(c *FunctionCall) (Value, error) {
261 }258 }
262 args[i] = v259 args[i] = v
263 }260 }
264- {261+ v, err := ext(e.ctx, args)
265- v, err := ext(e.ctx, args)262+ if err != nil {
266- if err != nil {263+ var ee *EvalError
267- var ee *EvalError264+ if errors.As(err, &ee) {
268- if errors.As(err, &ee) {265+ return nil, err
269- return nil, err
270- }
271- return nil, &EvalError{Message: err.Error(), Err: err}
272 }266 }
273- return v, nil267+ return nil, &EvalError{Message: err.Error(), Err: err}
274 }268 }
269+ return v, nil
275 }270 }
276 271
277 // indexer is idxHelper of the reference: how an index value is read, as a272 // indexer is idxHelper of the reference: how an index value is read, as a
M expr/eval_test.go +71 -58
@@ -81,14 +81,14 @@ func TestEvalLiteralsAndOperators(t *testing.T) {
8181 {"0x10", "Number:16"},
8282 {"0o17", "Number:15"},
8383 {"0xffffffff", "Number:-1"}, // read as a 32-bit integer, like the runner
84- {"0b101", "PARSE Unexpected symbol: '0b101'. Located at position 1 within expression: 0b101"},
85- {"0X10", "PARSE Unexpected symbol: '0X10'. Located at position 1 within expression: 0X10"},
84+ {"0b101", "PARSE Unexpected symbol: '0b101'"},
85+ {"0X10", "PARSE Unexpected symbol: '0X10'"},
8686 {"'0b11' == 3", "Boolean:false"},
8787 {"'0x1F' == 31", "Boolean:true"},
8888 {"'infinity' == Infinity", "Boolean:true"},
8989 {"'\u0085 1 \u0085' == 1", "Boolean:true"},
9090 {"'\ufeff1' == 1", "Boolean:false"},
91- {"-0x10", "PARSE Unexpected symbol: '-0x10'. Located at position 1 within expression: -0x10"},
91+ {"-0x10", "PARSE Unexpected symbol: '-0x10'"},
9292 {"NaN", "Number:NaN"},
9393 {"Infinity", "Number:Infinity"},
9494 {"-Infinity", "Number:-Infinity"},
@@ -290,7 +290,7 @@ func TestEvalAccess(t *testing.T) {
290290 {"steps[10].outputs.r", "String:ten"},
291291 {"steps['10'].outputs.r", "String:ten"},
292292 // After an identifier, ".10" is not a number: the lexer reads a DOT and then "10.outputs.r" as one number-like word.
293- {"steps.10.outputs.r", "PARSE Unexpected symbol: '10.outputs.r'. Located at position 7 within expression: steps.10.outputs.r"},
293+ {"steps.10.outputs.r", "PARSE Unexpected symbol: '10.outputs.r'"},
294294 {"steps[2].outputs.r", "String:two"},
295295 {"matrix.obj[0]", "Null:"},
296296 {"matrix.node['0']", "Number:14"},
@@ -366,9 +366,9 @@ func TestEvalBuiltinFunctions(t *testing.T) {
366366 {"contains('Dž', 'dž')", "Boolean:true"},
367367 {"contains('A\u03a3', 'a\u03c2')", "Boolean:true"},
368368 {"contains('K', 'k')", "Boolean:true"}, // KELVIN SIGN lower-cases to k
369- {"contains()", "PARSE Too few parameters supplied: 'contains'"},
370- {"contains(1)", "PARSE Too few parameters supplied: 'contains'"},
371- {"contains(1, 2, 3)", "PARSE Too many parameters supplied: 'contains'"},
369+ {"contains()", "PARSE Too few parameters supplied: '('"},
370+ {"contains(1)", "PARSE Too few parameters supplied: '('"},
371+ {"contains(1, 2, 3)", "PARSE Too many parameters supplied: '('"},
372372
373373 // startsWith, endsWith
374374 {"startsWith('abc', 'AB')", "Boolean:true"},
@@ -383,7 +383,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
383383 {"endsWith('abc', 'abcd')", "Boolean:false"},
384384 {"endsWith(1.50, '5')", "Boolean:true"},
385385 {"endsWith(1.50, 5)", "Boolean:true"},
386- {"endsWith('abc')", "PARSE Too few parameters supplied: 'endsWith'"},
386+ {"endsWith('abc')", "PARSE Too few parameters supplied: '('"},
387387 {"STARTSWITH('abc', 'a')", "Boolean:true"},
388388
389389 // join
@@ -400,8 +400,8 @@ func TestEvalBuiltinFunctions(t *testing.T) {
400400 {"join(fromJSON('[1,2,3]'), null)", "String:123"},
401401 {"join(fromJSON('[1,2,3]'), github)", "String:1,2,3"}, // a non-primitive separator is ignored
402402 {"join(fromJSON('[null,true,\"a\",1.5,[1],{}]'))", "String:,true,a,1.5,Array,Object"},
403- {"join()", "PARSE Too few parameters supplied: 'join'"},
404- {"join(1, 2, 3)", "PARSE Too many parameters supplied: 'join'"},
403+ {"join()", "PARSE Too few parameters supplied: '('"},
404+ {"join(1, 2, 3)", "PARSE Too many parameters supplied: '('"},
405405
406406 // toJSON
407407 {"toJSON(null)", "String:null"},
@@ -417,8 +417,8 @@ func TestEvalBuiltinFunctions(t *testing.T) {
417417 {"toJSON(fromJSON('[]'))", "String:[]"},
418418 {"toJSON(fromJSON('{}'))", "String:{}"},
419419 {"toJSON(fromJSON('[[],{}]'))", "String:[\n [],\n {}\n]"},
420- {"toJSON(1, 2)", "PARSE Too many parameters supplied: 'toJSON'"},
421- {"toJSON()", "PARSE Too few parameters supplied: 'toJSON'"},
420+ {"toJSON(1, 2)", "PARSE Too many parameters supplied: '('"},
421+ {"toJSON()", "PARSE Too few parameters supplied: '('"},
422422 {"fromJSON(toJSON(github.event.commits[0])).author.name", "String:Ann"},
423423
424424 // fromJSON
@@ -442,7 +442,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
442442 {"fromJSON(1)", "Number:1"},
443443 {"fromJSON(true)", "Boolean:true"},
444444 {"fromJSON(matrix.node)", "ERR Error parsing JSON when evaluating fromJson"}, // the string "Array"
445- {"fromJSON(1, 2)", "PARSE Too many parameters supplied: 'fromJSON'"},
445+ {"fromJSON(1, 2)", "PARSE Too many parameters supplied: '('"},
446446
447447 // format
448448 {"format('a')", "String:a"},
@@ -493,7 +493,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
493493 {"format('{0}}', 1)", "ERR The following format string is invalid: {0}}"},
494494 // The index is a byte: a longer number makes the string invalid.
495495 {"format('{12345678901234567890123}', 1)", "ERR The following format string is invalid: {12345678901234567890123}"},
496- {"format()", "PARSE Too few parameters supplied: 'format'"},
496+ {"format()", "PARSE Too few parameters supplied: '('"},
497497
498498 // case
499499 {"case(true, 'a', 'b')", "String:a"},
@@ -506,9 +506,9 @@ func TestEvalBuiltinFunctions(t *testing.T) {
506506 {"case(1, 2, 3)", "ERR case predicate must evaluate to a boolean value"},
507507 {"case('true', 2, 3)", "ERR case predicate must evaluate to a boolean value"},
508508 {"case(null, 2, 3)", "ERR case predicate must evaluate to a boolean value"},
509- {"case(true, 1, 2, 3)", "PARSE Even number of parameters supplied, requires an odd number of parameters: 'case'"},
510- {"case(true, 1)", "PARSE Too few parameters supplied: 'case'"},
511- {"case()", "PARSE Too few parameters supplied: 'case'"},
509+ {"case(true, 1, 2, 3)", "PARSE Even number of parameters supplied, requires an odd number of parameters: '('"},
510+ {"case(true, 1)", "PARSE Too few parameters supplied: '('"},
511+ {"case()", "PARSE Too few parameters supplied: '('"},
512512 // Predicates after the first true one are not checked...
513513 {"case(true, 1, 'bad', 2, 3)", "Number:1"},
514514 // ...but earlier ones are.
@@ -523,7 +523,7 @@ func TestEvalErrorMessages(t *testing.T) {
523523 {"", "PARSE Unexpected end of expression: 'x'"}, // replaced below: the empty expression is not an error
524524 }[:0])
525525 cases := []evalCase{
526- {"1 +", "PARSE Unexpected symbol: '+'. Located at position 3 within expression: 1 +"},
526+ {"1 +", "PARSE Unexpected symbol: '+'"},
527527 {"a b", "PARSE Unrecognized named-value: 'a'"},
528528 {"github github", "PARSE Unexpected symbol: 'github'"},
529529 {"github.event_name ==", "PARSE Unexpected end of expression: '=='"},
@@ -531,9 +531,9 @@ func TestEvalErrorMessages(t *testing.T) {
531531 {"(github.event_name", "PARSE Unexpected end of expression: 'event_name'"},
532532 {"github.event_name)", "PARSE Unexpected symbol: ')'"},
533533 {"contains('a',", "PARSE Unexpected end of expression: ','"},
534- {"contains('a'", "PARSE Unexpected symbol: 'EOF'"},
535- {"contains('a' 'b')", "PARSE Unexpected symbol: 'b'"},
536- {"foo(", "PARSE Unexpected end of expression: '('"},
534+ {"contains('a'", "PARSE Unexpected end of expression: ''a''"},
535+ {"contains('a' 'b')", "PARSE Unexpected symbol: ''b''"},
536+ {"foo(", "PARSE Unrecognized function: 'foo'"},
537537 {"foo()", "PARSE Unrecognized function: 'foo'"},
538538 {"foo", "PARSE Unrecognized named-value: 'foo'"},
539539 {"unknown(1)", "PARSE Unrecognized function: 'unknown'"},
@@ -545,22 +545,22 @@ func TestEvalErrorMessages(t *testing.T) {
545545 {"1 !", "PARSE Unexpected symbol: '!'"},
546546 {"[1]", "PARSE Unexpected symbol: '['"},
547547 {"github.[0]", "PARSE Unexpected symbol: '['"},
548- {"github.", "PARSE Unexpected symbol: 'EOF'"},
548+ {"github.", "PARSE Unexpected end of expression: '.'"},
549549 {"github[", "PARSE Unexpected end of expression: '['"},
550550 {"github[]", "PARSE Unexpected symbol: ']'"},
551- {"github[*", "PARSE Unexpected symbol: 'EOF'"},
551+ {"github[*", "PARSE Unexpected end of expression: '*'"},
552552 {"github.true", "Null:"}, // keywords are plain identifiers after a dot
553553 {"env.true", "Null:"},
554- {"1 = 1", "PARSE Unexpected symbol: '='. Located at position 3 within expression: 1 = 1"},
555- {"1 & 1", "PARSE Unexpected symbol: '&'. Located at position 3 within expression: 1 & 1"},
556- {"1 | 1", "PARSE Unexpected symbol: '|'. Located at position 3 within expression: 1 | 1"},
557- {"1 + 1", "PARSE Unexpected symbol: '+'. Located at position 3 within expression: 1 + 1"},
554+ {"1 = 1", "PARSE Unexpected symbol: '='"},
555+ {"1 & 1", "PARSE Unexpected symbol: '&'"},
556+ {"1 | 1", "PARSE Unexpected symbol: '|'"},
557+ {"1 + 1", "PARSE Unexpected symbol: '+'"},
558558 {"a-b", "PARSE Unrecognized named-value: 'a-b'"},
559- {"1a", "PARSE Unexpected symbol: '1a'. Located at position 1 within expression: 1a"},
560- {"'unterminated", "PARSE Unexpected symbol: ''unterminated'. Located at position 1 within expression: 'unterminated"},
561- {"@", "PARSE Unexpected symbol: '@'. Located at position 1 within expression: @"},
562- {"${{ 1 }}", "PARSE Unexpected symbol: '${{'. Located at position 1 within expression: ${{ 1 }}"},
563- {"1 \u00a0 2", "PARSE Unexpected symbol: '\u00a0'. Located at position 3 within expression: 1 \u00a0 2"},
559+ {"1a", "PARSE Unexpected symbol: '1a'"},
560+ {"'unterminated", "PARSE Unexpected symbol: ''unterminated'"},
561+ {"@", "PARSE Unexpected symbol: '@'"},
562+ {"${{ 1 }}", "PARSE Unexpected symbol: '${{'"},
563+ {"1 \u00a0 2", "PARSE Unexpected symbol: '2'"}, // U+00A0 is white space for .NET
564564 }
565565 runEvalCases(t, cases)
566566 }
@@ -577,6 +577,9 @@ func TestEmptyExpression(t *testing.T) {
577577 }
578578 }
579579
580+// TestParseErrorPositions: the runner (ParseException) reports the absolute
581+// index of the token in the expression, 1-based, and the whole expression;
582+// Error.Pos is the same place as a line and a column.
580583 func TestParseErrorPositions(t *testing.T) {
581584 tests := []struct {
582585 src string
@@ -584,14 +587,16 @@ func TestParseErrorPositions(t *testing.T) {
584587 msg string
585588 }{
586589 {"1 ==\n 2 ===\n 3", 1, 5, "Unexpected symbol: '='. Located at position 11 within expression: 1 ==\n 2 ===\n 3"},
587- {"true &&\n false ||\n nope", 2, 2, "Unrecognized named-value: 'nope'"},
588- {"\n\n bogus", 2, 2, "Unrecognized named-value: 'bogus'"},
589- {"github github", 0, 7, "Unexpected symbol: 'github'"},
590- {" ) ", 0, 2, "Unexpected symbol: ')'"},
591- {"contains('a',", 0, 12, "Unexpected end of expression: ','"},
592- {"foo()", 0, 0, "Unrecognized function: 'foo'"},
593- {"1 && foo()", 0, 5, "Unrecognized function: 'foo'"},
594- {"'a'\n 'b'\n zzz", 1, 1, "Unexpected symbol: 'b'"},
590+ {"true &&\n false ||\n nope", 2, 2, "Unrecognized named-value: 'nope'. Located at position 22 within expression: true &&\n false ||\n nope"},
591+ {"\n\n bogus", 2, 2, "Unrecognized named-value: 'bogus'. Located at position 5 within expression: \n\n bogus"},
592+ {"github github", 0, 7, "Unexpected symbol: 'github'. Located at position 8 within expression: github github"},
593+ {" ) ", 0, 2, "Unexpected symbol: ')'. Located at position 3 within expression: ) "},
594+ {"contains('a',", 0, 12, "Unexpected end of expression: ','. Located at position 13 within expression: contains('a',"},
595+ {"foo()", 0, 0, "Unrecognized function: 'foo'. Located at position 1 within expression: foo()"},
596+ {"1 && foo()", 0, 5, "Unrecognized function: 'foo'. Located at position 6 within expression: 1 && foo()"},
597+ {"'a'\n 'b'\n zzz", 1, 1, "Unexpected symbol: ''b''. Located at position 6 within expression: 'a'\n 'b'\n zzz"},
598+ {"contains()", 0, 8, "Too few parameters supplied: '('. Located at position 9 within expression: contains()"},
599+ {"1 2", 0, 2, "Unexpected symbol: '2'. Located at position 3 within expression: 1 2"},
595600 }
596601 for _, tc := range tests {
597602 _, err := Parse(tc.src, testContextNames, nil)
@@ -618,7 +623,7 @@ func TestParseErrorTypes(t *testing.T) {
618623 {"contains(1)", ErrorTooFewParameters},
619624 {"contains(1,2,3)", ErrorTooManyParameters},
620625 {"case(true, 1, 2, 3)", ErrorEvenParameters},
621- {strings.Repeat("(", 60) + "1" + strings.Repeat(")", 60), ErrorExceededMaxDepth},
626+ {strings.Repeat("!", 60) + "1", ErrorExceededMaxDepth},
622627 {"1 = 1", ErrorUnexpectedSymbol},
623628 }
624629 for _, tc := range tests {
@@ -630,6 +635,10 @@ func TestParseErrorTypes(t *testing.T) {
630635 }
631636 }
632637
638+// TestLimits: the expectations were established with the runner's own parser.
639+// Depth is that of the finished tree (CheckMaxDepth): the root is 1, parentheses
640+// add nothing, a run of && or || is a single node, a call adds one level for
641+// its arguments, "a.b" and "a[b]" one level each, and "!" one.
633642 func TestLimits(t *testing.T) {
634643 deepParens := func(n int) string { return strings.Repeat("(", n) + "1" + strings.Repeat(")", n) }
635644 tests := []struct {
@@ -637,9 +646,7 @@ func TestLimits(t *testing.T) {
637646 src string
638647 ok bool
639648 }{
640- // Each "(" costs one level, plus one for the outermost expression.
641- {"49 parens", deepParens(49), true},
642- {"50 parens", deepParens(50), false},
649+ {"parentheses do not count", deepParens(500), true},
643650 {"49 nots", strings.Repeat("!", 49) + "true", true},
644651 {"50 nots", strings.Repeat("!", 50) + "true", false},
645652 {"49 calls", strings.Repeat("format(", 49) + "'x'" + strings.Repeat(")", 49), true},
@@ -647,7 +654,9 @@ func TestLimits(t *testing.T) {
647654 {"long dotted path", "github" + strings.Repeat(".event", 49), true},
648655 {"too long dotted path", "github" + strings.Repeat(".event", 50), false},
649656 {"long flat ||", "1" + strings.Repeat(" || 1", 3000), true},
650- {"long flat ==", "1" + strings.Repeat(" == 1", 3000), true},
657+ {"nested || is flat", "1" + strings.Repeat(" || (1", 60) + strings.Repeat(")", 60), true},
658+ {"49 ==", "1" + strings.Repeat(" == 1", 49), true},
659+ {"50 ==", "1" + strings.Repeat(" == 1", 50), false},
651660 }
652661 for _, tc := range tests {
653662 _, err := Parse(tc.src, testContextNames, nil)
@@ -655,18 +664,18 @@ func TestLimits(t *testing.T) {
655664 t.Errorf("%s: err = %v, want ok=%v", tc.name, err, tc.ok)
656665 }
657666 var pe *Error
658- if err != nil && errors.As(err, &pe) && pe.Type == ErrorExceededMaxDepth && pe.Message != "Exceeded max expression depth 50: '1'" && !strings.HasPrefix(pe.Message, "Exceeded max expression depth 50: '") {
667+ if err != nil && errors.As(err, &pe) && pe.Type == ErrorExceededMaxDepth && pe.Message != "Exceeded max expression depth 50" {
659668 t.Errorf("%s: message %q", tc.name, pe.Message)
660669 }
661670 }
662671
663- // Length: counted in UTF-16 code units, error is the bare token.
672+ // Length: counted in UTF-16 code units, no token in the message.
664673 long := "'" + strings.Repeat("a", MaxExpressionLength-2) + "'"
665674 if _, err := Parse(long, nil, nil); err != nil {
666675 t.Errorf("expression of exactly %d characters: %v", MaxExpressionLength, err)
667676 }
668677 _, err := Parse(long+" ", nil, nil)
669- if err == nil || err.Error() != "ErrorExceededMaxLength" {
678+ if err == nil || err.Error() != "Exceeded max expression length 21000" {
670679 t.Errorf("expression of %d characters: err = %v", MaxExpressionLength+1, err)
671680 }
672681 // 10501 emoji are 21002 UTF-16 units but only 10501 runes.
@@ -681,8 +690,8 @@ func TestLimits(t *testing.T) {
681690 }
682691
683692 func TestDepthMessage(t *testing.T) {
684- _, err := Parse(strings.Repeat("(", 60)+"1"+strings.Repeat(")", 60), nil, nil)
685- want := "Exceeded max expression depth 50: '('"
693+ _, err := Parse(strings.Repeat("!", 60)+"1", nil, nil)
694+ want := "Exceeded max expression depth 50"
686695 if err == nil || err.Error() != want {
687696 t.Errorf("got %v, want %q", err, want)
688697 }
@@ -698,19 +707,23 @@ func TestExtensionFunctions(t *testing.T) {
698707 for _, c := range []struct{ src, err string }{
699708 {"hashFiles('a')", ""},
700709 {"HASHFILES('a', 'b', 'c', 'd')", ""},
701- {"hashFiles()", "Too few parameters supplied: 'hashFiles'"},
710+ {"hashFiles()", "Too few parameters supplied: '('"},
702711 {"success()", ""},
703- {"success(1)", "Too many parameters supplied: 'success'"},
712+ {"success(1)", "Too many parameters supplied: '('"},
704713 {"bounded(1, 2)", ""},
705- {"bounded(1, 2, 3)", "Too many parameters supplied: 'bounded'"},
706- {"contains()", "Too few parameters supplied: 'contains'"},
714+ {"bounded(1, 2, 3)", "Too many parameters supplied: '('"},
715+ {"contains()", "Too few parameters supplied: '('"},
707716 {"nothere()", "Unrecognized function: 'nothere'"},
708717 } {
709718 _, err := Parse(c.src, nil, funcs)
719+ msg := ""
720+ if err != nil {
721+ msg, _, _ = strings.Cut(err.Error(), ". Located at position ")
722+ }
710723 switch {
711724 case c.err == "" && err != nil:
712725 t.Errorf("%s: %v", c.src, err)
713- case c.err != "" && (err == nil || err.Error() != c.err):
726+ case c.err != "" && (err == nil || msg != c.err):
714727 t.Errorf("%s: err = %v, want %q", c.src, err, c.err)
715728 }
716729 }
@@ -725,7 +738,7 @@ func TestExtensionFunctions(t *testing.T) {
725738 },
726739 "success": func(context.Context, []Value) (Value, error) { return Boolean(true), nil },
727740 "boom": func(context.Context, []Value) (Value, error) { return nil, errors.New("kaput") },
728- // An extra function replaces a built-in of the same name.
741+ // An extra function does not replace a built-in of the same name.
729742 "contains": func(context.Context, []Value) (Value, error) { return String("mine"), nil },
730743 },
731744 }
@@ -734,7 +747,7 @@ func TestExtensionFunctions(t *testing.T) {
734747 t.Fatal(err)
735748 }
736749 v, err := Evaluate(context.Background(), node, ctx)
737- if err != nil || CoerceString(v) != "mine" || calls != 1 {
750+ if err != nil || CoerceString(v) != "false" || calls != 1 {
738751 t.Errorf("got %v, %v, %d calls", v, err, calls)
739752 }
740753 node, err = Parse("boom()", nil, []FuncInfo{{Name: "boom"}})
@@ -770,7 +783,7 @@ func TestContextNames(t *testing.T) {
770783 t.Errorf("absent context: %v, %v", v, err)
771784 }
772785 // Names that were not declared to Parse are errors, even if present at evaluation.
773- if _, err := Parse("secrets.X", []string{"github"}, nil); err == nil || err.Error() != "Unrecognized named-value: 'secrets'" {
786+ if _, err := Parse("secrets.X", []string{"github"}, nil); err == nil || !strings.HasPrefix(err.Error(), "Unrecognized named-value: 'secrets'. Located at position 1 ") {
774787 t.Errorf("undeclared context: %v", err)
775788 }
776789 // A nil Value in a context behaves as null.
@@ -81,14 +81,14 @@ func TestEvalLiteralsAndOperators(t *testing.T) {
81 {"0x10", "Number:16"},81 {"0x10", "Number:16"},
82 {"0o17", "Number:15"},82 {"0o17", "Number:15"},
83 {"0xffffffff", "Number:-1"}, // read as a 32-bit integer, like the runner83 {"0xffffffff", "Number:-1"}, // read as a 32-bit integer, like the runner
84- {"0b101", "PARSE Unexpected symbol: '0b101'. Located at position 1 within expression: 0b101"},84+ {"0b101", "PARSE Unexpected symbol: '0b101'"},
85- {"0X10", "PARSE Unexpected symbol: '0X10'. Located at position 1 within expression: 0X10"},85+ {"0X10", "PARSE Unexpected symbol: '0X10'"},
86 {"'0b11' == 3", "Boolean:false"},86 {"'0b11' == 3", "Boolean:false"},
87 {"'0x1F' == 31", "Boolean:true"},87 {"'0x1F' == 31", "Boolean:true"},
88 {"'infinity' == Infinity", "Boolean:true"},88 {"'infinity' == Infinity", "Boolean:true"},
89 {"'\u0085 1 \u0085' == 1", "Boolean:true"},89 {"'\u0085 1 \u0085' == 1", "Boolean:true"},
90 {"'\ufeff1' == 1", "Boolean:false"},90 {"'\ufeff1' == 1", "Boolean:false"},
91- {"-0x10", "PARSE Unexpected symbol: '-0x10'. Located at position 1 within expression: -0x10"},91+ {"-0x10", "PARSE Unexpected symbol: '-0x10'"},
92 {"NaN", "Number:NaN"},92 {"NaN", "Number:NaN"},
93 {"Infinity", "Number:Infinity"},93 {"Infinity", "Number:Infinity"},
94 {"-Infinity", "Number:-Infinity"},94 {"-Infinity", "Number:-Infinity"},
@@ -290,7 +290,7 @@ func TestEvalAccess(t *testing.T) {
290 {"steps[10].outputs.r", "String:ten"},290 {"steps[10].outputs.r", "String:ten"},
291 {"steps['10'].outputs.r", "String:ten"},291 {"steps['10'].outputs.r", "String:ten"},
292 // After an identifier, ".10" is not a number: the lexer reads a DOT and then "10.outputs.r" as one number-like word.292 // After an identifier, ".10" is not a number: the lexer reads a DOT and then "10.outputs.r" as one number-like word.
293- {"steps.10.outputs.r", "PARSE Unexpected symbol: '10.outputs.r'. Located at position 7 within expression: steps.10.outputs.r"},293+ {"steps.10.outputs.r", "PARSE Unexpected symbol: '10.outputs.r'"},
294 {"steps[2].outputs.r", "String:two"},294 {"steps[2].outputs.r", "String:two"},
295 {"matrix.obj[0]", "Null:"},295 {"matrix.obj[0]", "Null:"},
296 {"matrix.node['0']", "Number:14"},296 {"matrix.node['0']", "Number:14"},
@@ -366,9 +366,9 @@ func TestEvalBuiltinFunctions(t *testing.T) {
366 {"contains('Dž', 'dž')", "Boolean:true"},366 {"contains('Dž', 'dž')", "Boolean:true"},
367 {"contains('A\u03a3', 'a\u03c2')", "Boolean:true"},367 {"contains('A\u03a3', 'a\u03c2')", "Boolean:true"},
368 {"contains('K', 'k')", "Boolean:true"}, // KELVIN SIGN lower-cases to k368 {"contains('K', 'k')", "Boolean:true"}, // KELVIN SIGN lower-cases to k
369- {"contains()", "PARSE Too few parameters supplied: 'contains'"},369+ {"contains()", "PARSE Too few parameters supplied: '('"},
370- {"contains(1)", "PARSE Too few parameters supplied: 'contains'"},370+ {"contains(1)", "PARSE Too few parameters supplied: '('"},
371- {"contains(1, 2, 3)", "PARSE Too many parameters supplied: 'contains'"},371+ {"contains(1, 2, 3)", "PARSE Too many parameters supplied: '('"},
372 372
373 // startsWith, endsWith373 // startsWith, endsWith
374 {"startsWith('abc', 'AB')", "Boolean:true"},374 {"startsWith('abc', 'AB')", "Boolean:true"},
@@ -383,7 +383,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
383 {"endsWith('abc', 'abcd')", "Boolean:false"},383 {"endsWith('abc', 'abcd')", "Boolean:false"},
384 {"endsWith(1.50, '5')", "Boolean:true"},384 {"endsWith(1.50, '5')", "Boolean:true"},
385 {"endsWith(1.50, 5)", "Boolean:true"},385 {"endsWith(1.50, 5)", "Boolean:true"},
386- {"endsWith('abc')", "PARSE Too few parameters supplied: 'endsWith'"},386+ {"endsWith('abc')", "PARSE Too few parameters supplied: '('"},
387 {"STARTSWITH('abc', 'a')", "Boolean:true"},387 {"STARTSWITH('abc', 'a')", "Boolean:true"},
388 388
389 // join389 // join
@@ -400,8 +400,8 @@ func TestEvalBuiltinFunctions(t *testing.T) {
400 {"join(fromJSON('[1,2,3]'), null)", "String:123"},400 {"join(fromJSON('[1,2,3]'), null)", "String:123"},
401 {"join(fromJSON('[1,2,3]'), github)", "String:1,2,3"}, // a non-primitive separator is ignored401 {"join(fromJSON('[1,2,3]'), github)", "String:1,2,3"}, // a non-primitive separator is ignored
402 {"join(fromJSON('[null,true,\"a\",1.5,[1],{}]'))", "String:,true,a,1.5,Array,Object"},402 {"join(fromJSON('[null,true,\"a\",1.5,[1],{}]'))", "String:,true,a,1.5,Array,Object"},
403- {"join()", "PARSE Too few parameters supplied: 'join'"},403+ {"join()", "PARSE Too few parameters supplied: '('"},
404- {"join(1, 2, 3)", "PARSE Too many parameters supplied: 'join'"},404+ {"join(1, 2, 3)", "PARSE Too many parameters supplied: '('"},
405 405
406 // toJSON406 // toJSON
407 {"toJSON(null)", "String:null"},407 {"toJSON(null)", "String:null"},
@@ -417,8 +417,8 @@ func TestEvalBuiltinFunctions(t *testing.T) {
417 {"toJSON(fromJSON('[]'))", "String:[]"},417 {"toJSON(fromJSON('[]'))", "String:[]"},
418 {"toJSON(fromJSON('{}'))", "String:{}"},418 {"toJSON(fromJSON('{}'))", "String:{}"},
419 {"toJSON(fromJSON('[[],{}]'))", "String:[\n [],\n {}\n]"},419 {"toJSON(fromJSON('[[],{}]'))", "String:[\n [],\n {}\n]"},
420- {"toJSON(1, 2)", "PARSE Too many parameters supplied: 'toJSON'"},420+ {"toJSON(1, 2)", "PARSE Too many parameters supplied: '('"},
421- {"toJSON()", "PARSE Too few parameters supplied: 'toJSON'"},421+ {"toJSON()", "PARSE Too few parameters supplied: '('"},
422 {"fromJSON(toJSON(github.event.commits[0])).author.name", "String:Ann"},422 {"fromJSON(toJSON(github.event.commits[0])).author.name", "String:Ann"},
423 423
424 // fromJSON424 // fromJSON
@@ -442,7 +442,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
442 {"fromJSON(1)", "Number:1"},442 {"fromJSON(1)", "Number:1"},
443 {"fromJSON(true)", "Boolean:true"},443 {"fromJSON(true)", "Boolean:true"},
444 {"fromJSON(matrix.node)", "ERR Error parsing JSON when evaluating fromJson"}, // the string "Array"444 {"fromJSON(matrix.node)", "ERR Error parsing JSON when evaluating fromJson"}, // the string "Array"
445- {"fromJSON(1, 2)", "PARSE Too many parameters supplied: 'fromJSON'"},445+ {"fromJSON(1, 2)", "PARSE Too many parameters supplied: '('"},
446 446
447 // format447 // format
448 {"format('a')", "String:a"},448 {"format('a')", "String:a"},
@@ -493,7 +493,7 @@ func TestEvalBuiltinFunctions(t *testing.T) {
493 {"format('{0}}', 1)", "ERR The following format string is invalid: {0}}"},493 {"format('{0}}', 1)", "ERR The following format string is invalid: {0}}"},
494 // The index is a byte: a longer number makes the string invalid.494 // The index is a byte: a longer number makes the string invalid.
495 {"format('{12345678901234567890123}', 1)", "ERR The following format string is invalid: {12345678901234567890123}"},495 {"format('{12345678901234567890123}', 1)", "ERR The following format string is invalid: {12345678901234567890123}"},
496- {"format()", "PARSE Too few parameters supplied: 'format'"},496+ {"format()", "PARSE Too few parameters supplied: '('"},
497 497
498 // case498 // case
499 {"case(true, 'a', 'b')", "String:a"},499 {"case(true, 'a', 'b')", "String:a"},
@@ -506,9 +506,9 @@ func TestEvalBuiltinFunctions(t *testing.T) {
506 {"case(1, 2, 3)", "ERR case predicate must evaluate to a boolean value"},506 {"case(1, 2, 3)", "ERR case predicate must evaluate to a boolean value"},
507 {"case('true', 2, 3)", "ERR case predicate must evaluate to a boolean value"},507 {"case('true', 2, 3)", "ERR case predicate must evaluate to a boolean value"},
508 {"case(null, 2, 3)", "ERR case predicate must evaluate to a boolean value"},508 {"case(null, 2, 3)", "ERR case predicate must evaluate to a boolean value"},
509- {"case(true, 1, 2, 3)", "PARSE Even number of parameters supplied, requires an odd number of parameters: 'case'"},509+ {"case(true, 1, 2, 3)", "PARSE Even number of parameters supplied, requires an odd number of parameters: '('"},
510- {"case(true, 1)", "PARSE Too few parameters supplied: 'case'"},510+ {"case(true, 1)", "PARSE Too few parameters supplied: '('"},
511- {"case()", "PARSE Too few parameters supplied: 'case'"},511+ {"case()", "PARSE Too few parameters supplied: '('"},
512 // Predicates after the first true one are not checked...512 // Predicates after the first true one are not checked...
513 {"case(true, 1, 'bad', 2, 3)", "Number:1"},513 {"case(true, 1, 'bad', 2, 3)", "Number:1"},
514 // ...but earlier ones are.514 // ...but earlier ones are.
@@ -523,7 +523,7 @@ func TestEvalErrorMessages(t *testing.T) {
523 {"", "PARSE Unexpected end of expression: 'x'"}, // replaced below: the empty expression is not an error523 {"", "PARSE Unexpected end of expression: 'x'"}, // replaced below: the empty expression is not an error
524 }[:0])524 }[:0])
525 cases := []evalCase{525 cases := []evalCase{
526- {"1 +", "PARSE Unexpected symbol: '+'. Located at position 3 within expression: 1 +"},526+ {"1 +", "PARSE Unexpected symbol: '+'"},
527 {"a b", "PARSE Unrecognized named-value: 'a'"},527 {"a b", "PARSE Unrecognized named-value: 'a'"},
528 {"github github", "PARSE Unexpected symbol: 'github'"},528 {"github github", "PARSE Unexpected symbol: 'github'"},
529 {"github.event_name ==", "PARSE Unexpected end of expression: '=='"},529 {"github.event_name ==", "PARSE Unexpected end of expression: '=='"},
@@ -531,9 +531,9 @@ func TestEvalErrorMessages(t *testing.T) {
531 {"(github.event_name", "PARSE Unexpected end of expression: 'event_name'"},531 {"(github.event_name", "PARSE Unexpected end of expression: 'event_name'"},
532 {"github.event_name)", "PARSE Unexpected symbol: ')'"},532 {"github.event_name)", "PARSE Unexpected symbol: ')'"},
533 {"contains('a',", "PARSE Unexpected end of expression: ','"},533 {"contains('a',", "PARSE Unexpected end of expression: ','"},
534- {"contains('a'", "PARSE Unexpected symbol: 'EOF'"},534+ {"contains('a'", "PARSE Unexpected end of expression: ''a''"},
535- {"contains('a' 'b')", "PARSE Unexpected symbol: 'b'"},535+ {"contains('a' 'b')", "PARSE Unexpected symbol: ''b''"},
536- {"foo(", "PARSE Unexpected end of expression: '('"},536+ {"foo(", "PARSE Unrecognized function: 'foo'"},
537 {"foo()", "PARSE Unrecognized function: 'foo'"},537 {"foo()", "PARSE Unrecognized function: 'foo'"},
538 {"foo", "PARSE Unrecognized named-value: 'foo'"},538 {"foo", "PARSE Unrecognized named-value: 'foo'"},
539 {"unknown(1)", "PARSE Unrecognized function: 'unknown'"},539 {"unknown(1)", "PARSE Unrecognized function: 'unknown'"},
@@ -545,22 +545,22 @@ func TestEvalErrorMessages(t *testing.T) {
545 {"1 !", "PARSE Unexpected symbol: '!'"},545 {"1 !", "PARSE Unexpected symbol: '!'"},
546 {"[1]", "PARSE Unexpected symbol: '['"},546 {"[1]", "PARSE Unexpected symbol: '['"},
547 {"github.[0]", "PARSE Unexpected symbol: '['"},547 {"github.[0]", "PARSE Unexpected symbol: '['"},
548- {"github.", "PARSE Unexpected symbol: 'EOF'"},548+ {"github.", "PARSE Unexpected end of expression: '.'"},
549 {"github[", "PARSE Unexpected end of expression: '['"},549 {"github[", "PARSE Unexpected end of expression: '['"},
550 {"github[]", "PARSE Unexpected symbol: ']'"},550 {"github[]", "PARSE Unexpected symbol: ']'"},
551- {"github[*", "PARSE Unexpected symbol: 'EOF'"},551+ {"github[*", "PARSE Unexpected end of expression: '*'"},
552 {"github.true", "Null:"}, // keywords are plain identifiers after a dot552 {"github.true", "Null:"}, // keywords are plain identifiers after a dot
553 {"env.true", "Null:"},553 {"env.true", "Null:"},
554- {"1 = 1", "PARSE Unexpected symbol: '='. Located at position 3 within expression: 1 = 1"},554+ {"1 = 1", "PARSE Unexpected symbol: '='"},
555- {"1 & 1", "PARSE Unexpected symbol: '&'. Located at position 3 within expression: 1 & 1"},555+ {"1 & 1", "PARSE Unexpected symbol: '&'"},
556- {"1 | 1", "PARSE Unexpected symbol: '|'. Located at position 3 within expression: 1 | 1"},556+ {"1 | 1", "PARSE Unexpected symbol: '|'"},
557- {"1 + 1", "PARSE Unexpected symbol: '+'. Located at position 3 within expression: 1 + 1"},557+ {"1 + 1", "PARSE Unexpected symbol: '+'"},
558 {"a-b", "PARSE Unrecognized named-value: 'a-b'"},558 {"a-b", "PARSE Unrecognized named-value: 'a-b'"},
559- {"1a", "PARSE Unexpected symbol: '1a'. Located at position 1 within expression: 1a"},559+ {"1a", "PARSE Unexpected symbol: '1a'"},
560- {"'unterminated", "PARSE Unexpected symbol: ''unterminated'. Located at position 1 within expression: 'unterminated"},560+ {"'unterminated", "PARSE Unexpected symbol: ''unterminated'"},
561- {"@", "PARSE Unexpected symbol: '@'. Located at position 1 within expression: @"},561+ {"@", "PARSE Unexpected symbol: '@'"},
562- {"${{ 1 }}", "PARSE Unexpected symbol: '${{'. Located at position 1 within expression: ${{ 1 }}"},562+ {"${{ 1 }}", "PARSE Unexpected symbol: '${{'"},
563- {"1 \u00a0 2", "PARSE Unexpected symbol: '\u00a0'. Located at position 3 within expression: 1 \u00a0 2"},563+ {"1 \u00a0 2", "PARSE Unexpected symbol: '2'"}, // U+00A0 is white space for .NET
564 }564 }
565 runEvalCases(t, cases)565 runEvalCases(t, cases)
566 }566 }
@@ -577,6 +577,9 @@ func TestEmptyExpression(t *testing.T) {
577 }577 }
578 }578 }
579 579
580+// TestParseErrorPositions: the runner (ParseException) reports the absolute
581+// index of the token in the expression, 1-based, and the whole expression;
582+// Error.Pos is the same place as a line and a column.
580 func TestParseErrorPositions(t *testing.T) {583 func TestParseErrorPositions(t *testing.T) {
581 tests := []struct {584 tests := []struct {
582 src string585 src string
@@ -584,14 +587,16 @@ func TestParseErrorPositions(t *testing.T) {
584 msg string587 msg string
585 }{588 }{
586 {"1 ==\n 2 ===\n 3", 1, 5, "Unexpected symbol: '='. Located at position 11 within expression: 1 ==\n 2 ===\n 3"},589 {"1 ==\n 2 ===\n 3", 1, 5, "Unexpected symbol: '='. Located at position 11 within expression: 1 ==\n 2 ===\n 3"},
587- {"true &&\n false ||\n nope", 2, 2, "Unrecognized named-value: 'nope'"},590+ {"true &&\n false ||\n nope", 2, 2, "Unrecognized named-value: 'nope'. Located at position 22 within expression: true &&\n false ||\n nope"},
588- {"\n\n bogus", 2, 2, "Unrecognized named-value: 'bogus'"},591+ {"\n\n bogus", 2, 2, "Unrecognized named-value: 'bogus'. Located at position 5 within expression: \n\n bogus"},
589- {"github github", 0, 7, "Unexpected symbol: 'github'"},592+ {"github github", 0, 7, "Unexpected symbol: 'github'. Located at position 8 within expression: github github"},
590- {" ) ", 0, 2, "Unexpected symbol: ')'"},593+ {" ) ", 0, 2, "Unexpected symbol: ')'. Located at position 3 within expression: ) "},
591- {"contains('a',", 0, 12, "Unexpected end of expression: ','"},594+ {"contains('a',", 0, 12, "Unexpected end of expression: ','. Located at position 13 within expression: contains('a',"},
592- {"foo()", 0, 0, "Unrecognized function: 'foo'"},595+ {"foo()", 0, 0, "Unrecognized function: 'foo'. Located at position 1 within expression: foo()"},
593- {"1 && foo()", 0, 5, "Unrecognized function: 'foo'"},596+ {"1 && foo()", 0, 5, "Unrecognized function: 'foo'. Located at position 6 within expression: 1 && foo()"},
594- {"'a'\n 'b'\n zzz", 1, 1, "Unexpected symbol: 'b'"},597+ {"'a'\n 'b'\n zzz", 1, 1, "Unexpected symbol: ''b''. Located at position 6 within expression: 'a'\n 'b'\n zzz"},
598+ {"contains()", 0, 8, "Too few parameters supplied: '('. Located at position 9 within expression: contains()"},
599+ {"1 2", 0, 2, "Unexpected symbol: '2'. Located at position 3 within expression: 1 2"},
595 }600 }
596 for _, tc := range tests {601 for _, tc := range tests {
597 _, err := Parse(tc.src, testContextNames, nil)602 _, err := Parse(tc.src, testContextNames, nil)
@@ -618,7 +623,7 @@ func TestParseErrorTypes(t *testing.T) {
618 {"contains(1)", ErrorTooFewParameters},623 {"contains(1)", ErrorTooFewParameters},
619 {"contains(1,2,3)", ErrorTooManyParameters},624 {"contains(1,2,3)", ErrorTooManyParameters},
620 {"case(true, 1, 2, 3)", ErrorEvenParameters},625 {"case(true, 1, 2, 3)", ErrorEvenParameters},
621- {strings.Repeat("(", 60) + "1" + strings.Repeat(")", 60), ErrorExceededMaxDepth},626+ {strings.Repeat("!", 60) + "1", ErrorExceededMaxDepth},
622 {"1 = 1", ErrorUnexpectedSymbol},627 {"1 = 1", ErrorUnexpectedSymbol},
623 }628 }
624 for _, tc := range tests {629 for _, tc := range tests {
@@ -630,6 +635,10 @@ func TestParseErrorTypes(t *testing.T) {
630 }635 }
631 }636 }
632 637
638+// TestLimits: the expectations were established with the runner's own parser.
639+// Depth is that of the finished tree (CheckMaxDepth): the root is 1, parentheses
640+// add nothing, a run of && or || is a single node, a call adds one level for
641+// its arguments, "a.b" and "a[b]" one level each, and "!" one.
633 func TestLimits(t *testing.T) {642 func TestLimits(t *testing.T) {
634 deepParens := func(n int) string { return strings.Repeat("(", n) + "1" + strings.Repeat(")", n) }643 deepParens := func(n int) string { return strings.Repeat("(", n) + "1" + strings.Repeat(")", n) }
635 tests := []struct {644 tests := []struct {
@@ -637,9 +646,7 @@ func TestLimits(t *testing.T) {
637 src string646 src string
638 ok bool647 ok bool
639 }{648 }{
640- // Each "(" costs one level, plus one for the outermost expression.649+ {"parentheses do not count", deepParens(500), true},
641- {"49 parens", deepParens(49), true},
642- {"50 parens", deepParens(50), false},
643 {"49 nots", strings.Repeat("!", 49) + "true", true},650 {"49 nots", strings.Repeat("!", 49) + "true", true},
644 {"50 nots", strings.Repeat("!", 50) + "true", false},651 {"50 nots", strings.Repeat("!", 50) + "true", false},
645 {"49 calls", strings.Repeat("format(", 49) + "'x'" + strings.Repeat(")", 49), true},652 {"49 calls", strings.Repeat("format(", 49) + "'x'" + strings.Repeat(")", 49), true},
@@ -647,7 +654,9 @@ func TestLimits(t *testing.T) {
647 {"long dotted path", "github" + strings.Repeat(".event", 49), true},654 {"long dotted path", "github" + strings.Repeat(".event", 49), true},
648 {"too long dotted path", "github" + strings.Repeat(".event", 50), false},655 {"too long dotted path", "github" + strings.Repeat(".event", 50), false},
649 {"long flat ||", "1" + strings.Repeat(" || 1", 3000), true},656 {"long flat ||", "1" + strings.Repeat(" || 1", 3000), true},
650- {"long flat ==", "1" + strings.Repeat(" == 1", 3000), true},657+ {"nested || is flat", "1" + strings.Repeat(" || (1", 60) + strings.Repeat(")", 60), true},
658+ {"49 ==", "1" + strings.Repeat(" == 1", 49), true},
659+ {"50 ==", "1" + strings.Repeat(" == 1", 50), false},
651 }660 }
652 for _, tc := range tests {661 for _, tc := range tests {
653 _, err := Parse(tc.src, testContextNames, nil)662 _, err := Parse(tc.src, testContextNames, nil)
@@ -655,18 +664,18 @@ func TestLimits(t *testing.T) {
655 t.Errorf("%s: err = %v, want ok=%v", tc.name, err, tc.ok)664 t.Errorf("%s: err = %v, want ok=%v", tc.name, err, tc.ok)
656 }665 }
657 var pe *Error666 var pe *Error
658- if err != nil && errors.As(err, &pe) && pe.Type == ErrorExceededMaxDepth && pe.Message != "Exceeded max expression depth 50: '1'" && !strings.HasPrefix(pe.Message, "Exceeded max expression depth 50: '") {667+ if err != nil && errors.As(err, &pe) && pe.Type == ErrorExceededMaxDepth && pe.Message != "Exceeded max expression depth 50" {
659 t.Errorf("%s: message %q", tc.name, pe.Message)668 t.Errorf("%s: message %q", tc.name, pe.Message)
660 }669 }
661 }670 }
662 671
663- // Length: counted in UTF-16 code units, error is the bare token.672+ // Length: counted in UTF-16 code units, no token in the message.
664 long := "'" + strings.Repeat("a", MaxExpressionLength-2) + "'"673 long := "'" + strings.Repeat("a", MaxExpressionLength-2) + "'"
665 if _, err := Parse(long, nil, nil); err != nil {674 if _, err := Parse(long, nil, nil); err != nil {
666 t.Errorf("expression of exactly %d characters: %v", MaxExpressionLength, err)675 t.Errorf("expression of exactly %d characters: %v", MaxExpressionLength, err)
667 }676 }
668 _, err := Parse(long+" ", nil, nil)677 _, err := Parse(long+" ", nil, nil)
669- if err == nil || err.Error() != "ErrorExceededMaxLength" {678+ if err == nil || err.Error() != "Exceeded max expression length 21000" {
670 t.Errorf("expression of %d characters: err = %v", MaxExpressionLength+1, err)679 t.Errorf("expression of %d characters: err = %v", MaxExpressionLength+1, err)
671 }680 }
672 // 10501 emoji are 21002 UTF-16 units but only 10501 runes.681 // 10501 emoji are 21002 UTF-16 units but only 10501 runes.
@@ -681,8 +690,8 @@ func TestLimits(t *testing.T) {
681 }690 }
682 691
683 func TestDepthMessage(t *testing.T) {692 func TestDepthMessage(t *testing.T) {
684- _, err := Parse(strings.Repeat("(", 60)+"1"+strings.Repeat(")", 60), nil, nil)693+ _, err := Parse(strings.Repeat("!", 60)+"1", nil, nil)
685- want := "Exceeded max expression depth 50: '('"694+ want := "Exceeded max expression depth 50"
686 if err == nil || err.Error() != want {695 if err == nil || err.Error() != want {
687 t.Errorf("got %v, want %q", err, want)696 t.Errorf("got %v, want %q", err, want)
688 }697 }
@@ -698,19 +707,23 @@ func TestExtensionFunctions(t *testing.T) {
698 for _, c := range []struct{ src, err string }{707 for _, c := range []struct{ src, err string }{
699 {"hashFiles('a')", ""},708 {"hashFiles('a')", ""},
700 {"HASHFILES('a', 'b', 'c', 'd')", ""},709 {"HASHFILES('a', 'b', 'c', 'd')", ""},
701- {"hashFiles()", "Too few parameters supplied: 'hashFiles'"},710+ {"hashFiles()", "Too few parameters supplied: '('"},
702 {"success()", ""},711 {"success()", ""},
703- {"success(1)", "Too many parameters supplied: 'success'"},712+ {"success(1)", "Too many parameters supplied: '('"},
704 {"bounded(1, 2)", ""},713 {"bounded(1, 2)", ""},
705- {"bounded(1, 2, 3)", "Too many parameters supplied: 'bounded'"},714+ {"bounded(1, 2, 3)", "Too many parameters supplied: '('"},
706- {"contains()", "Too few parameters supplied: 'contains'"},715+ {"contains()", "Too few parameters supplied: '('"},
707 {"nothere()", "Unrecognized function: 'nothere'"},716 {"nothere()", "Unrecognized function: 'nothere'"},
708 } {717 } {
709 _, err := Parse(c.src, nil, funcs)718 _, err := Parse(c.src, nil, funcs)
719+ msg := ""
720+ if err != nil {
721+ msg, _, _ = strings.Cut(err.Error(), ". Located at position ")
722+ }
710 switch {723 switch {
711 case c.err == "" && err != nil:724 case c.err == "" && err != nil:
712 t.Errorf("%s: %v", c.src, err)725 t.Errorf("%s: %v", c.src, err)
713- case c.err != "" && (err == nil || err.Error() != c.err):726+ case c.err != "" && (err == nil || msg != c.err):
714 t.Errorf("%s: err = %v, want %q", c.src, err, c.err)727 t.Errorf("%s: err = %v, want %q", c.src, err, c.err)
715 }728 }
716 }729 }
@@ -725,7 +738,7 @@ func TestExtensionFunctions(t *testing.T) {
725 },738 },
726 "success": func(context.Context, []Value) (Value, error) { return Boolean(true), nil },739 "success": func(context.Context, []Value) (Value, error) { return Boolean(true), nil },
727 "boom": func(context.Context, []Value) (Value, error) { return nil, errors.New("kaput") },740 "boom": func(context.Context, []Value) (Value, error) { return nil, errors.New("kaput") },
728- // An extra function replaces a built-in of the same name.741+ // An extra function does not replace a built-in of the same name.
729 "contains": func(context.Context, []Value) (Value, error) { return String("mine"), nil },742 "contains": func(context.Context, []Value) (Value, error) { return String("mine"), nil },
730 },743 },
731 }744 }
@@ -734,7 +747,7 @@ func TestExtensionFunctions(t *testing.T) {
734 t.Fatal(err)747 t.Fatal(err)
735 }748 }
736 v, err := Evaluate(context.Background(), node, ctx)749 v, err := Evaluate(context.Background(), node, ctx)
737- if err != nil || CoerceString(v) != "mine" || calls != 1 {750+ if err != nil || CoerceString(v) != "false" || calls != 1 {
738 t.Errorf("got %v, %v, %d calls", v, err, calls)751 t.Errorf("got %v, %v, %d calls", v, err, calls)
739 }752 }
740 node, err = Parse("boom()", nil, []FuncInfo{{Name: "boom"}})753 node, err = Parse("boom()", nil, []FuncInfo{{Name: "boom"}})
@@ -770,7 +783,7 @@ func TestContextNames(t *testing.T) {
770 t.Errorf("absent context: %v, %v", v, err)783 t.Errorf("absent context: %v, %v", v, err)
771 }784 }
772 // Names that were not declared to Parse are errors, even if present at evaluation.785 // Names that were not declared to Parse are errors, even if present at evaluation.
773- if _, err := Parse("secrets.X", []string{"github"}, nil); err == nil || err.Error() != "Unrecognized named-value: 'secrets'" {786+ if _, err := Parse("secrets.X", []string{"github"}, nil); err == nil || !strings.HasPrefix(err.Error(), "Unrecognized named-value: 'secrets'. Located at position 1 ") {
774 t.Errorf("undeclared context: %v", err)787 t.Errorf("undeclared context: %v", err)
775 }788 }
776 // A nil Value in a context behaves as null.789 // A nil Value in a context behaves as null.
M expr/example_test.go +4 -4
@@ -27,10 +27,10 @@ func ExampleParse() {
2727 }
2828 // Output:
2929 // true <nil>
30- // secrets.TOKEN -> Unrecognized named-value: 'secrets' (line 0, column 0)
31- // contains('a') -> Too few parameters supplied: 'contains' (line 0, column 0)
32- // github.ref == -> Unexpected end of expression: '==' (line 0, column 11)
33- // nope() -> Unrecognized function: 'nope' (line 0, column 0)
30+ // secrets.TOKEN -> Unrecognized named-value: 'secrets'. Located at position 1 within expression: secrets.TOKEN (line 0, column 0)
31+ // contains('a') -> Too few parameters supplied: '('. Located at position 9 within expression: contains('a') (line 0, column 8)
32+ // github.ref == -> Unexpected end of expression: '=='. Located at position 12 within expression: github.ref == (line 0, column 11)
33+ // nope() -> Unrecognized function: 'nope'. Located at position 1 within expression: nope() (line 0, column 0)
3434 }
3535
3636 func ExampleEvaluate() {
@@ -27,10 +27,10 @@ func ExampleParse() {
27 }27 }
28 // Output:28 // Output:
29 // true <nil>29 // true <nil>
30- // secrets.TOKEN -> Unrecognized named-value: 'secrets' (line 0, column 0)30+ // secrets.TOKEN -> Unrecognized named-value: 'secrets'. Located at position 1 within expression: secrets.TOKEN (line 0, column 0)
31- // contains('a') -> Too few parameters supplied: 'contains' (line 0, column 0)31+ // contains('a') -> Too few parameters supplied: '('. Located at position 9 within expression: contains('a') (line 0, column 8)
32- // github.ref == -> Unexpected end of expression: '==' (line 0, column 11)32+ // github.ref == -> Unexpected end of expression: '=='. Located at position 12 within expression: github.ref == (line 0, column 11)
33- // nope() -> Unrecognized function: 'nope' (line 0, column 0)33+ // nope() -> Unrecognized function: 'nope'. Located at position 1 within expression: nope() (line 0, column 0)
34 }34 }
35 35
36 func ExampleEvaluate() {36 func ExampleEvaluate() {
M expr/fuzz_test.go +1 -1
@@ -81,7 +81,7 @@ func FuzzParse(f *testing.F) {
8181 }
8282 return
8383 }
84- if node == nil && strings.TrimFunc(s, func(r rune) bool { return r == ' ' || r == '\t' || r == '\r' || r == '\n' }) != "" {
84+ if node == nil && strings.TrimFunc(s, isNetSpace) != "" {
8585 t.Fatalf("empty tree for non-blank input %q", s)
8686 }
8787 if d := nodeDepth(node); d > 2*MaxParserDepth+2 {
@@ -81,7 +81,7 @@ func FuzzParse(f *testing.F) {
81 }81 }
82 return82 return
83 }83 }
84- if node == nil && strings.TrimFunc(s, func(r rune) bool { return r == ' ' || r == '\t' || r == '\r' || r == '\n' }) != "" {84+ if node == nil && strings.TrimFunc(s, isNetSpace) != "" {
85 t.Fatalf("empty tree for non-blank input %q", s)85 t.Fatalf("empty tree for non-blank input %q", s)
86 }86 }
87 if d := nodeDepth(node); d > 2*MaxParserDepth+2 {87 if d := nodeDepth(node); d > 2*MaxParserDepth+2 {
M expr/json.go +21 -0
@@ -6,6 +6,7 @@ import (
66 "errors"
77 "fmt"
88 "math"
9+ "math/big"
910 "strconv"
1011 "strings"
1112 "unicode"
@@ -157,10 +158,30 @@ func (p *jsonParser) number() (Value, error) {
157158 p.i = start
158159 return nil, fmt.Errorf("input string %q is not a valid number", tok)
159160 }
161+ if isBigInteger(body) {
162+ // Json.NET reads an integer that does not fit in 64 bits as a
163+ // BigInteger, and (Double)BigInteger truncates toward zero to 53 bits
164+ // instead of rounding to nearest.
165+ if _, err := strconv.ParseInt(tok, 10, 64); err != nil {
166+ n, _ := new(big.Int).SetString(tok, 10)
167+ f, _ := new(big.Float).SetPrec(53).SetMode(big.ToZero).SetInt(n).Float64()
168+ return Number(f), nil
169+ }
170+ }
160171 f, _ := strconv.ParseFloat(tok, 64) // range errors give ±Inf
161172 return Number(f), nil
162173 }
163174
175+// isBigInteger reports whether s is only digits.
176+func isBigInteger(s string) bool {
177+ for i := 0; i < len(s); i++ {
178+ if s[i] < '0' || s[i] > '9' {
179+ return false
180+ }
181+ }
182+ return s != ""
183+}
184+
164185 // str reads a string literal; p.i is on the opening quote.
165186 func (p *jsonParser) str() (string, error) {
166187 quote := p.s[p.i]
@@ -6,6 +6,7 @@ import (
6 "errors"6 "errors"
7 "fmt"7 "fmt"
8 "math"8 "math"
9+ "math/big"
9 "strconv"10 "strconv"
10 "strings"11 "strings"
11 "unicode"12 "unicode"
@@ -157,10 +158,30 @@ func (p *jsonParser) number() (Value, error) {
157 p.i = start158 p.i = start
158 return nil, fmt.Errorf("input string %q is not a valid number", tok)159 return nil, fmt.Errorf("input string %q is not a valid number", tok)
159 }160 }
161+ if isBigInteger(body) {
162+ // Json.NET reads an integer that does not fit in 64 bits as a
163+ // BigInteger, and (Double)BigInteger truncates toward zero to 53 bits
164+ // instead of rounding to nearest.
165+ if _, err := strconv.ParseInt(tok, 10, 64); err != nil {
166+ n, _ := new(big.Int).SetString(tok, 10)
167+ f, _ := new(big.Float).SetPrec(53).SetMode(big.ToZero).SetInt(n).Float64()
168+ return Number(f), nil
169+ }
170+ }
160 f, _ := strconv.ParseFloat(tok, 64) // range errors give ±Inf171 f, _ := strconv.ParseFloat(tok, 64) // range errors give ±Inf
161 return Number(f), nil172 return Number(f), nil
162 }173 }
163 174
175+// isBigInteger reports whether s is only digits.
176+func isBigInteger(s string) bool {
177+ for i := 0; i < len(s); i++ {
178+ if s[i] < '0' || s[i] > '9' {
179+ return false
180+ }
181+ }
182+ return s != ""
183+}
184+
164 // str reads a string literal; p.i is on the opening quote.185 // str reads a string literal; p.i is on the opening quote.
165 func (p *jsonParser) str() (string, error) {186 func (p *jsonParser) str() (string, error) {
166 quote := p.s[p.i]187 quote := p.s[p.i]
M expr/json_test.go +28 -0
@@ -100,6 +100,34 @@ func TestFromJSONNumbers(t *testing.T) {
100100 }
101101 }
102102
103+// TestFromJSONBigIntegers: Json.NET gives integers beyond 64 bits as BigInteger
104+// and (Double)BigInteger truncates; established with the runner's code
105+// (fromJSON('N') == N, where the literal N rounds to nearest).
106+func TestFromJSONBigIntegers(t *testing.T) {
107+ for _, tc := range []struct {
108+ src string
109+ want float64
110+ }{
111+ {"9223372036854775807", 9223372036854775807}, // fits a long: nearest
112+ {"-9223372036854775808", -9223372036854775808}, // exact
113+ {"12345678901234567890", 12345678901234567890}, // 64 bits, above long: truncated to the same double here
114+ {"18446744073709551615", 18446744073709549568}, // 2^64-1 truncates to 2^64-2048, not 2^64
115+ {"18446744073709551617", 18446744073709551616}, // 2^64+1 truncates to 2^64
116+ {"-18446744073709551615", -18446744073709549568},
117+ {"12345678901234567890.0", 12345678901234567168}, // a decimal point: an ordinary float
118+ {"1e30", 1e30},
119+ } {
120+ v, err := FromJSON([]byte(tc.src))
121+ if err != nil {
122+ t.Errorf("FromJSON(%q): %v", tc.src, err)
123+ continue
124+ }
125+ if got := float64(v.(Number)); got != tc.want {
126+ t.Errorf("FromJSON(%q) = %v, want %v", tc.src, got, tc.want)
127+ }
128+ }
129+}
130+
103131 func TestFromJSONInvalidUTF8(t *testing.T) {
104132 v, err := FromJSON([]byte("\"a\xffb\""))
105133 if err != nil {
@@ -100,6 +100,34 @@ func TestFromJSONNumbers(t *testing.T) {
100 }100 }
101 }101 }
102 102
103+// TestFromJSONBigIntegers: Json.NET gives integers beyond 64 bits as BigInteger
104+// and (Double)BigInteger truncates; established with the runner's code
105+// (fromJSON('N') == N, where the literal N rounds to nearest).
106+func TestFromJSONBigIntegers(t *testing.T) {
107+ for _, tc := range []struct {
108+ src string
109+ want float64
110+ }{
111+ {"9223372036854775807", 9223372036854775807}, // fits a long: nearest
112+ {"-9223372036854775808", -9223372036854775808}, // exact
113+ {"12345678901234567890", 12345678901234567890}, // 64 bits, above long: truncated to the same double here
114+ {"18446744073709551615", 18446744073709549568}, // 2^64-1 truncates to 2^64-2048, not 2^64
115+ {"18446744073709551617", 18446744073709551616}, // 2^64+1 truncates to 2^64
116+ {"-18446744073709551615", -18446744073709549568},
117+ {"12345678901234567890.0", 12345678901234567168}, // a decimal point: an ordinary float
118+ {"1e30", 1e30},
119+ } {
120+ v, err := FromJSON([]byte(tc.src))
121+ if err != nil {
122+ t.Errorf("FromJSON(%q): %v", tc.src, err)
123+ continue
124+ }
125+ if got := float64(v.(Number)); got != tc.want {
126+ t.Errorf("FromJSON(%q) = %v, want %v", tc.src, got, tc.want)
127+ }
128+ }
129+}
130+
103 func TestFromJSONInvalidUTF8(t *testing.T) {131 func TestFromJSONInvalidUTF8(t *testing.T) {
104 v, err := FromJSON([]byte("\"a\xffb\""))132 v, err := FromJSON([]byte("\"a\xffb\""))
105 if err != nil {133 if err != nil {
M expr/lexer.go +5 -2
@@ -81,8 +81,11 @@ func tokenString(t Token) string {
8181 return t.Lexeme
8282 }
8383
84-// Lex splits an expression into tokens. The result always ends with a
85-// TokenEOF token.
84+// Lex splits an expression into tokens the way the JavaScript reference's
85+// lexer does, ending with a TokenEOF token. [Parse] does not use it: it
86+// follows the C# runner, whose tokenizer is context sensitive and lazy (see
87+// parse_runner.go). Lex is kept because the oracle tests replay the reference's
88+// tokens with it.
8689 func Lex(input string) ([]Token, error) {
8790 if exceedsUTF16Len(input, MaxExpressionLength) {
8891 // The reference throws a bare Error("ErrorExceededMaxLength") here,
@@ -81,8 +81,11 @@ func tokenString(t Token) string {
81 return t.Lexeme81 return t.Lexeme
82 }82 }
83 83
84-// Lex splits an expression into tokens. The result always ends with a84+// Lex splits an expression into tokens the way the JavaScript reference's
85-// TokenEOF token.85+// lexer does, ending with a TokenEOF token. [Parse] does not use it: it
86+// follows the C# runner, whose tokenizer is context sensitive and lazy (see
87+// parse_runner.go). Lex is kept because the oracle tests replay the reference's
88+// tokens with it.
86 func Lex(input string) ([]Token, error) {89 func Lex(input string) ([]Token, error) {
87 if exceedsUTF16Len(input, MaxExpressionLength) {90 if exceedsUTF16Len(input, MaxExpressionLength) {
88 // The reference throws a bare Error("ErrorExceededMaxLength") here,91 // The reference throws a bare Error("ErrorExceededMaxLength") here,
A expr/parse_runner.go +790 -0
@@ -0,0 +1,790 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "strings"
7+ "unicode/utf16"
8+)
9+
10+// This file is a port of the runner's tokenizer and parser:
11+// src/Sdk/DTExpressions2/Expressions2/Tokens/LexicalAnalyzer.cs, Tokens/Token.cs
12+// and ExpressionParser.cs. They differ from the JavaScript reference's
13+// (lexer.go, parser.go) in ways users can see:
14+//
15+// - the tokenizer is context sensitive: each token is checked against the
16+// one before it, so "a b", "1 2" or "f(1,)" fail at the second token;
17+// - tokens are produced on demand and the parser stops at the first
18+// unexpected token or unknown name, in order of appearance (the JavaScript
19+// lexes everything first, so a later lexical error wins there);
20+// - white space is Char.IsWhiteSpace (U+0085 and U+00A0 included);
21+// - "Unexpected end of expression" is reported for an unfinished expression
22+// where the JavaScript says "Unexpected symbol: 'EOF'";
23+// - error positions are absolute indexes in the expression, not columns;
24+// - the depth limit applies to the shape of the finished tree, in which
25+// parentheses do not count and a flat run of && or || is one node;
26+// - the "too long" and "too deep" errors carry no token.
27+//
28+// Parse builds the same AST as the JavaScript-style parser did.
29+
30+type csKind int
31+
32+const (
33+ csStartGroup csKind = iota
34+ csStartIndex
35+ csStartParameters
36+ csEndGroup
37+ csEndIndex
38+ csEndParameters
39+ csSeparator
40+ csDereference
41+ csWildcard
42+ csLogicalOperator
43+ csNull
44+ csBoolean
45+ csNumber
46+ csString
47+ csPropertyName
48+ csFunction
49+ csNamedValue
50+ csUnexpected
51+)
52+
53+type csToken struct {
54+ kind csKind
55+ raw string
56+ index int
57+ value Value // Boolean, Number or String for literals
58+}
59+
60+func (t *csToken) isOperator() bool {
61+ switch t.kind {
62+ case csStartGroup, csStartIndex, csStartParameters, csEndGroup, csEndIndex, csEndParameters, csSeparator, csDereference, csLogicalOperator:
63+ return true
64+ }
65+ return false
66+}
67+
68+// leftToRight reports whether the token is a left-associative operator.
69+func (t *csToken) leftToRight() bool {
70+ switch {
71+ case t.kind == csStartGroup:
72+ return false
73+ case t.kind == csLogicalOperator && t.raw == "!":
74+ return false
75+ }
76+ return t.isOperator()
77+}
78+
79+func (t *csToken) precedence() int {
80+ switch t.kind {
81+ case csStartGroup:
82+ return 20
83+ case csStartIndex, csStartParameters, csDereference:
84+ return 19
85+ case csLogicalOperator:
86+ switch t.raw {
87+ case "!":
88+ return 16
89+ case ">", ">=", "<", "<=":
90+ return 11
91+ case "==", "!=":
92+ return 10
93+ case "&&":
94+ return 6
95+ case "||":
96+ return 5
97+ }
98+ case csEndGroup, csEndIndex, csEndParameters, csSeparator:
99+ return 1
100+ }
101+ return 0
102+}
103+
104+func (t *csToken) operandCount() int {
105+ switch t.kind {
106+ case csStartIndex, csDereference:
107+ return 2
108+ case csLogicalOperator:
109+ if t.raw == "!" {
110+ return 1
111+ }
112+ return 2
113+ }
114+ return 0
115+}
116+
117+// csLexer is LexicalAnalyzer.
118+type csLexer struct {
119+ units []uint16
120+ index int
121+ last *csToken
122+ unclosed []*csToken // stack of "(", "[" and function "("
123+}
124+
125+func (l *csLexer) substr(from, to int) string { return string(utf16.Decode(l.units[from:to])) }
126+
127+func isCSWhiteSpace(c uint16) bool { return isNetSpace(rune(c)) }
128+
129+func isCSBoundary(c uint16) bool {
130+ switch c {
131+ case '(', '[', ')', ']', ',', '.', '!', '>', '<', '=', '&', '|':
132+ return true
133+ }
134+ return isCSWhiteSpace(c)
135+}
136+
137+func (l *csLexer) topUnclosed() csKind {
138+ if len(l.unclosed) == 0 {
139+ return -1
140+ }
141+ return l.unclosed[len(l.unclosed)-1].kind
142+}
143+
144+// next returns the next token, or nil at the end of the expression.
145+func (l *csLexer) next() *csToken {
146+ for l.index < len(l.units) && isCSWhiteSpace(l.units[l.index]) {
147+ l.index++
148+ }
149+ if l.index >= len(l.units) {
150+ return nil
151+ }
152+ c := l.units[l.index]
153+ var tok *csToken
154+ switch c {
155+ case '(':
156+ if l.last != nil && l.last.kind == csFunction {
157+ tok = l.create(csStartParameters, "(", l.index, nil)
158+ } else {
159+ tok = l.create(csStartGroup, "(", l.index, nil)
160+ }
161+ l.index++
162+ case '[':
163+ tok = l.create(csStartIndex, "[", l.index, nil)
164+ l.index++
165+ case ')':
166+ if l.topUnclosed() == csStartParameters {
167+ tok = l.create(csEndParameters, ")", l.index, nil)
168+ } else {
169+ tok = l.create(csEndGroup, ")", l.index, nil)
170+ }
171+ l.index++
172+ case ']':
173+ tok = l.create(csEndIndex, "]", l.index, nil)
174+ l.index++
175+ case ',':
176+ tok = l.create(csSeparator, ",", l.index, nil)
177+ l.index++
178+ case '*':
179+ tok = l.create(csWildcard, "*", l.index, nil)
180+ l.index++
181+ case '\'':
182+ tok = l.readString()
183+ case '!', '>', '<', '=', '&', '|':
184+ tok = l.readOperator()
185+ default:
186+ switch {
187+ case c == '.':
188+ if l.last == nil || l.last.kind == csSeparator || l.last.kind == csStartGroup || l.last.kind == csStartIndex ||
189+ l.last.kind == csStartParameters || l.last.kind == csLogicalOperator {
190+ tok = l.readNumber()
191+ } else {
192+ tok = l.create(csDereference, ".", l.index, nil)
193+ l.index++
194+ }
195+ case c == '-' || c == '+' || (c >= '0' && c <= '9'):
196+ tok = l.readNumber()
197+ default:
198+ tok = l.readKeyword()
199+ }
200+ }
201+ l.last = tok
202+ return tok
203+}
204+
205+func (l *csLexer) readNumber() *csToken {
206+ start := l.index
207+ for {
208+ l.index++
209+ if !(l.index < len(l.units) && (!isCSBoundary(l.units[l.index]) || l.units[l.index] == '.')) {
210+ break
211+ }
212+ }
213+ str := l.substr(start, l.index)
214+ d := parseNumber(str)
215+ if d != d { // NaN
216+ return l.create(csUnexpected, str, start, nil)
217+ }
218+ return l.create(csNumber, str, start, Number(d))
219+}
220+
221+func isLegalKeyword(s string) bool { return isLegalIdentifier(s) }
222+
223+func (l *csLexer) readKeyword() *csToken {
224+ start := l.index
225+ l.index++
226+ for l.index < len(l.units) && !isCSBoundary(l.units[l.index]) {
227+ l.index++
228+ }
229+ str := l.substr(start, l.index)
230+ if !isLegalKeyword(str) {
231+ return l.create(csUnexpected, str, start, nil)
232+ }
233+ if l.last != nil && l.last.kind == csDereference {
234+ return l.create(csPropertyName, str, start, nil)
235+ }
236+ switch str {
237+ case "null":
238+ return l.create(csNull, str, start, Null{})
239+ case "true":
240+ return l.create(csBoolean, str, start, Boolean(true))
241+ case "false":
242+ return l.create(csBoolean, str, start, Boolean(false))
243+ case "NaN":
244+ return l.create(csNumber, str, start, Number(nan))
245+ case "Infinity":
246+ return l.create(csNumber, str, start, Number(posInf))
247+ }
248+ // Lookahead: a function is a name followed by "(", white space allowed.
249+ t := l.index
250+ for t < len(l.units) && isCSWhiteSpace(l.units[t]) {
251+ t++
252+ }
253+ if t < len(l.units) && l.units[t] == '(' {
254+ return l.create(csFunction, str, start, nil)
255+ }
256+ return l.create(csNamedValue, str, start, nil)
257+}
258+
259+func (l *csLexer) readString() *csToken {
260+ start := l.index
261+ var b strings.Builder
262+ closed := false
263+ l.index++ // the opening quote
264+ var run []uint16
265+ for l.index < len(l.units) {
266+ c := l.units[l.index]
267+ l.index++
268+ if c == '\'' {
269+ if l.index >= len(l.units) || l.units[l.index] != '\'' {
270+ closed = true
271+ break
272+ }
273+ l.index++ // an escaped quote
274+ }
275+ run = append(run, c)
276+ }
277+ b.WriteString(string(utf16.Decode(run)))
278+ raw := l.substr(start, l.index)
279+ if closed {
280+ return l.create(csString, raw, start, String(b.String()))
281+ }
282+ return l.create(csUnexpected, raw, start, nil)
283+}
284+
285+func (l *csLexer) readOperator() *csToken {
286+ start := l.index
287+ l.index++
288+ if l.index < len(l.units) {
289+ l.index++
290+ switch raw := l.substr(start, start+2); raw {
291+ case "!=", ">=", "<=", "==", "&&", "||":
292+ return l.create(csLogicalOperator, raw, start, nil)
293+ }
294+ l.index-- // back up
295+ }
296+ switch raw := l.substr(start, start+1); raw {
297+ case "!", ">", "<":
298+ return l.create(csLogicalOperator, raw, start, nil)
299+ }
300+ for l.index < len(l.units) && !isCSBoundary(l.units[l.index]) {
301+ l.index++
302+ }
303+ return l.create(csUnexpected, l.substr(start, l.index), start, nil)
304+}
305+
306+// lastIs reports whether the previous token is one of kinds; a nil kind list
307+// entry (csUnexpected here) stands for "no previous token".
308+func (l *csLexer) lastIs(kinds ...csKind) bool {
309+ if l.last == nil {
310+ for _, k := range kinds {
311+ if k == -1 {
312+ return true
313+ }
314+ }
315+ return false
316+ }
317+ for _, k := range kinds {
318+ if k == l.last.kind {
319+ return true
320+ }
321+ }
322+ return false
323+}
324+
325+const csFirst csKind = -1 // "is the first token"
326+
327+// create builds a token after checking that it may follow the previous one
328+// and that brackets are balanced; an illegal one becomes csUnexpected.
329+func (l *csLexer) create(kind csKind, raw string, index int, v Value) *csToken {
330+ var legal bool
331+ switch kind {
332+ case csStartGroup:
333+ legal = l.lastIs(csFirst, csSeparator, csStartGroup, csStartParameters, csStartIndex, csLogicalOperator)
334+ case csStartIndex:
335+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csPropertyName, csNamedValue)
336+ case csStartParameters:
337+ legal = l.lastIs(csFunction)
338+ case csEndGroup, csEndIndex:
339+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
340+ case csEndParameters:
341+ legal = l.lastIs(csStartParameters, csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
342+ case csSeparator:
343+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
344+ case csDereference:
345+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csPropertyName, csNamedValue)
346+ case csWildcard:
347+ legal = l.lastIs(csStartIndex, csDereference)
348+ case csLogicalOperator:
349+ if raw == "!" {
350+ legal = l.lastIs(csFirst, csSeparator, csStartGroup, csStartParameters, csStartIndex, csLogicalOperator)
351+ } else {
352+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
353+ }
354+ case csNull, csBoolean, csNumber, csString, csFunction, csNamedValue:
355+ legal = l.lastIs(csFirst, csSeparator, csStartIndex, csStartGroup, csStartParameters, csLogicalOperator)
356+ case csPropertyName:
357+ legal = l.lastIs(csDereference)
358+ }
359+ if !legal {
360+ return &csToken{kind: csUnexpected, raw: raw, index: index}
361+ }
362+ tok := &csToken{kind: kind, raw: raw, index: index, value: v}
363+ switch kind {
364+ case csStartGroup, csStartIndex, csStartParameters:
365+ l.unclosed = append(l.unclosed, tok)
366+ case csEndGroup:
367+ if l.topUnclosed() != csStartGroup {
368+ return &csToken{kind: csUnexpected, raw: raw, index: index}
369+ }
370+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
371+ case csEndIndex:
372+ if l.topUnclosed() != csStartIndex {
373+ return &csToken{kind: csUnexpected, raw: raw, index: index}
374+ }
375+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
376+ case csEndParameters:
377+ if l.topUnclosed() != csStartParameters {
378+ return &csToken{kind: csUnexpected, raw: raw, index: index}
379+ }
380+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
381+ case csSeparator:
382+ if l.topUnclosed() != csStartParameters {
383+ return &csToken{kind: csUnexpected, raw: raw, index: index}
384+ }
385+ }
386+ return tok
387+}
388+
389+// csParser is ExpressionParser.CreateTree: a shunting-yard parser.
390+type csParser struct {
391+ expression string
392+ units []uint16
393+ lex *csLexer
394+ contexts map[string]bool
395+ funcs map[string]FuncInfo
396+ operands []Node
397+ operators []*csToken
398+ last *csToken // the previous token
399+ cur *csToken
400+}
401+
402+// fail builds the ParseException of the runner: with a token, the message names
403+// it and its absolute position; without, only the description.
404+func (p *csParser) fail(t ErrorType, tok *csToken) *Error {
405+ if tok == nil {
406+ return &Error{Type: t, Message: t.description()}
407+ }
408+ return &Error{
409+ Type: t,
410+ Message: t.description() + ": '" + tok.raw + "'. Located at position " + itoa(tok.index+1) + " within expression: " + p.expression,
411+ Pos: p.position(tok.index),
412+ Token: tok.raw,
413+ }
414+}
415+
416+func itoa(n int) string { return strings.TrimSpace(strings.Repeat(" ", 0) + intString(n)) }
417+
418+func intString(n int) string {
419+ if n == 0 {
420+ return "0"
421+ }
422+ var b [20]byte
423+ i := len(b)
424+ for n > 0 {
425+ i--
426+ b[i] = byte('0' + n%10)
427+ n /= 10
428+ }
429+ return string(b[i:])
430+}
431+
432+// position converts an absolute index into the line and column of the
433+// JavaScript-style Position (line breaks are U+000A).
434+func (p *csParser) position(index int) Position {
435+ line, lineStart := 0, 0
436+ for i := 0; i < index && i < len(p.units); i++ {
437+ if p.units[i] == '\n' {
438+ line++
439+ lineStart = i + 1
440+ }
441+ }
442+ return Position{Line: line, Column: index - lineStart}
443+}
444+
445+func (p *csParser) token(t *csToken, typ TokenType) Token {
446+ end := t.index + len(utf16.Encode([]rune(t.raw)))
447+ return Token{Type: typ, Lexeme: t.raw, Range: Range{Start: p.position(t.index), End: p.position(end)}, Value: t.value}
448+}
449+
450+func (p *csParser) wellKnownOrExtension(name string) (FuncInfo, bool) {
451+ k := foldKey(name)
452+ if f, ok := builtinFuncs[k]; ok {
453+ return f, true
454+ }
455+ f, ok := p.funcs[k]
456+ return f, ok
457+}
458+
459+// parseRunner parses expression the way the runner does.
460+func parseRunner(expression string, contexts []string, funcs []FuncInfo) (Node, error) {
461+ p := &csParser{contexts: make(map[string]bool, len(contexts)), funcs: make(map[string]FuncInfo, len(funcs))}
462+ if exceedsUTF16Len(expression, MaxExpressionLength) {
463+ return nil, p.fail(ErrorExceededMaxLength, nil)
464+ }
465+ p.units = utf16.Encode([]rune(expression))
466+ p.expression = string(utf16.Decode(p.units))
467+ for _, c := range contexts {
468+ p.contexts[foldKey(c)] = true
469+ }
470+ for _, f := range funcs {
471+ p.funcs[foldKey(f.Name)] = f
472+ }
473+ p.lex = &csLexer{units: p.units}
474+ return p.createTree()
475+}
476+
477+func (p *csParser) createTree() (Node, error) {
478+ for {
479+ tok := p.lex.next()
480+ if tok == nil {
481+ break
482+ }
483+ p.cur = tok
484+ switch {
485+ case tok.kind == csUnexpected:
486+ return nil, p.fail(ErrorUnexpectedSymbol, tok)
487+ case tok.isOperator():
488+ if err := p.pushOperator(tok); err != nil {
489+ return nil, err
490+ }
491+ default:
492+ if err := p.pushOperand(tok); err != nil {
493+ return nil, err
494+ }
495+ }
496+ p.last = tok
497+ }
498+ if p.last == nil {
499+ return nil, nil
500+ }
501+ if len(p.operators) > 0 {
502+ unexpected := false
503+ switch p.last.kind {
504+ case csEndGroup, csEndIndex, csEndParameters:
505+ case csFunction:
506+ unexpected = true
507+ default:
508+ unexpected = p.last.isOperator()
509+ }
510+ if unexpected || len(p.lex.unclosed) > 0 {
511+ return nil, p.fail(ErrorUnexpectedEndOfExpression, p.last)
512+ }
513+ }
514+ for len(p.operators) > 0 {
515+ if err := p.flushTop(); err != nil {
516+ return nil, err
517+ }
518+ }
519+ if len(p.operands) != 1 {
520+ return nil, p.fail(ErrorUnexpectedSymbol, p.last)
521+ }
522+ root := p.operands[0]
523+ if exceedsDepth(root, 1) {
524+ return nil, p.fail(ErrorExceededMaxDepth, nil)
525+ }
526+ return root, nil
527+}
528+
529+func (p *csParser) pushOperand(tok *csToken) error {
530+ var n Node
531+ switch tok.kind {
532+ case csFunction:
533+ if _, ok := p.wellKnownOrExtension(tok.raw); !ok {
534+ return p.fail(ErrorUnrecognizedFunction, tok)
535+ }
536+ n = &FunctionCall{Name: p.token(tok, TokenIdentifier)}
537+ case csNamedValue:
538+ if !p.contexts[foldKey(tok.raw)] {
539+ return p.fail(ErrorUnrecognizedContext, tok)
540+ }
541+ n = &ContextAccess{Name: p.token(tok, TokenIdentifier)}
542+ case csNull:
543+ n = &Literal{Value: Null{}, Token: p.token(tok, TokenNull)}
544+ case csBoolean:
545+ typ := TokenFalse
546+ if tok.value == Boolean(true) {
547+ typ = TokenTrue
548+ }
549+ n = &Literal{Value: tok.value, Token: p.token(tok, typ)}
550+ case csNumber:
551+ n = &Literal{Value: tok.value, Token: p.token(tok, TokenNumber)}
552+ case csString:
553+ n = &Literal{Value: tok.value, Token: p.token(tok, TokenString)}
554+ case csPropertyName:
555+ n = &Literal{Value: String(tok.raw), Token: p.token(tok, TokenIdentifier)}
556+ case csWildcard:
557+ n = &Star{}
558+ default:
559+ return p.fail(ErrorUnexpectedSymbol, tok)
560+ }
561+ p.operands = append(p.operands, n)
562+ return nil
563+}
564+
565+func (p *csParser) pushOperator(tok *csToken) error {
566+ if tok.leftToRight() {
567+ prec := tok.precedence()
568+ for len(p.operators) > 0 {
569+ top := p.operators[len(p.operators)-1]
570+ if prec <= top.precedence() && top.kind != csStartGroup && top.kind != csStartIndex &&
571+ top.kind != csStartParameters && top.kind != csSeparator {
572+ if err := p.flushTop(); err != nil {
573+ return err
574+ }
575+ continue
576+ }
577+ break
578+ }
579+ }
580+ p.operators = append(p.operators, tok)
581+ switch tok.kind {
582+ case csEndGroup, csEndIndex, csEndParameters:
583+ return p.flushTop()
584+ }
585+ return nil
586+}
587+
588+func (p *csParser) popOperator(want csKind) (*csToken, error) {
589+ if len(p.operators) == 0 || p.operators[len(p.operators)-1].kind != want {
590+ return nil, p.fail(ErrorUnexpectedSymbol, p.cur)
591+ }
592+ t := p.operators[len(p.operators)-1]
593+ p.operators = p.operators[:len(p.operators)-1]
594+ return t, nil
595+}
596+
597+// popOperands pops n operands, returned in their written order.
598+func (p *csParser) popOperands(n int) ([]Node, error) {
599+ if len(p.operands) < n {
600+ return nil, p.fail(ErrorUnexpectedSymbol, p.cur)
601+ }
602+ out := append([]Node(nil), p.operands[len(p.operands)-n:]...)
603+ p.operands = p.operands[:len(p.operands)-n]
604+ return out, nil
605+}
606+
607+func (p *csParser) flushTop() error {
608+ top := p.operators[len(p.operators)-1]
609+ switch top.kind {
610+ case csEndIndex:
611+ if _, err := p.popOperator(csEndIndex); err != nil {
612+ return err
613+ }
614+ op, err := p.popOperator(csStartIndex)
615+ if err != nil {
616+ return err
617+ }
618+ args, err := p.popOperands(op.operandCount())
619+ if err != nil {
620+ return err
621+ }
622+ p.operands = append(p.operands, &IndexAccess{Expr: args[0], Index: args[1]})
623+ return nil
624+ case csEndGroup:
625+ if _, err := p.popOperator(csEndGroup); err != nil {
626+ return err
627+ }
628+ if _, err := p.popOperator(csStartGroup); err != nil {
629+ return err
630+ }
631+ if len(p.operands) == 0 {
632+ return p.fail(ErrorUnexpectedSymbol, p.cur)
633+ }
634+ g := &Grouping{Group: p.operands[len(p.operands)-1]}
635+ p.operands[len(p.operands)-1] = g
636+ return nil
637+ case csEndParameters:
638+ return p.flushEndParameters()
639+ }
640+ op, _ := p.popOperator(top.kind)
641+ args, err := p.popOperands(op.operandCount())
642+ if err != nil {
643+ return err
644+ }
645+ switch {
646+ case op.kind == csStartIndex || op.kind == csDereference:
647+ p.operands = append(p.operands, &IndexAccess{Expr: args[0], Index: args[1]})
648+ case op.kind == csLogicalOperator && op.raw == "!":
649+ p.operands = append(p.operands, &Unary{Operator: p.token(op, TokenBang), Expr: args[0]})
650+ case op.kind == csLogicalOperator && (op.raw == "&&" || op.raw == "||"):
651+ typ := TokenAnd
652+ if op.raw == "||" {
653+ typ = TokenOr
654+ }
655+ l := &Logical{Operator: p.token(op, typ)}
656+ for _, a := range args {
657+ // A nested chain of the same operator is merged, as the runner does.
658+ if nested, ok := a.(*Logical); ok && nested.Operator.Type == typ {
659+ l.Args = append(l.Args, nested.Args...)
660+ } else {
661+ l.Args = append(l.Args, a)
662+ }
663+ }
664+ p.operands = append(p.operands, l)
665+ case op.kind == csLogicalOperator:
666+ var typ TokenType
667+ switch op.raw {
668+ case "!=":
669+ typ = TokenBangEqual
670+ case "==":
671+ typ = TokenEqualEqual
672+ case ">":
673+ typ = TokenGreater
674+ case ">=":
675+ typ = TokenGreaterEqual
676+ case "<":
677+ typ = TokenLess
678+ default:
679+ typ = TokenLessEqual
680+ }
681+ p.operands = append(p.operands, &Binary{Left: args[0], Operator: p.token(op, typ), Right: args[1]})
682+ default:
683+ return p.fail(ErrorUnexpectedSymbol, op)
684+ }
685+ return nil
686+}
687+
688+func (p *csParser) flushEndParameters() error {
689+ end, err := p.popOperator(csEndParameters)
690+ if err != nil {
691+ return err
692+ }
693+ if end != p.cur || p.last == nil {
694+ return p.fail(ErrorUnexpectedSymbol, p.cur)
695+ }
696+ count := 0
697+ if p.last.kind != csStartParameters {
698+ count = 1
699+ for len(p.operators) > 0 && p.operators[len(p.operators)-1].kind == csSeparator {
700+ count++
701+ p.operators = p.operators[:len(p.operators)-1]
702+ }
703+ }
704+ args, err := p.popOperands(count)
705+ if err != nil {
706+ return err
707+ }
708+ if len(p.operands) == 0 {
709+ return p.fail(ErrorUnexpectedSymbol, p.cur)
710+ }
711+ call, ok := p.operands[len(p.operands)-1].(*FunctionCall)
712+ if !ok {
713+ return p.fail(ErrorUnexpectedSymbol, p.cur)
714+ }
715+ call.Args = args
716+ open, err := p.popOperator(csStartParameters)
717+ if err != nil {
718+ return err
719+ }
720+ info, _ := p.wellKnownOrExtension(call.Name.Lexeme)
721+ switch {
722+ case len(args) < info.MinArgs:
723+ return p.fail(ErrorTooFewParameters, open)
724+ case info.MaxArgs >= 0 && len(args) > info.MaxArgs:
725+ return p.fail(ErrorTooManyParameters, open)
726+ case strings.EqualFold(info.Name, "case") && len(args)%2 == 0:
727+ return p.fail(ErrorEvenParameters, open)
728+ }
729+ return nil
730+}
731+
732+// exceedsDepth reports whether the tree is deeper than MaxParserDepth, counted
733+// as the runner does (CheckMaxDepth): the root is at depth 1, the operands of
734+// an operator, the arguments of a call and the two parts of an index access
735+// are one deeper, parentheses add nothing, and a run of && or || is one node,
736+// also when parenthesized.
737+func exceedsDepth(n Node, depth int) bool {
738+ for {
739+ g, ok := n.(*Grouping)
740+ if !ok {
741+ break
742+ }
743+ n = g.Group
744+ }
745+ if depth > MaxParserDepth {
746+ return true
747+ }
748+ switch n := n.(type) {
749+ case *Unary:
750+ return exceedsDepth(n.Expr, depth+1)
751+ case *Binary:
752+ return exceedsDepth(n.Left, depth+1) || exceedsDepth(n.Right, depth+1)
753+ case *Logical:
754+ for _, a := range flatOperands(n, nil) {
755+ if exceedsDepth(a, depth+1) {
756+ return true
757+ }
758+ }
759+ case *FunctionCall:
760+ for _, a := range n.Args {
761+ if exceedsDepth(a, depth+1) {
762+ return true
763+ }
764+ }
765+ case *IndexAccess:
766+ return exceedsDepth(n.Expr, depth+1) || exceedsDepth(n.Index, depth+1)
767+ }
768+ return false
769+}
770+
771+// flatOperands lists the operands of a && or || chain with nested chains of
772+// the same operator (parenthesized or not) merged into it.
773+func flatOperands(l *Logical, out []Node) []Node {
774+ for _, a := range l.Args {
775+ inner := a
776+ for {
777+ g, ok := inner.(*Grouping)
778+ if !ok {
779+ break
780+ }
781+ inner = g.Group
782+ }
783+ if nested, ok := inner.(*Logical); ok && nested.Operator.Type == l.Operator.Type {
784+ out = flatOperands(nested, out)
785+ } else {
786+ out = append(out, a)
787+ }
788+ }
789+ return out
790+}
@@ -0,0 +1,790 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "strings"
7+ "unicode/utf16"
8+)
9+
10+// This file is a port of the runner's tokenizer and parser:
11+// src/Sdk/DTExpressions2/Expressions2/Tokens/LexicalAnalyzer.cs, Tokens/Token.cs
12+// and ExpressionParser.cs. They differ from the JavaScript reference's
13+// (lexer.go, parser.go) in ways users can see:
14+//
15+// - the tokenizer is context sensitive: each token is checked against the
16+// one before it, so "a b", "1 2" or "f(1,)" fail at the second token;
17+// - tokens are produced on demand and the parser stops at the first
18+// unexpected token or unknown name, in order of appearance (the JavaScript
19+// lexes everything first, so a later lexical error wins there);
20+// - white space is Char.IsWhiteSpace (U+0085 and U+00A0 included);
21+// - "Unexpected end of expression" is reported for an unfinished expression
22+// where the JavaScript says "Unexpected symbol: 'EOF'";
23+// - error positions are absolute indexes in the expression, not columns;
24+// - the depth limit applies to the shape of the finished tree, in which
25+// parentheses do not count and a flat run of && or || is one node;
26+// - the "too long" and "too deep" errors carry no token.
27+//
28+// Parse builds the same AST as the JavaScript-style parser did.
29+
30+type csKind int
31+
32+const (
33+ csStartGroup csKind = iota
34+ csStartIndex
35+ csStartParameters
36+ csEndGroup
37+ csEndIndex
38+ csEndParameters
39+ csSeparator
40+ csDereference
41+ csWildcard
42+ csLogicalOperator
43+ csNull
44+ csBoolean
45+ csNumber
46+ csString
47+ csPropertyName
48+ csFunction
49+ csNamedValue
50+ csUnexpected
51+)
52+
53+type csToken struct {
54+ kind csKind
55+ raw string
56+ index int
57+ value Value // Boolean, Number or String for literals
58+}
59+
60+func (t *csToken) isOperator() bool {
61+ switch t.kind {
62+ case csStartGroup, csStartIndex, csStartParameters, csEndGroup, csEndIndex, csEndParameters, csSeparator, csDereference, csLogicalOperator:
63+ return true
64+ }
65+ return false
66+}
67+
68+// leftToRight reports whether the token is a left-associative operator.
69+func (t *csToken) leftToRight() bool {
70+ switch {
71+ case t.kind == csStartGroup:
72+ return false
73+ case t.kind == csLogicalOperator && t.raw == "!":
74+ return false
75+ }
76+ return t.isOperator()
77+}
78+
79+func (t *csToken) precedence() int {
80+ switch t.kind {
81+ case csStartGroup:
82+ return 20
83+ case csStartIndex, csStartParameters, csDereference:
84+ return 19
85+ case csLogicalOperator:
86+ switch t.raw {
87+ case "!":
88+ return 16
89+ case ">", ">=", "<", "<=":
90+ return 11
91+ case "==", "!=":
92+ return 10
93+ case "&&":
94+ return 6
95+ case "||":
96+ return 5
97+ }
98+ case csEndGroup, csEndIndex, csEndParameters, csSeparator:
99+ return 1
100+ }
101+ return 0
102+}
103+
104+func (t *csToken) operandCount() int {
105+ switch t.kind {
106+ case csStartIndex, csDereference:
107+ return 2
108+ case csLogicalOperator:
109+ if t.raw == "!" {
110+ return 1
111+ }
112+ return 2
113+ }
114+ return 0
115+}
116+
117+// csLexer is LexicalAnalyzer.
118+type csLexer struct {
119+ units []uint16
120+ index int
121+ last *csToken
122+ unclosed []*csToken // stack of "(", "[" and function "("
123+}
124+
125+func (l *csLexer) substr(from, to int) string { return string(utf16.Decode(l.units[from:to])) }
126+
127+func isCSWhiteSpace(c uint16) bool { return isNetSpace(rune(c)) }
128+
129+func isCSBoundary(c uint16) bool {
130+ switch c {
131+ case '(', '[', ')', ']', ',', '.', '!', '>', '<', '=', '&', '|':
132+ return true
133+ }
134+ return isCSWhiteSpace(c)
135+}
136+
137+func (l *csLexer) topUnclosed() csKind {
138+ if len(l.unclosed) == 0 {
139+ return -1
140+ }
141+ return l.unclosed[len(l.unclosed)-1].kind
142+}
143+
144+// next returns the next token, or nil at the end of the expression.
145+func (l *csLexer) next() *csToken {
146+ for l.index < len(l.units) && isCSWhiteSpace(l.units[l.index]) {
147+ l.index++
148+ }
149+ if l.index >= len(l.units) {
150+ return nil
151+ }
152+ c := l.units[l.index]
153+ var tok *csToken
154+ switch c {
155+ case '(':
156+ if l.last != nil && l.last.kind == csFunction {
157+ tok = l.create(csStartParameters, "(", l.index, nil)
158+ } else {
159+ tok = l.create(csStartGroup, "(", l.index, nil)
160+ }
161+ l.index++
162+ case '[':
163+ tok = l.create(csStartIndex, "[", l.index, nil)
164+ l.index++
165+ case ')':
166+ if l.topUnclosed() == csStartParameters {
167+ tok = l.create(csEndParameters, ")", l.index, nil)
168+ } else {
169+ tok = l.create(csEndGroup, ")", l.index, nil)
170+ }
171+ l.index++
172+ case ']':
173+ tok = l.create(csEndIndex, "]", l.index, nil)
174+ l.index++
175+ case ',':
176+ tok = l.create(csSeparator, ",", l.index, nil)
177+ l.index++
178+ case '*':
179+ tok = l.create(csWildcard, "*", l.index, nil)
180+ l.index++
181+ case '\'':
182+ tok = l.readString()
183+ case '!', '>', '<', '=', '&', '|':
184+ tok = l.readOperator()
185+ default:
186+ switch {
187+ case c == '.':
188+ if l.last == nil || l.last.kind == csSeparator || l.last.kind == csStartGroup || l.last.kind == csStartIndex ||
189+ l.last.kind == csStartParameters || l.last.kind == csLogicalOperator {
190+ tok = l.readNumber()
191+ } else {
192+ tok = l.create(csDereference, ".", l.index, nil)
193+ l.index++
194+ }
195+ case c == '-' || c == '+' || (c >= '0' && c <= '9'):
196+ tok = l.readNumber()
197+ default:
198+ tok = l.readKeyword()
199+ }
200+ }
201+ l.last = tok
202+ return tok
203+}
204+
205+func (l *csLexer) readNumber() *csToken {
206+ start := l.index
207+ for {
208+ l.index++
209+ if !(l.index < len(l.units) && (!isCSBoundary(l.units[l.index]) || l.units[l.index] == '.')) {
210+ break
211+ }
212+ }
213+ str := l.substr(start, l.index)
214+ d := parseNumber(str)
215+ if d != d { // NaN
216+ return l.create(csUnexpected, str, start, nil)
217+ }
218+ return l.create(csNumber, str, start, Number(d))
219+}
220+
221+func isLegalKeyword(s string) bool { return isLegalIdentifier(s) }
222+
223+func (l *csLexer) readKeyword() *csToken {
224+ start := l.index
225+ l.index++
226+ for l.index < len(l.units) && !isCSBoundary(l.units[l.index]) {
227+ l.index++
228+ }
229+ str := l.substr(start, l.index)
230+ if !isLegalKeyword(str) {
231+ return l.create(csUnexpected, str, start, nil)
232+ }
233+ if l.last != nil && l.last.kind == csDereference {
234+ return l.create(csPropertyName, str, start, nil)
235+ }
236+ switch str {
237+ case "null":
238+ return l.create(csNull, str, start, Null{})
239+ case "true":
240+ return l.create(csBoolean, str, start, Boolean(true))
241+ case "false":
242+ return l.create(csBoolean, str, start, Boolean(false))
243+ case "NaN":
244+ return l.create(csNumber, str, start, Number(nan))
245+ case "Infinity":
246+ return l.create(csNumber, str, start, Number(posInf))
247+ }
248+ // Lookahead: a function is a name followed by "(", white space allowed.
249+ t := l.index
250+ for t < len(l.units) && isCSWhiteSpace(l.units[t]) {
251+ t++
252+ }
253+ if t < len(l.units) && l.units[t] == '(' {
254+ return l.create(csFunction, str, start, nil)
255+ }
256+ return l.create(csNamedValue, str, start, nil)
257+}
258+
259+func (l *csLexer) readString() *csToken {
260+ start := l.index
261+ var b strings.Builder
262+ closed := false
263+ l.index++ // the opening quote
264+ var run []uint16
265+ for l.index < len(l.units) {
266+ c := l.units[l.index]
267+ l.index++
268+ if c == '\'' {
269+ if l.index >= len(l.units) || l.units[l.index] != '\'' {
270+ closed = true
271+ break
272+ }
273+ l.index++ // an escaped quote
274+ }
275+ run = append(run, c)
276+ }
277+ b.WriteString(string(utf16.Decode(run)))
278+ raw := l.substr(start, l.index)
279+ if closed {
280+ return l.create(csString, raw, start, String(b.String()))
281+ }
282+ return l.create(csUnexpected, raw, start, nil)
283+}
284+
285+func (l *csLexer) readOperator() *csToken {
286+ start := l.index
287+ l.index++
288+ if l.index < len(l.units) {
289+ l.index++
290+ switch raw := l.substr(start, start+2); raw {
291+ case "!=", ">=", "<=", "==", "&&", "||":
292+ return l.create(csLogicalOperator, raw, start, nil)
293+ }
294+ l.index-- // back up
295+ }
296+ switch raw := l.substr(start, start+1); raw {
297+ case "!", ">", "<":
298+ return l.create(csLogicalOperator, raw, start, nil)
299+ }
300+ for l.index < len(l.units) && !isCSBoundary(l.units[l.index]) {
301+ l.index++
302+ }
303+ return l.create(csUnexpected, l.substr(start, l.index), start, nil)
304+}
305+
306+// lastIs reports whether the previous token is one of kinds; a nil kind list
307+// entry (csUnexpected here) stands for "no previous token".
308+func (l *csLexer) lastIs(kinds ...csKind) bool {
309+ if l.last == nil {
310+ for _, k := range kinds {
311+ if k == -1 {
312+ return true
313+ }
314+ }
315+ return false
316+ }
317+ for _, k := range kinds {
318+ if k == l.last.kind {
319+ return true
320+ }
321+ }
322+ return false
323+}
324+
325+const csFirst csKind = -1 // "is the first token"
326+
327+// create builds a token after checking that it may follow the previous one
328+// and that brackets are balanced; an illegal one becomes csUnexpected.
329+func (l *csLexer) create(kind csKind, raw string, index int, v Value) *csToken {
330+ var legal bool
331+ switch kind {
332+ case csStartGroup:
333+ legal = l.lastIs(csFirst, csSeparator, csStartGroup, csStartParameters, csStartIndex, csLogicalOperator)
334+ case csStartIndex:
335+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csPropertyName, csNamedValue)
336+ case csStartParameters:
337+ legal = l.lastIs(csFunction)
338+ case csEndGroup, csEndIndex:
339+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
340+ case csEndParameters:
341+ legal = l.lastIs(csStartParameters, csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
342+ case csSeparator:
343+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
344+ case csDereference:
345+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csPropertyName, csNamedValue)
346+ case csWildcard:
347+ legal = l.lastIs(csStartIndex, csDereference)
348+ case csLogicalOperator:
349+ if raw == "!" {
350+ legal = l.lastIs(csFirst, csSeparator, csStartGroup, csStartParameters, csStartIndex, csLogicalOperator)
351+ } else {
352+ legal = l.lastIs(csEndGroup, csEndParameters, csEndIndex, csWildcard, csNull, csBoolean, csNumber, csString, csPropertyName, csNamedValue)
353+ }
354+ case csNull, csBoolean, csNumber, csString, csFunction, csNamedValue:
355+ legal = l.lastIs(csFirst, csSeparator, csStartIndex, csStartGroup, csStartParameters, csLogicalOperator)
356+ case csPropertyName:
357+ legal = l.lastIs(csDereference)
358+ }
359+ if !legal {
360+ return &csToken{kind: csUnexpected, raw: raw, index: index}
361+ }
362+ tok := &csToken{kind: kind, raw: raw, index: index, value: v}
363+ switch kind {
364+ case csStartGroup, csStartIndex, csStartParameters:
365+ l.unclosed = append(l.unclosed, tok)
366+ case csEndGroup:
367+ if l.topUnclosed() != csStartGroup {
368+ return &csToken{kind: csUnexpected, raw: raw, index: index}
369+ }
370+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
371+ case csEndIndex:
372+ if l.topUnclosed() != csStartIndex {
373+ return &csToken{kind: csUnexpected, raw: raw, index: index}
374+ }
375+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
376+ case csEndParameters:
377+ if l.topUnclosed() != csStartParameters {
378+ return &csToken{kind: csUnexpected, raw: raw, index: index}
379+ }
380+ l.unclosed = l.unclosed[:len(l.unclosed)-1]
381+ case csSeparator:
382+ if l.topUnclosed() != csStartParameters {
383+ return &csToken{kind: csUnexpected, raw: raw, index: index}
384+ }
385+ }
386+ return tok
387+}
388+
389+// csParser is ExpressionParser.CreateTree: a shunting-yard parser.
390+type csParser struct {
391+ expression string
392+ units []uint16
393+ lex *csLexer
394+ contexts map[string]bool
395+ funcs map[string]FuncInfo
396+ operands []Node
397+ operators []*csToken
398+ last *csToken // the previous token
399+ cur *csToken
400+}
401+
402+// fail builds the ParseException of the runner: with a token, the message names
403+// it and its absolute position; without, only the description.
404+func (p *csParser) fail(t ErrorType, tok *csToken) *Error {
405+ if tok == nil {
406+ return &Error{Type: t, Message: t.description()}
407+ }
408+ return &Error{
409+ Type: t,
410+ Message: t.description() + ": '" + tok.raw + "'. Located at position " + itoa(tok.index+1) + " within expression: " + p.expression,
411+ Pos: p.position(tok.index),
412+ Token: tok.raw,
413+ }
414+}
415+
416+func itoa(n int) string { return strings.TrimSpace(strings.Repeat(" ", 0) + intString(n)) }
417+
418+func intString(n int) string {
419+ if n == 0 {
420+ return "0"
421+ }
422+ var b [20]byte
423+ i := len(b)
424+ for n > 0 {
425+ i--
426+ b[i] = byte('0' + n%10)
427+ n /= 10
428+ }
429+ return string(b[i:])
430+}
431+
432+// position converts an absolute index into the line and column of the
433+// JavaScript-style Position (line breaks are U+000A).
434+func (p *csParser) position(index int) Position {
435+ line, lineStart := 0, 0
436+ for i := 0; i < index && i < len(p.units); i++ {
437+ if p.units[i] == '\n' {
438+ line++
439+ lineStart = i + 1
440+ }
441+ }
442+ return Position{Line: line, Column: index - lineStart}
443+}
444+
445+func (p *csParser) token(t *csToken, typ TokenType) Token {
446+ end := t.index + len(utf16.Encode([]rune(t.raw)))
447+ return Token{Type: typ, Lexeme: t.raw, Range: Range{Start: p.position(t.index), End: p.position(end)}, Value: t.value}
448+}
449+
450+func (p *csParser) wellKnownOrExtension(name string) (FuncInfo, bool) {
451+ k := foldKey(name)
452+ if f, ok := builtinFuncs[k]; ok {
453+ return f, true
454+ }
455+ f, ok := p.funcs[k]
456+ return f, ok
457+}
458+
459+// parseRunner parses expression the way the runner does.
460+func parseRunner(expression string, contexts []string, funcs []FuncInfo) (Node, error) {
461+ p := &csParser{contexts: make(map[string]bool, len(contexts)), funcs: make(map[string]FuncInfo, len(funcs))}
462+ if exceedsUTF16Len(expression, MaxExpressionLength) {
463+ return nil, p.fail(ErrorExceededMaxLength, nil)
464+ }
465+ p.units = utf16.Encode([]rune(expression))
466+ p.expression = string(utf16.Decode(p.units))
467+ for _, c := range contexts {
468+ p.contexts[foldKey(c)] = true
469+ }
470+ for _, f := range funcs {
471+ p.funcs[foldKey(f.Name)] = f
472+ }
473+ p.lex = &csLexer{units: p.units}
474+ return p.createTree()
475+}
476+
477+func (p *csParser) createTree() (Node, error) {
478+ for {
479+ tok := p.lex.next()
480+ if tok == nil {
481+ break
482+ }
483+ p.cur = tok
484+ switch {
485+ case tok.kind == csUnexpected:
486+ return nil, p.fail(ErrorUnexpectedSymbol, tok)
487+ case tok.isOperator():
488+ if err := p.pushOperator(tok); err != nil {
489+ return nil, err
490+ }
491+ default:
492+ if err := p.pushOperand(tok); err != nil {
493+ return nil, err
494+ }
495+ }
496+ p.last = tok
497+ }
498+ if p.last == nil {
499+ return nil, nil
500+ }
501+ if len(p.operators) > 0 {
502+ unexpected := false
503+ switch p.last.kind {
504+ case csEndGroup, csEndIndex, csEndParameters:
505+ case csFunction:
506+ unexpected = true
507+ default:
508+ unexpected = p.last.isOperator()
509+ }
510+ if unexpected || len(p.lex.unclosed) > 0 {
511+ return nil, p.fail(ErrorUnexpectedEndOfExpression, p.last)
512+ }
513+ }
514+ for len(p.operators) > 0 {
515+ if err := p.flushTop(); err != nil {
516+ return nil, err
517+ }
518+ }
519+ if len(p.operands) != 1 {
520+ return nil, p.fail(ErrorUnexpectedSymbol, p.last)
521+ }
522+ root := p.operands[0]
523+ if exceedsDepth(root, 1) {
524+ return nil, p.fail(ErrorExceededMaxDepth, nil)
525+ }
526+ return root, nil
527+}
528+
529+func (p *csParser) pushOperand(tok *csToken) error {
530+ var n Node
531+ switch tok.kind {
532+ case csFunction:
533+ if _, ok := p.wellKnownOrExtension(tok.raw); !ok {
534+ return p.fail(ErrorUnrecognizedFunction, tok)
535+ }
536+ n = &FunctionCall{Name: p.token(tok, TokenIdentifier)}
537+ case csNamedValue:
538+ if !p.contexts[foldKey(tok.raw)] {
539+ return p.fail(ErrorUnrecognizedContext, tok)
540+ }
541+ n = &ContextAccess{Name: p.token(tok, TokenIdentifier)}
542+ case csNull:
543+ n = &Literal{Value: Null{}, Token: p.token(tok, TokenNull)}
544+ case csBoolean:
545+ typ := TokenFalse
546+ if tok.value == Boolean(true) {
547+ typ = TokenTrue
548+ }
549+ n = &Literal{Value: tok.value, Token: p.token(tok, typ)}
550+ case csNumber:
551+ n = &Literal{Value: tok.value, Token: p.token(tok, TokenNumber)}
552+ case csString:
553+ n = &Literal{Value: tok.value, Token: p.token(tok, TokenString)}
554+ case csPropertyName:
555+ n = &Literal{Value: String(tok.raw), Token: p.token(tok, TokenIdentifier)}
556+ case csWildcard:
557+ n = &Star{}
558+ default:
559+ return p.fail(ErrorUnexpectedSymbol, tok)
560+ }
561+ p.operands = append(p.operands, n)
562+ return nil
563+}
564+
565+func (p *csParser) pushOperator(tok *csToken) error {
566+ if tok.leftToRight() {
567+ prec := tok.precedence()
568+ for len(p.operators) > 0 {
569+ top := p.operators[len(p.operators)-1]
570+ if prec <= top.precedence() && top.kind != csStartGroup && top.kind != csStartIndex &&
571+ top.kind != csStartParameters && top.kind != csSeparator {
572+ if err := p.flushTop(); err != nil {
573+ return err
574+ }
575+ continue
576+ }
577+ break
578+ }
579+ }
580+ p.operators = append(p.operators, tok)
581+ switch tok.kind {
582+ case csEndGroup, csEndIndex, csEndParameters:
583+ return p.flushTop()
584+ }
585+ return nil
586+}
587+
588+func (p *csParser) popOperator(want csKind) (*csToken, error) {
589+ if len(p.operators) == 0 || p.operators[len(p.operators)-1].kind != want {
590+ return nil, p.fail(ErrorUnexpectedSymbol, p.cur)
591+ }
592+ t := p.operators[len(p.operators)-1]
593+ p.operators = p.operators[:len(p.operators)-1]
594+ return t, nil
595+}
596+
597+// popOperands pops n operands, returned in their written order.
598+func (p *csParser) popOperands(n int) ([]Node, error) {
599+ if len(p.operands) < n {
600+ return nil, p.fail(ErrorUnexpectedSymbol, p.cur)
601+ }
602+ out := append([]Node(nil), p.operands[len(p.operands)-n:]...)
603+ p.operands = p.operands[:len(p.operands)-n]
604+ return out, nil
605+}
606+
607+func (p *csParser) flushTop() error {
608+ top := p.operators[len(p.operators)-1]
609+ switch top.kind {
610+ case csEndIndex:
611+ if _, err := p.popOperator(csEndIndex); err != nil {
612+ return err
613+ }
614+ op, err := p.popOperator(csStartIndex)
615+ if err != nil {
616+ return err
617+ }
618+ args, err := p.popOperands(op.operandCount())
619+ if err != nil {
620+ return err
621+ }
622+ p.operands = append(p.operands, &IndexAccess{Expr: args[0], Index: args[1]})
623+ return nil
624+ case csEndGroup:
625+ if _, err := p.popOperator(csEndGroup); err != nil {
626+ return err
627+ }
628+ if _, err := p.popOperator(csStartGroup); err != nil {
629+ return err
630+ }
631+ if len(p.operands) == 0 {
632+ return p.fail(ErrorUnexpectedSymbol, p.cur)
633+ }
634+ g := &Grouping{Group: p.operands[len(p.operands)-1]}
635+ p.operands[len(p.operands)-1] = g
636+ return nil
637+ case csEndParameters:
638+ return p.flushEndParameters()
639+ }
640+ op, _ := p.popOperator(top.kind)
641+ args, err := p.popOperands(op.operandCount())
642+ if err != nil {
643+ return err
644+ }
645+ switch {
646+ case op.kind == csStartIndex || op.kind == csDereference:
647+ p.operands = append(p.operands, &IndexAccess{Expr: args[0], Index: args[1]})
648+ case op.kind == csLogicalOperator && op.raw == "!":
649+ p.operands = append(p.operands, &Unary{Operator: p.token(op, TokenBang), Expr: args[0]})
650+ case op.kind == csLogicalOperator && (op.raw == "&&" || op.raw == "||"):
651+ typ := TokenAnd
652+ if op.raw == "||" {
653+ typ = TokenOr
654+ }
655+ l := &Logical{Operator: p.token(op, typ)}
656+ for _, a := range args {
657+ // A nested chain of the same operator is merged, as the runner does.
658+ if nested, ok := a.(*Logical); ok && nested.Operator.Type == typ {
659+ l.Args = append(l.Args, nested.Args...)
660+ } else {
661+ l.Args = append(l.Args, a)
662+ }
663+ }
664+ p.operands = append(p.operands, l)
665+ case op.kind == csLogicalOperator:
666+ var typ TokenType
667+ switch op.raw {
668+ case "!=":
669+ typ = TokenBangEqual
670+ case "==":
671+ typ = TokenEqualEqual
672+ case ">":
673+ typ = TokenGreater
674+ case ">=":
675+ typ = TokenGreaterEqual
676+ case "<":
677+ typ = TokenLess
678+ default:
679+ typ = TokenLessEqual
680+ }
681+ p.operands = append(p.operands, &Binary{Left: args[0], Operator: p.token(op, typ), Right: args[1]})
682+ default:
683+ return p.fail(ErrorUnexpectedSymbol, op)
684+ }
685+ return nil
686+}
687+
688+func (p *csParser) flushEndParameters() error {
689+ end, err := p.popOperator(csEndParameters)
690+ if err != nil {
691+ return err
692+ }
693+ if end != p.cur || p.last == nil {
694+ return p.fail(ErrorUnexpectedSymbol, p.cur)
695+ }
696+ count := 0
697+ if p.last.kind != csStartParameters {
698+ count = 1
699+ for len(p.operators) > 0 && p.operators[len(p.operators)-1].kind == csSeparator {
700+ count++
701+ p.operators = p.operators[:len(p.operators)-1]
702+ }
703+ }
704+ args, err := p.popOperands(count)
705+ if err != nil {
706+ return err
707+ }
708+ if len(p.operands) == 0 {
709+ return p.fail(ErrorUnexpectedSymbol, p.cur)
710+ }
711+ call, ok := p.operands[len(p.operands)-1].(*FunctionCall)
712+ if !ok {
713+ return p.fail(ErrorUnexpectedSymbol, p.cur)
714+ }
715+ call.Args = args
716+ open, err := p.popOperator(csStartParameters)
717+ if err != nil {
718+ return err
719+ }
720+ info, _ := p.wellKnownOrExtension(call.Name.Lexeme)
721+ switch {
722+ case len(args) < info.MinArgs:
723+ return p.fail(ErrorTooFewParameters, open)
724+ case info.MaxArgs >= 0 && len(args) > info.MaxArgs:
725+ return p.fail(ErrorTooManyParameters, open)
726+ case strings.EqualFold(info.Name, "case") && len(args)%2 == 0:
727+ return p.fail(ErrorEvenParameters, open)
728+ }
729+ return nil
730+}
731+
732+// exceedsDepth reports whether the tree is deeper than MaxParserDepth, counted
733+// as the runner does (CheckMaxDepth): the root is at depth 1, the operands of
734+// an operator, the arguments of a call and the two parts of an index access
735+// are one deeper, parentheses add nothing, and a run of && or || is one node,
736+// also when parenthesized.
737+func exceedsDepth(n Node, depth int) bool {
738+ for {
739+ g, ok := n.(*Grouping)
740+ if !ok {
741+ break
742+ }
743+ n = g.Group
744+ }
745+ if depth > MaxParserDepth {
746+ return true
747+ }
748+ switch n := n.(type) {
749+ case *Unary:
750+ return exceedsDepth(n.Expr, depth+1)
751+ case *Binary:
752+ return exceedsDepth(n.Left, depth+1) || exceedsDepth(n.Right, depth+1)
753+ case *Logical:
754+ for _, a := range flatOperands(n, nil) {
755+ if exceedsDepth(a, depth+1) {
756+ return true
757+ }
758+ }
759+ case *FunctionCall:
760+ for _, a := range n.Args {
761+ if exceedsDepth(a, depth+1) {
762+ return true
763+ }
764+ }
765+ case *IndexAccess:
766+ return exceedsDepth(n.Expr, depth+1) || exceedsDepth(n.Index, depth+1)
767+ }
768+ return false
769+}
770+
771+// flatOperands lists the operands of a && or || chain with nested chains of
772+// the same operator (parenthesized or not) merged into it.
773+func flatOperands(l *Logical, out []Node) []Node {
774+ for _, a := range l.Args {
775+ inner := a
776+ for {
777+ g, ok := inner.(*Grouping)
778+ if !ok {
779+ break
780+ }
781+ inner = g.Group
782+ }
783+ if nested, ok := inner.(*Logical); ok && nested.Operator.Type == l.Operator.Type {
784+ out = flatOperands(nested, out)
785+ } else {
786+ out = append(out, a)
787+ }
788+ }
789+ return out
790+}
M expr/parser.go +4 -319
@@ -18,8 +18,8 @@ type FuncInfo struct {
1818 MaxArgs int
1919 }
2020
21-// builtinFuncs are the functions of the reference's wellKnownFunctions, keyed
22-// by lower-cased name. They take precedence over caller-supplied FuncInfo for
21+// builtinFuncs are the functions of ExpressionConstants.WellKnownFunctions, keyed
22+// by upper-cased name (see foldKey). They take precedence over caller-supplied FuncInfo for
2323 // validation.
2424 var builtinFuncs = map[string]FuncInfo{
2525 "CASE": {"case", 3, math.MaxInt},
@@ -43,239 +43,11 @@ var builtinFuncs = map[string]FuncInfo{
4343 // "" or " ", is not an error: the reference returns nothing for it, and Parse
4444 // returns a nil [Node], which [Evaluate] rejects.
4545 func Parse(expr string, contexts []string, funcs []FuncInfo) (Node, error) {
46- tokens, err := Lex(expr)
47- if err != nil {
48- return nil, err
49- }
50- p := &parser{
51- tokens: tokens,
52- contexts: make(map[string]bool, len(contexts)),
53- funcs: make(map[string]FuncInfo, len(funcs)),
54- }
55- for _, c := range contexts {
56- p.contexts[foldKey(c)] = true
57- }
58- for _, f := range funcs {
59- p.funcs[foldKey(f.Name)] = f
60- }
61- return p.parse()
62-}
63-
64-// parser is a port of Parser in parser.js: recursive descent, with the same
65-// precedence (lowest first): ||, &&, == !=, < <= > >=, !, then index and
66-// call.
67-type parser struct {
68- tokens []Token
69- offset int
70- depth int
71- contexts map[string]bool
72- funcs map[string]FuncInfo
73-}
74-
75-func (p *parser) parse() (Node, error) {
76- if p.atEnd() {
77- return nil, nil
78- }
79- n, err := p.expression()
80- if err != nil {
81- return nil, err
82- }
83- if !p.atEnd() {
84- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
85- }
86- return n, nil
87-}
88-
89-func (p *parser) expression() (Node, error) {
90- if err := p.incrDepth(); err != nil {
91- return nil, err
92- }
93- n, err := p.logicalOr()
94- p.depth--
95- return n, err
96-}
97-
98-func (p *parser) logicalOr() (Node, error) {
99- n, err := p.logicalAnd()
100- if err != nil || !p.check(TokenOr) {
101- return n, err
102- }
103- return p.logicalChain(TokenOr, n, p.logicalAnd)
104-}
105-
106-func (p *parser) logicalAnd() (Node, error) {
107- n, err := p.equality()
108- if err != nil || !p.check(TokenAnd) {
109- return n, err
110- }
111- return p.logicalChain(TokenAnd, n, p.equality)
112-}
113-
114-// logicalChain parses the rest of a flat chain of operator operands, the
115-// first of which is already parsed.
116-func (p *parser) logicalChain(op TokenType, first Node, operand func() (Node, error)) (Node, error) {
117- if err := p.incrDepth(); err != nil {
118- return nil, err
119- }
120- logical := &Logical{Operator: p.peek(), Args: []Node{first}}
121- for p.match(op) {
122- right, err := operand()
123- if err != nil {
124- return nil, err
125- }
126- logical.Args = append(logical.Args, right)
127- }
128- p.depth--
129- return logical, nil
130-}
131-
132-func (p *parser) equality() (Node, error) {
133- n, err := p.comparison()
134- if err != nil {
135- return nil, err
136- }
137- for p.match(TokenBangEqual, TokenEqualEqual) {
138- op := p.previous()
139- right, err := p.comparison()
140- if err != nil {
141- return nil, err
142- }
143- n = &Binary{Left: n, Operator: op, Right: right}
144- }
145- return n, nil
146-}
147-
148-func (p *parser) comparison() (Node, error) {
149- n, err := p.unary()
150- if err != nil {
151- return nil, err
152- }
153- for p.match(TokenGreater, TokenGreaterEqual, TokenLess, TokenLessEqual) {
154- op := p.previous()
155- right, err := p.unary()
156- if err != nil {
157- return nil, err
158- }
159- n = &Binary{Left: n, Operator: op, Right: right}
160- }
161- return n, nil
162-}
163-
164-func (p *parser) unary() (Node, error) {
165- if p.match(TokenBang) {
166- if err := p.incrDepth(); err != nil {
167- return nil, err
168- }
169- op := p.previous()
170- operand, err := p.unary()
171- if err != nil {
172- return nil, err
173- }
174- p.depth--
175- return &Unary{Operator: op, Expr: operand}, nil
176- }
177- return p.index()
178-}
179-
180-func (p *parser) index() (Node, error) {
181- n, err := p.call()
182- if err != nil {
183- return nil, err
184- }
185- depthIncreased := 0
186- switch n.(type) {
187- case *Grouping, *FunctionCall, *ContextAccess:
188- loop:
189- for {
190- switch {
191- case p.match(TokenLeftBracket):
192- var idx Node
193- if p.match(TokenStar) {
194- idx = &Star{}
195- } else if idx, err = p.expression(); err != nil {
196- return nil, err
197- }
198- if err := p.consume(TokenRightBracket, ErrorUnexpectedSymbol); err != nil {
199- return nil, err
200- }
201- if err := p.incrDepth(); err != nil {
202- return nil, err
203- }
204- depthIncreased++
205- n = &IndexAccess{Expr: n, Index: idx}
206- case p.match(TokenDot):
207- if err := p.incrDepth(); err != nil {
208- return nil, err
209- }
210- depthIncreased++
211- switch {
212- case p.match(TokenIdentifier):
213- prop := p.previous()
214- n = &IndexAccess{Expr: n, Index: &Literal{Value: String(prop.Lexeme), Token: prop}}
215- case p.match(TokenStar):
216- n = &IndexAccess{Expr: n, Index: &Star{}}
217- default:
218- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
219- }
220- default:
221- break loop
222- }
223- }
224- }
225- p.depth -= depthIncreased
226- return n, nil
227-}
228-
229-func (p *parser) call() (Node, error) {
230- if !p.check(TokenIdentifier) {
231- return p.primary()
232- }
233- ident := p.next()
234- if !p.match(TokenLeftParen) {
235- if !p.contexts[foldKey(ident.Lexeme)] {
236- return nil, newTokenError(ErrorUnrecognizedContext, ident)
237- }
238- return &ContextAccess{Name: ident}, nil
239- }
240- var args []Node
241- for !p.match(TokenRightParen) {
242- arg, err := p.expression()
243- if err != nil {
244- return nil, err
245- }
246- args = append(args, arg)
247- if !p.check(TokenRightParen) {
248- // A trailing comma before ")" is accepted, as in the reference.
249- if err := p.consume(TokenComma, ErrorUnexpectedSymbol); err != nil {
250- return nil, err
251- }
252- }
253- }
254- if err := p.validateFunction(ident, len(args)); err != nil {
255- return nil, err
256- }
257- return &FunctionCall{Name: ident, Args: args}, nil
258-}
259-
260-// validateFunction checks that the function exists and accepts argCount
261-// arguments (funcs.js validateFunction).
262-func (p *parser) validateFunction(ident Token, argCount int) error {
263- name := foldKey(ident.Lexeme)
264- f, ok := builtinFuncs[name]
265- if !ok {
266- f, ok = p.funcs[name]
267- if !ok {
268- return newTokenError(ErrorUnrecognizedFunction, ident)
269- }
270- }
271- if t, bad := arityError(name, f, argCount); bad {
272- return newTokenError(t, ident)
273- }
274- return nil
46+ return parseRunner(expr, contexts, funcs)
27547 }
27648
27749 // arityError reports whether argCount is wrong for f, and how. name is the
278-// lower-cased function name: case also needs an odd number of arguments.
50+// upper-cased function name (see foldKey): case also needs an odd number of arguments.
27951 func arityError(name string, f FuncInfo, argCount int) (ErrorType, bool) {
28052 switch {
28153 case argCount < f.MinArgs:
@@ -287,90 +59,3 @@ func arityError(name string, f FuncInfo, argCount int) (ErrorType, bool) {
28759 }
28860 return 0, false
28961 }
290-
291-func (p *parser) primary() (Node, error) {
292- switch {
293- case p.match(TokenFalse):
294- return &Literal{Value: Boolean(false), Token: p.previous()}, nil
295- case p.match(TokenTrue):
296- return &Literal{Value: Boolean(true), Token: p.previous()}, nil
297- case p.match(TokenNull):
298- return &Literal{Value: Null{}, Token: p.previous()}, nil
299- case p.match(TokenNumber):
300- return &Literal{Value: p.previous().Value, Token: p.previous()}, nil
301- case p.match(TokenString):
302- return &Literal{Value: p.previous().Value, Token: p.previous()}, nil
303- case p.match(TokenLeftParen):
304- n, err := p.expression()
305- if err != nil {
306- return nil, err
307- }
308- if p.atEnd() {
309- // Back up to the last token before the EOF.
310- return nil, newTokenError(ErrorUnexpectedEndOfExpression, p.previous())
311- }
312- if err := p.consume(TokenRightParen, ErrorUnexpectedSymbol); err != nil {
313- return nil, err
314- }
315- return &Grouping{Group: n}, nil
316- case p.atEnd():
317- return nil, newTokenError(ErrorUnexpectedEndOfExpression, p.previous())
318- }
319- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
320-}
321-
322-// match consumes the next token if it has one of the given types.
323-func (p *parser) match(types ...TokenType) bool {
324- for _, t := range types {
325- if p.check(t) {
326- p.next()
327- return true
328- }
329- }
330- return false
331-}
332-
333-func (p *parser) check(t TokenType) bool {
334- if p.atEnd() {
335- return false
336- }
337- return p.peek().Type == t
338-}
339-
340-func (p *parser) atEnd() bool { return p.peek().Type == TokenEOF }
341-
342-func (p *parser) next() Token {
343- if !p.atEnd() {
344- p.offset++
345- }
346- return p.previous()
347-}
348-
349-func (p *parser) peek() Token { return p.tokens[p.offset] }
350-
351-// previous returns the token before the current one. At offset 0 the
352-// reference reads tokens[-1], which is undefined and crashes; here it
353-// returns the zero Token. That is unreachable: previous is only used after
354-// a token was consumed.
355-func (p *parser) previous() Token {
356- if p.offset == 0 {
357- return Token{}
358- }
359- return p.tokens[p.offset-1]
360-}
361-
362-func (p *parser) consume(t TokenType, errType ErrorType) error {
363- if p.check(t) {
364- p.next()
365- return nil
366- }
367- return newTokenError(errType, p.peek())
368-}
369-
370-func (p *parser) incrDepth() error {
371- p.depth++
372- if p.depth > MaxParserDepth {
373- return newTokenError(ErrorExceededMaxDepth, p.peek())
374- }
375- return nil
376-}
@@ -18,8 +18,8 @@ type FuncInfo struct {
18 MaxArgs int18 MaxArgs int
19 }19 }
20 20
21-// builtinFuncs are the functions of the reference's wellKnownFunctions, keyed21+// builtinFuncs are the functions of ExpressionConstants.WellKnownFunctions, keyed
22-// by lower-cased name. They take precedence over caller-supplied FuncInfo for22+// by upper-cased name (see foldKey). They take precedence over caller-supplied FuncInfo for
23 // validation.23 // validation.
24 var builtinFuncs = map[string]FuncInfo{24 var builtinFuncs = map[string]FuncInfo{
25 "CASE": {"case", 3, math.MaxInt},25 "CASE": {"case", 3, math.MaxInt},
@@ -43,239 +43,11 @@ var builtinFuncs = map[string]FuncInfo{
43 // "" or " ", is not an error: the reference returns nothing for it, and Parse43 // "" or " ", is not an error: the reference returns nothing for it, and Parse
44 // returns a nil [Node], which [Evaluate] rejects.44 // returns a nil [Node], which [Evaluate] rejects.
45 func Parse(expr string, contexts []string, funcs []FuncInfo) (Node, error) {45 func Parse(expr string, contexts []string, funcs []FuncInfo) (Node, error) {
46- tokens, err := Lex(expr)46+ return parseRunner(expr, contexts, funcs)
47- if err != nil {
48- return nil, err
49- }
50- p := &parser{
51- tokens: tokens,
52- contexts: make(map[string]bool, len(contexts)),
53- funcs: make(map[string]FuncInfo, len(funcs)),
54- }
55- for _, c := range contexts {
56- p.contexts[foldKey(c)] = true
57- }
58- for _, f := range funcs {
59- p.funcs[foldKey(f.Name)] = f
60- }
61- return p.parse()
62-}
63-
64-// parser is a port of Parser in parser.js: recursive descent, with the same
65-// precedence (lowest first): ||, &&, == !=, < <= > >=, !, then index and
66-// call.
67-type parser struct {
68- tokens []Token
69- offset int
70- depth int
71- contexts map[string]bool
72- funcs map[string]FuncInfo
73-}
74-
75-func (p *parser) parse() (Node, error) {
76- if p.atEnd() {
77- return nil, nil
78- }
79- n, err := p.expression()
80- if err != nil {
81- return nil, err
82- }
83- if !p.atEnd() {
84- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
85- }
86- return n, nil
87-}
88-
89-func (p *parser) expression() (Node, error) {
90- if err := p.incrDepth(); err != nil {
91- return nil, err
92- }
93- n, err := p.logicalOr()
94- p.depth--
95- return n, err
96-}
97-
98-func (p *parser) logicalOr() (Node, error) {
99- n, err := p.logicalAnd()
100- if err != nil || !p.check(TokenOr) {
101- return n, err
102- }
103- return p.logicalChain(TokenOr, n, p.logicalAnd)
104-}
105-
106-func (p *parser) logicalAnd() (Node, error) {
107- n, err := p.equality()
108- if err != nil || !p.check(TokenAnd) {
109- return n, err
110- }
111- return p.logicalChain(TokenAnd, n, p.equality)
112-}
113-
114-// logicalChain parses the rest of a flat chain of operator operands, the
115-// first of which is already parsed.
116-func (p *parser) logicalChain(op TokenType, first Node, operand func() (Node, error)) (Node, error) {
117- if err := p.incrDepth(); err != nil {
118- return nil, err
119- }
120- logical := &Logical{Operator: p.peek(), Args: []Node{first}}
121- for p.match(op) {
122- right, err := operand()
123- if err != nil {
124- return nil, err
125- }
126- logical.Args = append(logical.Args, right)
127- }
128- p.depth--
129- return logical, nil
130-}
131-
132-func (p *parser) equality() (Node, error) {
133- n, err := p.comparison()
134- if err != nil {
135- return nil, err
136- }
137- for p.match(TokenBangEqual, TokenEqualEqual) {
138- op := p.previous()
139- right, err := p.comparison()
140- if err != nil {
141- return nil, err
142- }
143- n = &Binary{Left: n, Operator: op, Right: right}
144- }
145- return n, nil
146-}
147-
148-func (p *parser) comparison() (Node, error) {
149- n, err := p.unary()
150- if err != nil {
151- return nil, err
152- }
153- for p.match(TokenGreater, TokenGreaterEqual, TokenLess, TokenLessEqual) {
154- op := p.previous()
155- right, err := p.unary()
156- if err != nil {
157- return nil, err
158- }
159- n = &Binary{Left: n, Operator: op, Right: right}
160- }
161- return n, nil
162-}
163-
164-func (p *parser) unary() (Node, error) {
165- if p.match(TokenBang) {
166- if err := p.incrDepth(); err != nil {
167- return nil, err
168- }
169- op := p.previous()
170- operand, err := p.unary()
171- if err != nil {
172- return nil, err
173- }
174- p.depth--
175- return &Unary{Operator: op, Expr: operand}, nil
176- }
177- return p.index()
178-}
179-
180-func (p *parser) index() (Node, error) {
181- n, err := p.call()
182- if err != nil {
183- return nil, err
184- }
185- depthIncreased := 0
186- switch n.(type) {
187- case *Grouping, *FunctionCall, *ContextAccess:
188- loop:
189- for {
190- switch {
191- case p.match(TokenLeftBracket):
192- var idx Node
193- if p.match(TokenStar) {
194- idx = &Star{}
195- } else if idx, err = p.expression(); err != nil {
196- return nil, err
197- }
198- if err := p.consume(TokenRightBracket, ErrorUnexpectedSymbol); err != nil {
199- return nil, err
200- }
201- if err := p.incrDepth(); err != nil {
202- return nil, err
203- }
204- depthIncreased++
205- n = &IndexAccess{Expr: n, Index: idx}
206- case p.match(TokenDot):
207- if err := p.incrDepth(); err != nil {
208- return nil, err
209- }
210- depthIncreased++
211- switch {
212- case p.match(TokenIdentifier):
213- prop := p.previous()
214- n = &IndexAccess{Expr: n, Index: &Literal{Value: String(prop.Lexeme), Token: prop}}
215- case p.match(TokenStar):
216- n = &IndexAccess{Expr: n, Index: &Star{}}
217- default:
218- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
219- }
220- default:
221- break loop
222- }
223- }
224- }
225- p.depth -= depthIncreased
226- return n, nil
227-}
228-
229-func (p *parser) call() (Node, error) {
230- if !p.check(TokenIdentifier) {
231- return p.primary()
232- }
233- ident := p.next()
234- if !p.match(TokenLeftParen) {
235- if !p.contexts[foldKey(ident.Lexeme)] {
236- return nil, newTokenError(ErrorUnrecognizedContext, ident)
237- }
238- return &ContextAccess{Name: ident}, nil
239- }
240- var args []Node
241- for !p.match(TokenRightParen) {
242- arg, err := p.expression()
243- if err != nil {
244- return nil, err
245- }
246- args = append(args, arg)
247- if !p.check(TokenRightParen) {
248- // A trailing comma before ")" is accepted, as in the reference.
249- if err := p.consume(TokenComma, ErrorUnexpectedSymbol); err != nil {
250- return nil, err
251- }
252- }
253- }
254- if err := p.validateFunction(ident, len(args)); err != nil {
255- return nil, err
256- }
257- return &FunctionCall{Name: ident, Args: args}, nil
258-}
259-
260-// validateFunction checks that the function exists and accepts argCount
261-// arguments (funcs.js validateFunction).
262-func (p *parser) validateFunction(ident Token, argCount int) error {
263- name := foldKey(ident.Lexeme)
264- f, ok := builtinFuncs[name]
265- if !ok {
266- f, ok = p.funcs[name]
267- if !ok {
268- return newTokenError(ErrorUnrecognizedFunction, ident)
269- }
270- }
271- if t, bad := arityError(name, f, argCount); bad {
272- return newTokenError(t, ident)
273- }
274- return nil
275 }47 }
276 48
277 // arityError reports whether argCount is wrong for f, and how. name is the49 // arityError reports whether argCount is wrong for f, and how. name is the
278-// lower-cased function name: case also needs an odd number of arguments.50+// upper-cased function name (see foldKey): case also needs an odd number of arguments.
279 func arityError(name string, f FuncInfo, argCount int) (ErrorType, bool) {51 func arityError(name string, f FuncInfo, argCount int) (ErrorType, bool) {
280 switch {52 switch {
281 case argCount < f.MinArgs:53 case argCount < f.MinArgs:
@@ -287,90 +59,3 @@ func arityError(name string, f FuncInfo, argCount int) (ErrorType, bool) {
287 }59 }
288 return 0, false60 return 0, false
289 }61 }
290-
291-func (p *parser) primary() (Node, error) {
292- switch {
293- case p.match(TokenFalse):
294- return &Literal{Value: Boolean(false), Token: p.previous()}, nil
295- case p.match(TokenTrue):
296- return &Literal{Value: Boolean(true), Token: p.previous()}, nil
297- case p.match(TokenNull):
298- return &Literal{Value: Null{}, Token: p.previous()}, nil
299- case p.match(TokenNumber):
300- return &Literal{Value: p.previous().Value, Token: p.previous()}, nil
301- case p.match(TokenString):
302- return &Literal{Value: p.previous().Value, Token: p.previous()}, nil
303- case p.match(TokenLeftParen):
304- n, err := p.expression()
305- if err != nil {
306- return nil, err
307- }
308- if p.atEnd() {
309- // Back up to the last token before the EOF.
310- return nil, newTokenError(ErrorUnexpectedEndOfExpression, p.previous())
311- }
312- if err := p.consume(TokenRightParen, ErrorUnexpectedSymbol); err != nil {
313- return nil, err
314- }
315- return &Grouping{Group: n}, nil
316- case p.atEnd():
317- return nil, newTokenError(ErrorUnexpectedEndOfExpression, p.previous())
318- }
319- return nil, newTokenError(ErrorUnexpectedSymbol, p.peek())
320-}
321-
322-// match consumes the next token if it has one of the given types.
323-func (p *parser) match(types ...TokenType) bool {
324- for _, t := range types {
325- if p.check(t) {
326- p.next()
327- return true
328- }
329- }
330- return false
331-}
332-
333-func (p *parser) check(t TokenType) bool {
334- if p.atEnd() {
335- return false
336- }
337- return p.peek().Type == t
338-}
339-
340-func (p *parser) atEnd() bool { return p.peek().Type == TokenEOF }
341-
342-func (p *parser) next() Token {
343- if !p.atEnd() {
344- p.offset++
345- }
346- return p.previous()
347-}
348-
349-func (p *parser) peek() Token { return p.tokens[p.offset] }
350-
351-// previous returns the token before the current one. At offset 0 the
352-// reference reads tokens[-1], which is undefined and crashes; here it
353-// returns the zero Token. That is unreachable: previous is only used after
354-// a token was consumed.
355-func (p *parser) previous() Token {
356- if p.offset == 0 {
357- return Token{}
358- }
359- return p.tokens[p.offset-1]
360-}
361-
362-func (p *parser) consume(t TokenType, errType ErrorType) error {
363- if p.check(t) {
364- p.next()
365- return nil
366- }
367- return newTokenError(errType, p.peek())
368-}
369-
370-func (p *parser) incrDepth() error {
371- p.depth++
372- if p.depth > MaxParserDepth {
373- return newTokenError(ErrorExceededMaxDepth, p.peek())
374- }
375- return nil
376-}
D expr/runner_oracle_wip_test.go +0 -66
@@ -1,66 +0,0 @@
1-//go:build wip
2-
3-package expr
4-
5-import (
6- "encoding/json"
7- "os"
8- "sort"
9- "testing"
10-)
11-
12-// runnerResidual lists, by reason, the corpus expressions on which this package
13-// knowingly differs from the runner. It is filled by classifyResidual.
14-func classifyResidual(g, r runnerCase) string {
15- return ""
16-}
17-
18-func TestRunnerOracle(t *testing.T) {
19- of, ctxValues, ctxNames := loadOracle(t)
20- rf := loadRunner(t)
21- if len(rf.Cases) != len(of.Cases) {
22- t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
23- }
24- t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
25- ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
26- residual := map[string]int{}
27- failures := 0
28- for i, rc := range rf.Cases {
29- if rc.E != of.Cases[i].E {
30- t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
31- }
32- g := goAnswer(rc.E, ctxNames, ec)
33- if sameAnswer(g, rc) {
34- continue
35- }
36- if reason := classifyResidual(g, rc); reason != "" {
37- residual[reason]++
38- continue
39- }
40- failures++
41- if p := os.Getenv("WIP_OUT"); p != "" {
42- f, _ := os.OpenFile(p, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
43- b1, _ := json.Marshal(g)
44- b2, _ := json.Marshal(rc)
45- _, _ = f.WriteString(string(b1) + "\n" + string(b2) + "\n")
46- _ = f.Close()
47- }
48- if failures <= 30 {
49- gj, _ := json.Marshal(g)
50- rj, _ := json.Marshal(rc)
51- t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
52- }
53- }
54- reasons := make([]string, 0, len(residual))
55- for k := range residual {
56- reasons = append(reasons, k)
57- }
58- sort.Strings(reasons)
59- for _, k := range reasons {
60- t.Logf("known difference from the runner (%d cases): %s", residual[k], k)
61- }
62- t.Logf("%d cases, %d differences from the runner", len(rf.Cases), failures)
63- if failures > 0 {
64- t.Fatalf("%d mismatches with the runner", failures)
65- }
66-}
@@ -1,66 +0,0 @@
1-//go:build wip
2-
3-package expr
4-
5-import (
6- "encoding/json"
7- "os"
8- "sort"
9- "testing"
10-)
11-
12-// runnerResidual lists, by reason, the corpus expressions on which this package
13-// knowingly differs from the runner. It is filled by classifyResidual.
14-func classifyResidual(g, r runnerCase) string {
15- return ""
16-}
17-
18-func TestRunnerOracle(t *testing.T) {
19- of, ctxValues, ctxNames := loadOracle(t)
20- rf := loadRunner(t)
21- if len(rf.Cases) != len(of.Cases) {
22- t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
23- }
24- t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
25- ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
26- residual := map[string]int{}
27- failures := 0
28- for i, rc := range rf.Cases {
29- if rc.E != of.Cases[i].E {
30- t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
31- }
32- g := goAnswer(rc.E, ctxNames, ec)
33- if sameAnswer(g, rc) {
34- continue
35- }
36- if reason := classifyResidual(g, rc); reason != "" {
37- residual[reason]++
38- continue
39- }
40- failures++
41- if p := os.Getenv("WIP_OUT"); p != "" {
42- f, _ := os.OpenFile(p, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
43- b1, _ := json.Marshal(g)
44- b2, _ := json.Marshal(rc)
45- _, _ = f.WriteString(string(b1) + "\n" + string(b2) + "\n")
46- _ = f.Close()
47- }
48- if failures <= 30 {
49- gj, _ := json.Marshal(g)
50- rj, _ := json.Marshal(rc)
51- t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
52- }
53- }
54- reasons := make([]string, 0, len(residual))
55- for k := range residual {
56- reasons = append(reasons, k)
57- }
58- sort.Strings(reasons)
59- for _, k := range reasons {
60- t.Logf("known difference from the runner (%d cases): %s", residual[k], k)
61- }
62- t.Logf("%d cases, %d differences from the runner", len(rf.Cases), failures)
63- if failures > 0 {
64- t.Fatalf("%d mismatches with the runner", failures)
65- }
66-}
A expr/runner_semantics_test.go +115 -0
@@ -0,0 +1,115 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "context"
7+ "testing"
8+)
9+
10+// TestRunnerSemantics pins behaviours of the PORTING.md list that the C# runner
11+// and the JavaScript reference share or differ on. Every expectation was
12+// obtained by running the expression through the runner's own code
13+// (gen/runner); the bulk is also covered by TestRunnerOracle.
14+func TestRunnerSemantics(t *testing.T) {
15+ ctx := map[string]Value{"github": mustJSON(t, `{
16+ "commits": [{"id": "1"}, {"id": "2", "x": 1}, {"y": 3}],
17+ "obj": {"a": {"a": 1}, "K": "v", "b": {"c": 2}},
18+ "nested": [[1, 2], [3], []]}`)}
19+ tests := []struct{ src, want string }{
20+ // Index access: strings and numbers index arrays by their numeric value,
21+ // booleans and null become 1 and 0, negative, NaN and infinite give null.
22+ {"github.commits['1'].id", "String:2"},
23+ {"github.commits[' 1 '].id", "String:2"},
24+ {"github.commits['1.9'].id", "String:2"},
25+ {"github.commits[1.9].id", "String:2"},
26+ {"github.commits[true].id", "String:2"},
27+ {"github.commits[null].id", "String:1"},
28+ {"github.commits[-1]", "Null:"},
29+ {"github.commits[NaN]", "Null:"},
30+ {"github.commits[Infinity]", "Null:"},
31+ {"github.commits['x']", "Null:"},
32+ {"github.commits[5].id", "Null:"},
33+ // On a dictionary only the string form counts, case-insensitively.
34+ {"github.obj[true]", "Null:"},
35+ {"github.obj[null]", "Null:"},
36+ {"github.obj[1]", "Null:"},
37+ {"github.obj['K']", "String:v"},
38+ {"GITHUB.OBJ.K", "String:v"},
39+ {"github.OBJ.k", "String:v"},
40+ {"github.nope.deeper", "Null:"},
41+ // Filters.
42+ {"toJSON(github.obj.*.a)", "String:[\n 1\n]"},
43+ {"toJSON(github.commits.*.id)", "String:[\n \"1\",\n \"2\"\n]"},
44+ {"toJSON(github.commits.*.*)", "String:[\n \"1\",\n \"2\",\n 1,\n 3\n]"},
45+ {"toJSON(github.nested.*.*)", "String:[\n 1,\n 2,\n 3\n]"},
46+ {"toJSON(github.nested.*[0])", "String:[\n 1,\n 3\n]"},
47+ {"github.commits.* == github.commits.*", "Boolean:false"},
48+ {"github.obj.* == github.obj.*", "Boolean:false"},
49+ {"github.commits == github.commits", "Boolean:true"},
50+ {"github.obj == github.obj", "Boolean:true"},
51+ // Comparison.
52+ {"null < 1", "Boolean:true"},
53+ {"null > -1", "Boolean:true"},
54+ {"null == 0", "Boolean:true"},
55+ {"null < true", "Boolean:true"},
56+ {"null < ''", "Boolean:false"},
57+ {"null == ''", "Boolean:true"},
58+ {"NaN == NaN", "Boolean:false"},
59+ {"NaN != NaN", "Boolean:true"},
60+ {"NaN < 1", "Boolean:false"},
61+ {"true > false", "Boolean:true"},
62+ {"true > 0", "Boolean:true"},
63+ // && and || return an operand.
64+ {"'' || 'x'", "String:x"},
65+ {"'a' && 'b'", "String:b"},
66+ {"0 && 'x'", "Number:0"},
67+ {"null || false", "Boolean:false"},
68+ // White space may surround "." and "[" and sit between a function name
69+ // and its "(" (the lexer looks ahead past it).
70+ {"github . obj . K", "String:v"},
71+ {"github.obj.\nK", "String:v"},
72+ {"github.obj .K", "String:v"},
73+ {"github [ 'obj' ].K", "String:v"},
74+ {"toJSON (1)", "String:1"},
75+ // White space is Char.IsWhiteSpace: U+00A0, U+000B and U+0085 are skipped.
76+ {"1\u00a0", "Number:1"},
77+ {"\u00a01", "Number:1"},
78+ {"\v1", "Number:1"},
79+ {"'a'\u0085", "String:a"},
80+ }
81+ for _, tc := range tests {
82+ node, err := Parse(tc.src, []string{"github"}, nil)
83+ if err != nil {
84+ t.Errorf("Parse(%q): %v", tc.src, err)
85+ continue
86+ }
87+ v, err := Evaluate(context.Background(), node, EvalContext{Contexts: ctx})
88+ if err != nil {
89+ t.Errorf("Evaluate(%q): %v", tc.src, err)
90+ continue
91+ }
92+ if got := v.Kind().String() + ":" + CoerceString(v); got != tc.want {
93+ t.Errorf("%s = %q, want %q", tc.src, got, tc.want)
94+ }
95+ }
96+
97+ // Parse errors.
98+ for _, tc := range []struct{ src, want string }{
99+ {"contains('a','b',)", "Unexpected symbol: ')'. Located at position 18 within expression: contains('a','b',)"},
100+ {"contains(,'a')", "Unexpected symbol: ','. Located at position 10 within expression: contains(,'a')"},
101+ {"[1] == [1]", "Unexpected symbol: '['. Located at position 1 within expression: [1] == [1]"},
102+ {"1 \u00a0 2", "Unexpected symbol: '2'. Located at position 5 within expression: 1 \u00a0 2"},
103+ } {
104+ if _, err := Parse(tc.src, []string{"github"}, nil); err == nil || err.Error() != tc.want {
105+ t.Errorf("Parse(%q) = %v, want %q", tc.src, err, tc.want)
106+ }
107+ }
108+
109+ // An empty expression is not an error and has no tree.
110+ for _, src := range []string{"", " ", "\n", "\u00a0"} {
111+ if n, err := Parse(src, nil, nil); n != nil || err != nil {
112+ t.Errorf("Parse(%q) = %v, %v; want nil, nil", src, n, err)
113+ }
114+ }
115+}
@@ -0,0 +1,115 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "context"
7+ "testing"
8+)
9+
10+// TestRunnerSemantics pins behaviours of the PORTING.md list that the C# runner
11+// and the JavaScript reference share or differ on. Every expectation was
12+// obtained by running the expression through the runner's own code
13+// (gen/runner); the bulk is also covered by TestRunnerOracle.
14+func TestRunnerSemantics(t *testing.T) {
15+ ctx := map[string]Value{"github": mustJSON(t, `{
16+ "commits": [{"id": "1"}, {"id": "2", "x": 1}, {"y": 3}],
17+ "obj": {"a": {"a": 1}, "K": "v", "b": {"c": 2}},
18+ "nested": [[1, 2], [3], []]}`)}
19+ tests := []struct{ src, want string }{
20+ // Index access: strings and numbers index arrays by their numeric value,
21+ // booleans and null become 1 and 0, negative, NaN and infinite give null.
22+ {"github.commits['1'].id", "String:2"},
23+ {"github.commits[' 1 '].id", "String:2"},
24+ {"github.commits['1.9'].id", "String:2"},
25+ {"github.commits[1.9].id", "String:2"},
26+ {"github.commits[true].id", "String:2"},
27+ {"github.commits[null].id", "String:1"},
28+ {"github.commits[-1]", "Null:"},
29+ {"github.commits[NaN]", "Null:"},
30+ {"github.commits[Infinity]", "Null:"},
31+ {"github.commits['x']", "Null:"},
32+ {"github.commits[5].id", "Null:"},
33+ // On a dictionary only the string form counts, case-insensitively.
34+ {"github.obj[true]", "Null:"},
35+ {"github.obj[null]", "Null:"},
36+ {"github.obj[1]", "Null:"},
37+ {"github.obj['K']", "String:v"},
38+ {"GITHUB.OBJ.K", "String:v"},
39+ {"github.OBJ.k", "String:v"},
40+ {"github.nope.deeper", "Null:"},
41+ // Filters.
42+ {"toJSON(github.obj.*.a)", "String:[\n 1\n]"},
43+ {"toJSON(github.commits.*.id)", "String:[\n \"1\",\n \"2\"\n]"},
44+ {"toJSON(github.commits.*.*)", "String:[\n \"1\",\n \"2\",\n 1,\n 3\n]"},
45+ {"toJSON(github.nested.*.*)", "String:[\n 1,\n 2,\n 3\n]"},
46+ {"toJSON(github.nested.*[0])", "String:[\n 1,\n 3\n]"},
47+ {"github.commits.* == github.commits.*", "Boolean:false"},
48+ {"github.obj.* == github.obj.*", "Boolean:false"},
49+ {"github.commits == github.commits", "Boolean:true"},
50+ {"github.obj == github.obj", "Boolean:true"},
51+ // Comparison.
52+ {"null < 1", "Boolean:true"},
53+ {"null > -1", "Boolean:true"},
54+ {"null == 0", "Boolean:true"},
55+ {"null < true", "Boolean:true"},
56+ {"null < ''", "Boolean:false"},
57+ {"null == ''", "Boolean:true"},
58+ {"NaN == NaN", "Boolean:false"},
59+ {"NaN != NaN", "Boolean:true"},
60+ {"NaN < 1", "Boolean:false"},
61+ {"true > false", "Boolean:true"},
62+ {"true > 0", "Boolean:true"},
63+ // && and || return an operand.
64+ {"'' || 'x'", "String:x"},
65+ {"'a' && 'b'", "String:b"},
66+ {"0 && 'x'", "Number:0"},
67+ {"null || false", "Boolean:false"},
68+ // White space may surround "." and "[" and sit between a function name
69+ // and its "(" (the lexer looks ahead past it).
70+ {"github . obj . K", "String:v"},
71+ {"github.obj.\nK", "String:v"},
72+ {"github.obj .K", "String:v"},
73+ {"github [ 'obj' ].K", "String:v"},
74+ {"toJSON (1)", "String:1"},
75+ // White space is Char.IsWhiteSpace: U+00A0, U+000B and U+0085 are skipped.
76+ {"1\u00a0", "Number:1"},
77+ {"\u00a01", "Number:1"},
78+ {"\v1", "Number:1"},
79+ {"'a'\u0085", "String:a"},
80+ }
81+ for _, tc := range tests {
82+ node, err := Parse(tc.src, []string{"github"}, nil)
83+ if err != nil {
84+ t.Errorf("Parse(%q): %v", tc.src, err)
85+ continue
86+ }
87+ v, err := Evaluate(context.Background(), node, EvalContext{Contexts: ctx})
88+ if err != nil {
89+ t.Errorf("Evaluate(%q): %v", tc.src, err)
90+ continue
91+ }
92+ if got := v.Kind().String() + ":" + CoerceString(v); got != tc.want {
93+ t.Errorf("%s = %q, want %q", tc.src, got, tc.want)
94+ }
95+ }
96+
97+ // Parse errors.
98+ for _, tc := range []struct{ src, want string }{
99+ {"contains('a','b',)", "Unexpected symbol: ')'. Located at position 18 within expression: contains('a','b',)"},
100+ {"contains(,'a')", "Unexpected symbol: ','. Located at position 10 within expression: contains(,'a')"},
101+ {"[1] == [1]", "Unexpected symbol: '['. Located at position 1 within expression: [1] == [1]"},
102+ {"1 \u00a0 2", "Unexpected symbol: '2'. Located at position 5 within expression: 1 \u00a0 2"},
103+ } {
104+ if _, err := Parse(tc.src, []string{"github"}, nil); err == nil || err.Error() != tc.want {
105+ t.Errorf("Parse(%q) = %v, want %q", tc.src, err, tc.want)
106+ }
107+ }
108+
109+ // An empty expression is not an error and has no tree.
110+ for _, src := range []string{"", " ", "\n", "\u00a0"} {
111+ if n, err := Parse(src, nil, nil); n != nil || err != nil {
112+ t.Errorf("Parse(%q) = %v, %v; want nil, nil", src, n, err)
113+ }
114+ }
115+}
A expr/runner_strict_test.go +69 -0
@@ -0,0 +1,69 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "encoding/json"
7+ "sort"
8+ "testing"
9+)
10+
11+// knownRunnerDifference classifies the answers of the corpus on which this
12+// package knowingly differs from the runner. It returns the reason, or "".
13+func knownRunnerDifference(g, r runnerCase) string {
14+ switch {
15+ case r.E == "-0" || r.E == "toJSON(-0)":
16+ // .NET Core formats negative zero as "-0". The runner's newer copy
17+ // (src/Sdk/Expressions/EvaluationResult.cs) maps it back to "0" and the
18+ // shared fixtures (coerce_string.json) say "0", so "0" is kept.
19+ return "negative zero is written 0"
20+ }
21+ return ""
22+}
23+
24+// TestRunnerOracle holds the package to the answers of the runner's own code on
25+// the 34 765 expressions of the oracle corpus: same result kind, string form
26+// and toJSON text, same error phase and kind (the error texts of the runner's
27+// Json.NET are not reproduced).
28+func TestRunnerOracle(t *testing.T) {
29+ of, ctxValues, ctxNames := loadOracle(t)
30+ rf := loadRunner(t)
31+ if len(rf.Cases) != len(of.Cases) {
32+ t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
33+ }
34+ t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
35+ ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
36+ known := map[string]int{}
37+ failures := 0
38+ for i, rc := range rf.Cases {
39+ if rc.E != of.Cases[i].E {
40+ t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
41+ }
42+ g := goAnswer(rc.E, ctxNames, ec)
43+ if sameAnswer(g, rc) {
44+ continue
45+ }
46+ if reason := knownRunnerDifference(g, rc); reason != "" {
47+ known[reason]++
48+ continue
49+ }
50+ failures++
51+ if failures <= 30 {
52+ gj, _ := json.Marshal(g)
53+ rj, _ := json.Marshal(rc)
54+ t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
55+ }
56+ }
57+ reasons := make([]string, 0, len(known))
58+ for k := range known {
59+ reasons = append(reasons, k)
60+ }
61+ sort.Strings(reasons)
62+ for _, k := range reasons {
63+ t.Logf("known difference from the runner (%d cases): %s", known[k], k)
64+ }
65+ t.Logf("%d cases, %d failures", len(rf.Cases), failures)
66+ if failures > 0 {
67+ t.Fatalf("%d mismatches with the runner", failures)
68+ }
69+}
@@ -0,0 +1,69 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "encoding/json"
7+ "sort"
8+ "testing"
9+)
10+
11+// knownRunnerDifference classifies the answers of the corpus on which this
12+// package knowingly differs from the runner. It returns the reason, or "".
13+func knownRunnerDifference(g, r runnerCase) string {
14+ switch {
15+ case r.E == "-0" || r.E == "toJSON(-0)":
16+ // .NET Core formats negative zero as "-0". The runner's newer copy
17+ // (src/Sdk/Expressions/EvaluationResult.cs) maps it back to "0" and the
18+ // shared fixtures (coerce_string.json) say "0", so "0" is kept.
19+ return "negative zero is written 0"
20+ }
21+ return ""
22+}
23+
24+// TestRunnerOracle holds the package to the answers of the runner's own code on
25+// the 34 765 expressions of the oracle corpus: same result kind, string form
26+// and toJSON text, same error phase and kind (the error texts of the runner's
27+// Json.NET are not reproduced).
28+func TestRunnerOracle(t *testing.T) {
29+ of, ctxValues, ctxNames := loadOracle(t)
30+ rf := loadRunner(t)
31+ if len(rf.Cases) != len(of.Cases) {
32+ t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
33+ }
34+ t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
35+ ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
36+ known := map[string]int{}
37+ failures := 0
38+ for i, rc := range rf.Cases {
39+ if rc.E != of.Cases[i].E {
40+ t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
41+ }
42+ g := goAnswer(rc.E, ctxNames, ec)
43+ if sameAnswer(g, rc) {
44+ continue
45+ }
46+ if reason := knownRunnerDifference(g, rc); reason != "" {
47+ known[reason]++
48+ continue
49+ }
50+ failures++
51+ if failures <= 30 {
52+ gj, _ := json.Marshal(g)
53+ rj, _ := json.Marshal(rc)
54+ t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
55+ }
56+ }
57+ reasons := make([]string, 0, len(known))
58+ for k := range known {
59+ reasons = append(reasons, k)
60+ }
61+ sort.Strings(reasons)
62+ for _, k := range reasons {
63+ t.Logf("known difference from the runner (%d cases): %s", known[k], k)
64+ }
65+ t.Logf("%d cases, %d failures", len(rf.Cases), failures)
66+ if failures > 0 {
67+ t.Fatalf("%d mismatches with the runner", failures)
68+ }
69+}
M expr/testutil_test.go +5 -1
@@ -4,6 +4,7 @@ package expr
44
55 import (
66 "context"
7+ "strings"
78 "testing"
89 )
910
@@ -52,7 +53,10 @@ func evalString(tb testing.TB, src string) string {
5253 tb.Helper()
5354 node, err := Parse(src, testContextNames, nil)
5455 if err != nil {
55- return "PARSE " + err.Error()
56+ // The runner appends ". Located at position N within expression: ..."
57+ // to errors that name a token; TestRunnerErrorPositions covers that.
58+ msg, _, _ := strings.Cut(err.Error(), ". Located at position ")
59+ return "PARSE " + msg
5660 }
5761 v, err := Evaluate(context.Background(), node, EvalContext{Contexts: testContexts(tb)})
5862 if err != nil {
@@ -4,6 +4,7 @@ package expr
4 4
5 import (5 import (
6 "context"6 "context"
7+ "strings"
7 "testing"8 "testing"
8 )9 )
9 10
@@ -52,7 +53,10 @@ func evalString(tb testing.TB, src string) string {
52 tb.Helper()53 tb.Helper()
53 node, err := Parse(src, testContextNames, nil)54 node, err := Parse(src, testContextNames, nil)
54 if err != nil {55 if err != nil {
55- return "PARSE " + err.Error()56+ // The runner appends ". Located at position N within expression: ..."
57+ // to errors that name a token; TestRunnerErrorPositions covers that.
58+ msg, _, _ := strings.Cut(err.Error(), ". Located at position ")
59+ return "PARSE " + msg
56 }60 }
57 v, err := Evaluate(context.Background(), node, EvalContext{Contexts: testContexts(tb)})61 v, err := Evaluate(context.Background(), node, EvalContext{Contexts: testContexts(tb)})
58 if err != nil {62 if err != nil {
M expr/xlang_test.go +8 -6
@@ -59,11 +59,8 @@ type xlangCase struct {
5959 // xlangKnownDifferences maps "file: group (expr prefix)" keys to the reason
6060 // the fixture is not satisfied. Keep it empty if possible.
6161 var xlangKnownDifferences = map[string]string{
62- "syntax-errors: memory-errors#0": "memory accounting differs from the runner's EvaluationMemory",
63- "syntax-errors: parsing-errors#25": "position is the column in the line, the runner reports the index in the expression",
64- "syntax-errors: parsing-errors#26": "position is the column in the line, the runner reports the index in the expression",
65- "syntax-errors: parsing-errors#27": "position is the column in the line, the runner reports the index in the expression",
66- "syntax-errors: parsing-errors#28": "position is the column in the line, the runner reports the index in the expression",
62+ "case: case#27": "the runner reports an even number of case() arguments on the '(' token (position 5), not on the name: ExpressionParser.FlushTopEndParameters passes the popped StartParameters token; checked by running the runner's code",
63+ "syntax-errors: memory-errors#0": "memory accounting differs from the runner's EvaluationMemory",
6764 }
6865
6966 // xlangKind is the kind name the fixtures use.
@@ -178,7 +175,12 @@ func runXlangCase(tb testing.TB, c xlangCase) string {
178175 if !errors.As(err, &pe) {
179176 return fmt.Sprintf("parse error has type %T", err)
180177 }
181- full := fmt.Sprintf("%s. Located at position %d within expression: %s", pe.Message, pe.Pos.Column+1, c.Expr)
178+ // The TypeScript runner builds the suffix from a column; this
179+ // package, like the C# runner, already has it in the message.
180+ full := pe.Message
181+ if !strings.Contains(full, ". Located at position ") {
182+ full = fmt.Sprintf("%s. Located at position %d within expression: %s", pe.Message, pe.Pos.Column+1, c.Expr)
183+ }
182184 if strings.Contains(full, c.Err.Value) {
183185 return ""
184186 }
@@ -59,11 +59,8 @@ type xlangCase struct {
59 // xlangKnownDifferences maps "file: group (expr prefix)" keys to the reason59 // xlangKnownDifferences maps "file: group (expr prefix)" keys to the reason
60 // the fixture is not satisfied. Keep it empty if possible.60 // the fixture is not satisfied. Keep it empty if possible.
61 var xlangKnownDifferences = map[string]string{61 var xlangKnownDifferences = map[string]string{
62- "syntax-errors: memory-errors#0": "memory accounting differs from the runner's EvaluationMemory",62+ "case: case#27": "the runner reports an even number of case() arguments on the '(' token (position 5), not on the name: ExpressionParser.FlushTopEndParameters passes the popped StartParameters token; checked by running the runner's code",
63- "syntax-errors: parsing-errors#25": "position is the column in the line, the runner reports the index in the expression",63+ "syntax-errors: memory-errors#0": "memory accounting differs from the runner's EvaluationMemory",
64- "syntax-errors: parsing-errors#26": "position is the column in the line, the runner reports the index in the expression",
65- "syntax-errors: parsing-errors#27": "position is the column in the line, the runner reports the index in the expression",
66- "syntax-errors: parsing-errors#28": "position is the column in the line, the runner reports the index in the expression",
67 }64 }
68 65
69 // xlangKind is the kind name the fixtures use.66 // xlangKind is the kind name the fixtures use.
@@ -178,7 +175,12 @@ func runXlangCase(tb testing.TB, c xlangCase) string {
178 if !errors.As(err, &pe) {175 if !errors.As(err, &pe) {
179 return fmt.Sprintf("parse error has type %T", err)176 return fmt.Sprintf("parse error has type %T", err)
180 }177 }
181- full := fmt.Sprintf("%s. Located at position %d within expression: %s", pe.Message, pe.Pos.Column+1, c.Expr)178+ // The TypeScript runner builds the suffix from a column; this
179+ // package, like the C# runner, already has it in the message.
180+ full := pe.Message
181+ if !strings.Contains(full, ". Located at position ") {
182+ full = fmt.Sprintf("%s. Located at position %d within expression: %s", pe.Message, pe.Pos.Column+1, c.Expr)
183+ }
182 if strings.Contains(full, c.Err.Value) {184 if strings.Contains(full, c.Err.Value) {
183 return ""185 return ""
184 }186 }
M template/read_test.go +5 -4
@@ -22,17 +22,18 @@ const head = "on: push\njobs:\n a:\n runs-on: x\n steps:\n"
2222
2323 func TestReadErrors(t *testing.T) {
2424 // Expected messages come from @actions/workflow-parser 0.3.61 run on the
25- // same inputs.
25+ // same inputs, except that expression errors carry the position suffix of
26+ // the C# runner (package expr follows it).
2627 tests := []struct {
2728 name, src string
2829 want []string
2930 }{
3031 {"unknown context, block scalar", head + " - run: |\n echo hi\n echo ${{ foo.bar }}\n",
31- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},
32+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
3233 {"unknown context, plain", head + " - run: echo ${{ foo.bar }} abc\n",
33- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},
34+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
3435 {"unknown context, double quoted", head + " - run: \"echo ${{ foo.bar }} abc\"\n",
35- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},
36+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
3637 {"bad expression and insert in a name", head + " - run: echo ${{ 1 + }}\n - name: ${{ insert }}\n run: x\n",
3738 []string{"a.yml (Line: 6, Col: 14): Unexpected symbol: '+'. Located at position 3 within expression: 1 +", "a.yml (Line: 7, Col: 15): Unexpected value '${{ insert }}'"}},
3839 {"not closed", head + " - run: echo ${{ 1\n",
@@ -22,17 +22,18 @@ const head = "on: push\njobs:\n a:\n runs-on: x\n steps:\n"
22 22
23 func TestReadErrors(t *testing.T) {23 func TestReadErrors(t *testing.T) {
24 // Expected messages come from @actions/workflow-parser 0.3.61 run on the24 // Expected messages come from @actions/workflow-parser 0.3.61 run on the
25- // same inputs.25+ // same inputs, except that expression errors carry the position suffix of
26+ // the C# runner (package expr follows it).
26 tests := []struct {27 tests := []struct {
27 name, src string28 name, src string
28 want []string29 want []string
29 }{30 }{
30 {"unknown context, block scalar", head + " - run: |\n echo hi\n echo ${{ foo.bar }}\n",31 {"unknown context, block scalar", head + " - run: |\n echo hi\n echo ${{ foo.bar }}\n",
31- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},32+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
32 {"unknown context, plain", head + " - run: echo ${{ foo.bar }} abc\n",33 {"unknown context, plain", head + " - run: echo ${{ foo.bar }} abc\n",
33- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},34+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
34 {"unknown context, double quoted", head + " - run: \"echo ${{ foo.bar }} abc\"\n",35 {"unknown context, double quoted", head + " - run: \"echo ${{ foo.bar }} abc\"\n",
35- []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'"}},36+ []string{"a.yml (Line: 6, Col: 14): Unrecognized named-value: 'foo'. Located at position 1 within expression: foo.bar"}},
36 {"bad expression and insert in a name", head + " - run: echo ${{ 1 + }}\n - name: ${{ insert }}\n run: x\n",37 {"bad expression and insert in a name", head + " - run: echo ${{ 1 + }}\n - name: ${{ insert }}\n run: x\n",
37 []string{"a.yml (Line: 6, Col: 14): Unexpected symbol: '+'. Located at position 3 within expression: 1 +", "a.yml (Line: 7, Col: 15): Unexpected value '${{ insert }}'"}},38 []string{"a.yml (Line: 6, Col: 14): Unexpected symbol: '+'. Located at position 3 within expression: 1 +", "a.yml (Line: 7, Col: 15): Unexpected value '${{ insert }}'"}},
38 {"not closed", head + " - run: echo ${{ 1\n",39 {"not closed", head + " - run: echo ${{ 1\n",
M workflow/convert_test.go +22 -1
@@ -7,6 +7,7 @@ import (
77 "os"
88 "path/filepath"
99 "reflect"
10+ "regexp"
1011 "sort"
1112 "strings"
1213 "testing"
@@ -50,6 +51,26 @@ func errorTexts(errs []Error) []string {
5051 return out
5152 }
5253
54+// The reference (@actions/workflow-parser) words expression parse errors like
55+// the JavaScript expression library: no position suffix, and the function name
56+// in arity errors. package expr follows the C# runner, which is what runs the
57+// workflow: "<description>: '<token>'. Located at position N within
58+// expression: <expression>", with the "(" token in arity errors. The tests
59+// compare after reducing both to the part the two share.
60+var (
61+ runnerSuffix = regexp.MustCompile(`(?s)\. Located at position \d+ within expression: .*$`)
62+ runnerArity = regexp.MustCompile(`((?:Too few|Too many) parameters supplied|Even number of parameters supplied, requires an odd number of parameters): '[^']*'`)
63+)
64+
65+func runnerNeutral(msgs []string) []string {
66+ out := make([]string, len(msgs))
67+ for i, m := range msgs {
68+ m = runnerSuffix.ReplaceAllString(m, "")
69+ out[i] = runnerArity.ReplaceAllString(m, "$1: '...'")
70+ }
71+ return out
72+}
73+
5374 // isYAMLError recognises the reference's YAML syntax errors, whose text is
5475 // the JavaScript library's and cannot be reproduced word for word.
5576 func isYAMLError(s string) bool { return strings.Contains(s, " at line ") }
@@ -101,7 +122,7 @@ func TestConvertAgainstReference(t *testing.T) {
101122 }
102123 return
103124 }
104- if !reflect.DeepEqual(nonNil(got), nonNil(want.Errors)) {
125+ if !reflect.DeepEqual(runnerNeutral(nonNil(got)), runnerNeutral(nonNil(want.Errors))) {
105126 t.Errorf("errors differ\n got: %q\nwant: %q", got, want.Errors)
106127 }
107128 if len(want.Errors) > 0 {
@@ -7,6 +7,7 @@ import (
7 "os"7 "os"
8 "path/filepath"8 "path/filepath"
9 "reflect"9 "reflect"
10+ "regexp"
10 "sort"11 "sort"
11 "strings"12 "strings"
12 "testing"13 "testing"
@@ -50,6 +51,26 @@ func errorTexts(errs []Error) []string {
50 return out51 return out
51 }52 }
52 53
54+// The reference (@actions/workflow-parser) words expression parse errors like
55+// the JavaScript expression library: no position suffix, and the function name
56+// in arity errors. package expr follows the C# runner, which is what runs the
57+// workflow: "<description>: '<token>'. Located at position N within
58+// expression: <expression>", with the "(" token in arity errors. The tests
59+// compare after reducing both to the part the two share.
60+var (
61+ runnerSuffix = regexp.MustCompile(`(?s)\. Located at position \d+ within expression: .*$`)
62+ runnerArity = regexp.MustCompile(`((?:Too few|Too many) parameters supplied|Even number of parameters supplied, requires an odd number of parameters): '[^']*'`)
63+)
64+
65+func runnerNeutral(msgs []string) []string {
66+ out := make([]string, len(msgs))
67+ for i, m := range msgs {
68+ m = runnerSuffix.ReplaceAllString(m, "")
69+ out[i] = runnerArity.ReplaceAllString(m, "$1: '...'")
70+ }
71+ return out
72+}
73+
53 // isYAMLError recognises the reference's YAML syntax errors, whose text is74 // isYAMLError recognises the reference's YAML syntax errors, whose text is
54 // the JavaScript library's and cannot be reproduced word for word.75 // the JavaScript library's and cannot be reproduced word for word.
55 func isYAMLError(s string) bool { return strings.Contains(s, " at line ") }76 func isYAMLError(s string) bool { return strings.Contains(s, " at line ") }
@@ -101,7 +122,7 @@ func TestConvertAgainstReference(t *testing.T) {
101 }122 }
102 return123 return
103 }124 }
104- if !reflect.DeepEqual(nonNil(got), nonNil(want.Errors)) {125+ if !reflect.DeepEqual(runnerNeutral(nonNil(got)), runnerNeutral(nonNil(want.Errors))) {
105 t.Errorf("errors differ\n got: %q\nwant: %q", got, want.Errors)126 t.Errorf("errors differ\n got: %q\nwant: %q", got, want.Errors)
106 }127 }
107 if len(want.Errors) > 0 {128 if len(want.Errors) > 0 {

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 1

Tags 3

↑↓ moveenter openesc close