detached 633b184

fix(expr): format numbers like the runner (G15), and write them that way in toJSON

commit
633b1847005e66677490525e2786944c0222a4de
author
brice · · 2026-10-12 17:33 UTC
parent
5f59a46
signature
GPG VerifiedCD88F9E564E0102646FB6077A1A07C7C7C3E349D · arch-gpg Verified for the committer email
21 files · +331 -101
Split
UnifiedSplit
M expr/PORTING.md +17 -4
@@ -93,10 +93,23 @@ whether to keep the JS behaviour or follow the runner.
9393 the upper-cased UTF-16 code units. C# `OrdinalIgnoreCase` upper-cases
9494 per UTF-16 code unit with simple mappings only: `ß == 'SS'` should be
9595 false there.
96-- [ ] **Number to string (`CoerceString`)**: `(+x.toFixed(15)).toString()`.
97- C# formats a `double` with `"G15"`-like rules. They agree for ordinary
98- values; check very small and very large numbers (`1e-7` here is `1e-7`,
99- `1e21` is `1e+21`), `-0`, and 16-17 significant digit values.
96+- [x] **Number to string (`CoerceString`)**: resolved, follows the C#. The
97+ runner converts every number with `Double.ToString("G15",
98+ InvariantCulture)` (`ExpressionConstants.NumberFormat`,
99+ `EvaluationResult.ConvertToString`, `src/Sdk/DTExpressions2/Expressions2/EvaluationResult.cs`),
100+ not `(+x.toFixed(15)).toString()`: 15 significant digits, positional
101+ for decimal exponents -5 < X < 15, otherwise `1E+21`, `1E-07`,
102+ `1.23456789012346E+17` (uppercase `E`, at least two exponent digits).
103+ So `1e15` is `1E+15`, `123456789012345678` is `1.23456789012346E+17`,
104+ `0.000001` is `1E-06`, `5e-324` is `4.94065645841247E-324`, `1e309` is
105+ `Infinity`. Negative zero is `0`: .NET Core prints `-0`, and
106+ `src/Sdk/Expressions/EvaluationResult.cs` (the copy used by the newer
107+ evaluator) maps it back to `0` ("keep our behavior consistent"); the
108+ older `DTExpressions2` copy still prints `-0`, but the shared fixtures
109+ (`coerce_string.json`, `format('{0}', -0)`) say `0`. Checked against the
110+ runner's code on 200 000 random doubles. The number *literal* in an
111+ expression is parsed independently of this (see below) and the JSON
112+ writer of `toJSON` uses the same conversion (next item).
100113 - [ ] **String to number**: `Number()` semantics (hex, octal, binary,
101114 `Infinity`, whitespace trimmed, empty is 0). C# `double.TryParse` with
102115 `NumberStyles` differs (no octal/binary; check `0x` and `Infinity`).
@@ -93,10 +93,23 @@ whether to keep the JS behaviour or follow the runner.
93 the upper-cased UTF-16 code units. C# `OrdinalIgnoreCase` upper-cases93 the upper-cased UTF-16 code units. C# `OrdinalIgnoreCase` upper-cases
94 per UTF-16 code unit with simple mappings only: `ß == 'SS'` should be94 per UTF-16 code unit with simple mappings only: `ß == 'SS'` should be
95 false there.95 false there.
96-- [ ] **Number to string (`CoerceString`)**: `(+x.toFixed(15)).toString()`.96+- [x] **Number to string (`CoerceString`)**: resolved, follows the C#. The
97- C# formats a `double` with `"G15"`-like rules. They agree for ordinary97+ runner converts every number with `Double.ToString("G15",
98- values; check very small and very large numbers (`1e-7` here is `1e-7`,98+ InvariantCulture)` (`ExpressionConstants.NumberFormat`,
99- `1e21` is `1e+21`), `-0`, and 16-17 significant digit values.99+ `EvaluationResult.ConvertToString`, `src/Sdk/DTExpressions2/Expressions2/EvaluationResult.cs`),
100+ not `(+x.toFixed(15)).toString()`: 15 significant digits, positional
101+ for decimal exponents -5 < X < 15, otherwise `1E+21`, `1E-07`,
102+ `1.23456789012346E+17` (uppercase `E`, at least two exponent digits).
103+ So `1e15` is `1E+15`, `123456789012345678` is `1.23456789012346E+17`,
104+ `0.000001` is `1E-06`, `5e-324` is `4.94065645841247E-324`, `1e309` is
105+ `Infinity`. Negative zero is `0`: .NET Core prints `-0`, and
106+ `src/Sdk/Expressions/EvaluationResult.cs` (the copy used by the newer
107+ evaluator) maps it back to `0` ("keep our behavior consistent"); the
108+ older `DTExpressions2` copy still prints `-0`, but the shared fixtures
109+ (`coerce_string.json`, `format('{0}', -0)`) say `0`. Checked against the
110+ runner's code on 200 000 random doubles. The number *literal* in an
111+ expression is parsed independently of this (see below) and the JSON
112+ writer of `toJSON` uses the same conversion (next item).
100 - [ ] **String to number**: `Number()` semantics (hex, octal, binary,113 - [ ] **String to number**: `Number()` semantics (hex, octal, binary,
101 `Infinity`, whitespace trimmed, empty is 0). C# `double.TryParse` with114 `Infinity`, whitespace trimmed, empty is 0). C# `double.TryParse` with
102 `NumberStyles` differs (no octal/binary; check `0x` and `Infinity`).115 `NumberStyles` differs (no octal/binary; check `0x` and `Infinity`).
M expr/eval_test.go +33 -18
@@ -388,12 +388,12 @@ func TestEvalBuiltinFunctions(t *testing.T) {
388388 {"toJSON(null)", "String:null"},
389389 {"toJSON(true)", "String:true"},
390390 {"toJSON(1)", "String:1"},
391- {"toJSON(NaN)", "String:null"},
392- {"toJSON(Infinity)", "String:null"},
391+ {"toJSON(NaN)", "String:NaN"},
392+ {"toJSON(Infinity)", "String:Infinity"},
393393 {"toJSON(-0)", "String:0"},
394394 {"toJSON('x')", "String:\"x\""},
395- {"toJSON(1e21)", "String:1e+21"},
396- {"toJSON(1e-7)", "String:1e-7"},
395+ {"toJSON(1e21)", "String:1E+21"},
396+ {"toJSON(1e-7)", "String:1E-07"},
397397 {"toJSON(0.1)", "String:0.1"},
398398 {"toJSON(fromJSON('[]'))", "String:[]"},
399399 {"toJSON(fromJSON('{}'))", "String:{}"},
@@ -782,13 +782,18 @@ func TestEvaluateDoesNotMutateContext(t *testing.T) {
782782 }
783783 }
784784
785+// TestFloatCoercion checks the string form of numbers against the C# runner:
786+// Double.ToString("G15", InvariantCulture) (ExpressionConstants.NumberFormat,
787+// EvaluationResult.ConvertToString). Each expectation was obtained by running
788+// the runner's own code (gen/runner, --g15), and numberToCoercedString agrees
789+// with it on 200 000 random doubles.
785790 func TestFloatCoercion(t *testing.T) {
786791 tests := []struct {
787792 in float64
788793 want string
789794 }{
790795 {0, "0"},
791- {math.Copysign(0, -1), "0"},
796+ {math.Copysign(0, -1), "0"}, // see numberToCoercedString
792797 {1, "1"},
793798 {-1, "-1"},
794799 {1.5, "1.5"},
@@ -796,24 +801,34 @@ func TestFloatCoercion(t *testing.T) {
796801 {0.30000000000000004, "0.3"},
797802 {1.005, "1.005"},
798803 {100, "100"},
799- {1e20, "100000000000000000000"},
800- {1e21, "1e+21"},
801- {123456789012345680000, "123456789012345680000"},
802- {1e-7, "1e-7"}, // toFixed(15) keeps 0.000000100000000, which prints as 1e-7
803- {0.000001, "0.000001"},
804- {5e-16, "1e-15"}, // toFixed rounds half up on the exact binary value
805- {1e-15, "1e-15"},
806- {1e-16, "0"},
807- {-1e-16, "0"},
808- {12345678901234567890, "12345678901234567000"},
804+ {123456789012345, "123456789012345"},
805+ {999999999999999, "999999999999999"},
806+ {1e14, "100000000000000"},
807+ {1e15, "1E+15"},
808+ {1e16, "1E+16"},
809+ {1e20, "1E+20"},
810+ {1e21, "1E+21"},
811+ {123456789012345680000, "1.23456789012346E+20"},
812+ {1234567890123456, "1.23456789012346E+15"},
813+ {0.0001, "0.0001"},
814+ {0.00001, "1E-05"},
815+ {1.5e-5, "1.5E-05"},
816+ {1e-6, "1E-06"},
817+ {1e-7, "1E-07"},
818+ {5e-16, "5E-16"},
819+ {-1e-16, "-1E-16"},
820+ {9.999999999999995e-5, "0.0001"},
821+ {99999999999999.95, "100000000000000"},
822+ {999999999999999.5, "1E+15"},
823+ {123456789.123456789, "123456789.123457"},
824+ {123456789012345.5, "123456789012346"},
809825 {math.NaN(), "NaN"},
810826 {math.Inf(1), "Infinity"},
811827 {math.Inf(-1), "-Infinity"},
812828 {2.5, "2.5"},
813829 {-0.25, "-0.25"},
814- {4503599627370497.5, "4503599627370498"}, // the literal itself rounds to the nearest double
815- {1.7976931348623157e308, "1.7976931348623157e+308"},
816- {5e-324, "0"},
830+ {1.7976931348623157e308, "1.79769313486232E+308"},
831+ {5e-324, "4.94065645841247E-324"},
817832 }
818833 for _, tc := range tests {
819834 if got := CoerceString(Number(tc.in)); got != tc.want {
@@ -388,12 +388,12 @@ func TestEvalBuiltinFunctions(t *testing.T) {
388 {"toJSON(null)", "String:null"},388 {"toJSON(null)", "String:null"},
389 {"toJSON(true)", "String:true"},389 {"toJSON(true)", "String:true"},
390 {"toJSON(1)", "String:1"},390 {"toJSON(1)", "String:1"},
391- {"toJSON(NaN)", "String:null"},391+ {"toJSON(NaN)", "String:NaN"},
392- {"toJSON(Infinity)", "String:null"},392+ {"toJSON(Infinity)", "String:Infinity"},
393 {"toJSON(-0)", "String:0"},393 {"toJSON(-0)", "String:0"},
394 {"toJSON('x')", "String:\"x\""},394 {"toJSON('x')", "String:\"x\""},
395- {"toJSON(1e21)", "String:1e+21"},395+ {"toJSON(1e21)", "String:1E+21"},
396- {"toJSON(1e-7)", "String:1e-7"},396+ {"toJSON(1e-7)", "String:1E-07"},
397 {"toJSON(0.1)", "String:0.1"},397 {"toJSON(0.1)", "String:0.1"},
398 {"toJSON(fromJSON('[]'))", "String:[]"},398 {"toJSON(fromJSON('[]'))", "String:[]"},
399 {"toJSON(fromJSON('{}'))", "String:{}"},399 {"toJSON(fromJSON('{}'))", "String:{}"},
@@ -782,13 +782,18 @@ func TestEvaluateDoesNotMutateContext(t *testing.T) {
782 }782 }
783 }783 }
784 784
785+// TestFloatCoercion checks the string form of numbers against the C# runner:
786+// Double.ToString("G15", InvariantCulture) (ExpressionConstants.NumberFormat,
787+// EvaluationResult.ConvertToString). Each expectation was obtained by running
788+// the runner's own code (gen/runner, --g15), and numberToCoercedString agrees
789+// with it on 200 000 random doubles.
785 func TestFloatCoercion(t *testing.T) {790 func TestFloatCoercion(t *testing.T) {
786 tests := []struct {791 tests := []struct {
787 in float64792 in float64
788 want string793 want string
789 }{794 }{
790 {0, "0"},795 {0, "0"},
791- {math.Copysign(0, -1), "0"},796+ {math.Copysign(0, -1), "0"}, // see numberToCoercedString
792 {1, "1"},797 {1, "1"},
793 {-1, "-1"},798 {-1, "-1"},
794 {1.5, "1.5"},799 {1.5, "1.5"},
@@ -796,24 +801,34 @@ func TestFloatCoercion(t *testing.T) {
796 {0.30000000000000004, "0.3"},801 {0.30000000000000004, "0.3"},
797 {1.005, "1.005"},802 {1.005, "1.005"},
798 {100, "100"},803 {100, "100"},
799- {1e20, "100000000000000000000"},804+ {123456789012345, "123456789012345"},
800- {1e21, "1e+21"},805+ {999999999999999, "999999999999999"},
801- {123456789012345680000, "123456789012345680000"},806+ {1e14, "100000000000000"},
802- {1e-7, "1e-7"}, // toFixed(15) keeps 0.000000100000000, which prints as 1e-7807+ {1e15, "1E+15"},
803- {0.000001, "0.000001"},808+ {1e16, "1E+16"},
804- {5e-16, "1e-15"}, // toFixed rounds half up on the exact binary value809+ {1e20, "1E+20"},
805- {1e-15, "1e-15"},810+ {1e21, "1E+21"},
806- {1e-16, "0"},811+ {123456789012345680000, "1.23456789012346E+20"},
807- {-1e-16, "0"},812+ {1234567890123456, "1.23456789012346E+15"},
808- {12345678901234567890, "12345678901234567000"},813+ {0.0001, "0.0001"},
814+ {0.00001, "1E-05"},
815+ {1.5e-5, "1.5E-05"},
816+ {1e-6, "1E-06"},
817+ {1e-7, "1E-07"},
818+ {5e-16, "5E-16"},
819+ {-1e-16, "-1E-16"},
820+ {9.999999999999995e-5, "0.0001"},
821+ {99999999999999.95, "100000000000000"},
822+ {999999999999999.5, "1E+15"},
823+ {123456789.123456789, "123456789.123457"},
824+ {123456789012345.5, "123456789012346"},
809 {math.NaN(), "NaN"},825 {math.NaN(), "NaN"},
810 {math.Inf(1), "Infinity"},826 {math.Inf(1), "Infinity"},
811 {math.Inf(-1), "-Infinity"},827 {math.Inf(-1), "-Infinity"},
812 {2.5, "2.5"},828 {2.5, "2.5"},
813 {-0.25, "-0.25"},829 {-0.25, "-0.25"},
814- {4503599627370497.5, "4503599627370498"}, // the literal itself rounds to the nearest double830+ {1.7976931348623157e308, "1.79769313486232E+308"},
815- {1.7976931348623157e308, "1.7976931348623157e+308"},831+ {5e-324, "4.94065645841247E-324"},
816- {5e-324, "0"},
817 }832 }
818 for _, tc := range tests {833 for _, tc := range tests {
819 if got := CoerceString(Number(tc.in)); got != tc.want {834 if got := CoerceString(Number(tc.in)); got != tc.want {
M expr/fuzz_test.go +10 -1
@@ -157,7 +157,7 @@ func FuzzEvaluate(f *testing.F) {
157157 if !utf8.ValidString(j) {
158158 t.Fatalf("toJSON produced invalid UTF-8: %q", j)
159159 }
160- if _, err := FromJSON([]byte(j)); err != nil {
160+ if _, err := FromJSON([]byte(j)); err != nil && !hasNonFinite(j) {
161161 t.Fatalf("toJSON output does not parse: %v\n%s", err, j)
162162 }
163163 for k, c := range ctx.Contexts {
@@ -181,6 +181,9 @@ func FuzzFromJSON(f *testing.F) {
181181 }
182182 j := ToJSON(v)
183183 v2, err := FromJSON([]byte(j))
184+ if err != nil && hasNonFinite(j) {
185+ return
186+ }
184187 if err != nil {
185188 t.Fatalf("round trip failed: %v\n%s", err, j)
186189 }
@@ -189,3 +192,9 @@ func FuzzFromJSON(f *testing.F) {
189192 }
190193 })
191194 }
195+
196+// hasNonFinite reports whether toJSON output holds NaN or an infinity, which
197+// the runner writes as bare words (not JSON).
198+func hasNonFinite(j string) bool {
199+ return strings.Contains(j, "NaN") || strings.Contains(j, "Infinity")
200+}
@@ -157,7 +157,7 @@ func FuzzEvaluate(f *testing.F) {
157 if !utf8.ValidString(j) {157 if !utf8.ValidString(j) {
158 t.Fatalf("toJSON produced invalid UTF-8: %q", j)158 t.Fatalf("toJSON produced invalid UTF-8: %q", j)
159 }159 }
160- if _, err := FromJSON([]byte(j)); err != nil {160+ if _, err := FromJSON([]byte(j)); err != nil && !hasNonFinite(j) {
161 t.Fatalf("toJSON output does not parse: %v\n%s", err, j)161 t.Fatalf("toJSON output does not parse: %v\n%s", err, j)
162 }162 }
163 for k, c := range ctx.Contexts {163 for k, c := range ctx.Contexts {
@@ -181,6 +181,9 @@ func FuzzFromJSON(f *testing.F) {
181 }181 }
182 j := ToJSON(v)182 j := ToJSON(v)
183 v2, err := FromJSON([]byte(j))183 v2, err := FromJSON([]byte(j))
184+ if err != nil && hasNonFinite(j) {
185+ return
186+ }
184 if err != nil {187 if err != nil {
185 t.Fatalf("round trip failed: %v\n%s", err, j)188 t.Fatalf("round trip failed: %v\n%s", err, j)
186 }189 }
@@ -189,3 +192,9 @@ func FuzzFromJSON(f *testing.F) {
189 }192 }
190 })193 })
191 }194 }
195+
196+// hasNonFinite reports whether toJSON output holds NaN or an infinity, which
197+// the runner writes as bare words (not JSON).
198+func hasNonFinite(j string) bool {
199+ return strings.Contains(j, "NaN") || strings.Contains(j, "Infinity")
200+}
M expr/gen/runner/Program.cs +17 -1
@@ -60,6 +60,16 @@ internal static class P
6060
6161 static int Main(string[] args)
6262 {
63+ if (args[0] == "--g15")
64+ {
65+ // Debug aid: one hexadecimal IEEE 754 bit pattern per input line, the G15 form per output line.
66+ foreach (var line in File.ReadLines(args[1]))
67+ {
68+ var d = BitConverter.Int64BitsToDouble(Convert.ToInt64(line.Trim(), 16));
69+ Console.WriteLine(d.ToString("G15", System.Globalization.CultureInfo.InvariantCulture));
70+ }
71+ return 0;
72+ }
6373 JObject root; using (var jr0 = new JsonTextReader(new StringReader(File.ReadAllText(args[0]))) { DateParseHandling = DateParseHandling.None }) root = JObject.Load(jr0);
6474 var defaultCtx = (JObject)root["contexts"];
6575 var funcs = new IFunctionInfo[]
@@ -118,7 +128,13 @@ internal static class P
118128 }
119129 }
120130 }
121- output.WriteLine(res.ToString(Formatting.None));
131+ // ASCII-escaped, so that a lone surrogate (which UTF-8 cannot carry) survives.
132+ var sw = new StringWriter();
133+ using (var jw = new JsonTextWriter(sw) { StringEscapeHandling = StringEscapeHandling.EscapeNonAscii, Formatting = Formatting.None })
134+ {
135+ res.WriteTo(jw);
136+ }
137+ output.WriteLine(sw.ToString());
122138 }
123139 output.Flush();
124140 return 0;
@@ -60,6 +60,16 @@ internal static class P
60 60
61 static int Main(string[] args)61 static int Main(string[] args)
62 {62 {
63+ if (args[0] == "--g15")
64+ {
65+ // Debug aid: one hexadecimal IEEE 754 bit pattern per input line, the G15 form per output line.
66+ foreach (var line in File.ReadLines(args[1]))
67+ {
68+ var d = BitConverter.Int64BitsToDouble(Convert.ToInt64(line.Trim(), 16));
69+ Console.WriteLine(d.ToString("G15", System.Globalization.CultureInfo.InvariantCulture));
70+ }
71+ return 0;
72+ }
63 JObject root; using (var jr0 = new JsonTextReader(new StringReader(File.ReadAllText(args[0]))) { DateParseHandling = DateParseHandling.None }) root = JObject.Load(jr0);73 JObject root; using (var jr0 = new JsonTextReader(new StringReader(File.ReadAllText(args[0]))) { DateParseHandling = DateParseHandling.None }) root = JObject.Load(jr0);
64 var defaultCtx = (JObject)root["contexts"];74 var defaultCtx = (JObject)root["contexts"];
65 var funcs = new IFunctionInfo[]75 var funcs = new IFunctionInfo[]
@@ -118,7 +128,13 @@ internal static class P
118 }128 }
119 }129 }
120 }130 }
121- output.WriteLine(res.ToString(Formatting.None));131+ // ASCII-escaped, so that a lone surrogate (which UTF-8 cannot carry) survives.
132+ var sw = new StringWriter();
133+ using (var jw = new JsonTextWriter(sw) { StringEscapeHandling = StringEscapeHandling.EscapeNonAscii, Formatting = Formatting.None })
134+ {
135+ res.WriteTo(jw);
136+ }
137+ output.WriteLine(sw.ToString());
122 }138 }
123 output.Flush();139 output.Flush();
124 return 0;140 return 0;
A expr/gen/runner/goldens.mjs +29 -0
@@ -0,0 +1,29 @@
1+// SPDX-License-Identifier: MIT
2+//
3+// For each testdata/tojson/NAME.json, runs toJSON(fromJSON(<file>)) through the
4+// C# runner's engine and writes testdata/tojson/NAME.runner when the answer is
5+// not the JavaScript one (NAME.golden). golden_test.go prefers NAME.runner.
6+//
7+// dotnet build -c Release expr/gen/runner -p:RunnerSrc=<runner>/src/Sdk
8+// node expr/gen/runner/goldens.mjs
9+
10+import { execFileSync } from "node:child_process";
11+import { readFileSync, readdirSync, writeFileSync, mkdtempSync, rmSync } from "node:fs";
12+import { tmpdir } from "node:os";
13+import { join, dirname } from "node:path";
14+import { fileURLToPath } from "node:url";
15+
16+const here = dirname(fileURLToPath(import.meta.url));
17+const dir = join(here, "../../testdata/tojson");
18+const names = readdirSync(dir).filter(f => f.endsWith(".json")).map(f => f.slice(0, -5)).sort();
19+const cases = names.map(n => ({ e: `toJSON(fromJSON('${readFileSync(join(dir, n + ".json"), "utf8").replaceAll("'", "''")}'))` }));
20+const tmp = mkdtempSync(join(tmpdir(), "goldens-"));
21+writeFileSync(join(tmp, "in.json"), JSON.stringify({ contexts: {}, cases }));
22+const out = execFileSync("dotnet", [join(here, "bin/Release/net9.0/runner.dll"), join(tmp, "in.json")], { maxBuffer: 1 << 28 }).toString("utf8").split("\n").filter(Boolean);
23+rmSync(tmp, { recursive: true });
24+names.forEach((n, i) => {
25+ const r = JSON.parse(out[i]);
26+ const golden = readFileSync(join(dir, n + ".golden"), "utf8");
27+ if (r.err) { console.error(`${n}: runner error: ${r.err.m}`); return; }
28+ if (r.s !== golden) { writeFileSync(join(dir, n + ".runner"), r.s); console.log(`${n}: differs`); }
29+});
@@ -0,0 +1,29 @@
1+// SPDX-License-Identifier: MIT
2+//
3+// For each testdata/tojson/NAME.json, runs toJSON(fromJSON(<file>)) through the
4+// C# runner's engine and writes testdata/tojson/NAME.runner when the answer is
5+// not the JavaScript one (NAME.golden). golden_test.go prefers NAME.runner.
6+//
7+// dotnet build -c Release expr/gen/runner -p:RunnerSrc=<runner>/src/Sdk
8+// node expr/gen/runner/goldens.mjs
9+
10+import { execFileSync } from "node:child_process";
11+import { readFileSync, readdirSync, writeFileSync, mkdtempSync, rmSync } from "node:fs";
12+import { tmpdir } from "node:os";
13+import { join, dirname } from "node:path";
14+import { fileURLToPath } from "node:url";
15+
16+const here = dirname(fileURLToPath(import.meta.url));
17+const dir = join(here, "../../testdata/tojson");
18+const names = readdirSync(dir).filter(f => f.endsWith(".json")).map(f => f.slice(0, -5)).sort();
19+const cases = names.map(n => ({ e: `toJSON(fromJSON('${readFileSync(join(dir, n + ".json"), "utf8").replaceAll("'", "''")}'))` }));
20+const tmp = mkdtempSync(join(tmpdir(), "goldens-"));
21+writeFileSync(join(tmp, "in.json"), JSON.stringify({ contexts: {}, cases }));
22+const out = execFileSync("dotnet", [join(here, "bin/Release/net9.0/runner.dll"), join(tmp, "in.json")], { maxBuffer: 1 << 28 }).toString("utf8").split("\n").filter(Boolean);
23+rmSync(tmp, { recursive: true });
24+names.forEach((n, i) => {
25+ const r = JSON.parse(out[i]);
26+ const golden = readFileSync(join(dir, n + ".golden"), "utf8");
27+ if (r.err) { console.error(`${n}: runner error: ${r.err.m}`); return; }
28+ if (r.s !== golden) { writeFileSync(join(dir, n + ".runner"), r.s); console.log(`${n}: differs`); }
29+});
M expr/golden_test.go +13 -1
@@ -24,7 +24,14 @@ func TestToJSONGolden(t *testing.T) {
2424 if err != nil {
2525 t.Fatal(err)
2626 }
27- want, err := os.ReadFile(strings.TrimSuffix(in, ".json") + ".golden")
27+ // NAME.golden is the answer of the JavaScript reference. Where
28+ // the C# runner answers differently, NAME.runner holds its
29+ // answer and is what this package follows (see PORTING.md).
30+ want, err := os.ReadFile(strings.TrimSuffix(in, ".json") + ".runner")
31+ fromRunner := err == nil
32+ if os.IsNotExist(err) {
33+ want, err = os.ReadFile(strings.TrimSuffix(in, ".json") + ".golden")
34+ }
2835 if err != nil {
2936 t.Fatal(err)
3037 }
@@ -35,6 +42,11 @@ func TestToJSONGolden(t *testing.T) {
3542 if got := ToJSON(v); got != string(want) {
3643 t.Errorf("toJSON mismatch\n got: %q\nwant: %q", got, want)
3744 }
45+ if fromRunner {
46+ // The runner's output is not JSON in general: numbers have 15
47+ // significant digits and NaN and Infinity are bare words.
48+ return
49+ }
3850 // The golden output is itself JSON that must round-trip.
3951 v2, err := FromJSON(want)
4052 if err != nil {
@@ -24,7 +24,14 @@ func TestToJSONGolden(t *testing.T) {
24 if err != nil {24 if err != nil {
25 t.Fatal(err)25 t.Fatal(err)
26 }26 }
27- want, err := os.ReadFile(strings.TrimSuffix(in, ".json") + ".golden")27+ // NAME.golden is the answer of the JavaScript reference. Where
28+ // the C# runner answers differently, NAME.runner holds its
29+ // answer and is what this package follows (see PORTING.md).
30+ want, err := os.ReadFile(strings.TrimSuffix(in, ".json") + ".runner")
31+ fromRunner := err == nil
32+ if os.IsNotExist(err) {
33+ want, err = os.ReadFile(strings.TrimSuffix(in, ".json") + ".golden")
34+ }
28 if err != nil {35 if err != nil {
29 t.Fatal(err)36 t.Fatal(err)
30 }37 }
@@ -35,6 +42,11 @@ func TestToJSONGolden(t *testing.T) {
35 if got := ToJSON(v); got != string(want) {42 if got := ToJSON(v); got != string(want) {
36 t.Errorf("toJSON mismatch\n got: %q\nwant: %q", got, want)43 t.Errorf("toJSON mismatch\n got: %q\nwant: %q", got, want)
37 }44 }
45+ if fromRunner {
46+ // The runner's output is not JSON in general: numbers have 15
47+ // significant digits and NaN and Infinity are bare words.
48+ return
49+ }
38 // The golden output is itself JSON that must round-trip.50 // The golden output is itself JSON that must round-trip.
39 v2, err := FromJSON(want)51 v2, err := FromJSON(want)
40 if err != nil {52 if err != nil {
A expr/jsnumber_test.go +58 -0
@@ -0,0 +1,58 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "math"
7+ "strconv"
8+ "strings"
9+)
10+
11+// The helpers below reproduce what the JavaScript reference prints for a
12+// number. The package itself follows the C# runner (numberToCoercedString);
13+// the oracle tests need the JavaScript form to read the recorded tokens.
14+
15+// jsNumberToString is Number.prototype.toString: the shortest decimal that
16+// round-trips, in positional notation for exponents from -7 to 20 and in
17+// exponent notation ("1e+21", "1.5e-7") beyond.
18+func jsNumberToString(x float64) string {
19+ switch {
20+ case math.IsNaN(x):
21+ return "NaN"
22+ case x == 0:
23+ return "0"
24+ case math.IsInf(x, 1):
25+ return "Infinity"
26+ case math.IsInf(x, -1):
27+ return "-Infinity"
28+ }
29+ sign := ""
30+ if x < 0 {
31+ sign = "-"
32+ x = -x
33+ }
34+ s := strconv.FormatFloat(x, 'e', -1, 64) // d.ddde±xx
35+ mant, expStr, _ := strings.Cut(s, "e")
36+ exp, _ := strconv.Atoi(expStr)
37+ digits := strings.Replace(mant, ".", "", 1)
38+ k := len(digits)
39+ n := exp + 1
40+ switch {
41+ case k <= n && n <= 21:
42+ return sign + digits + strings.Repeat("0", n-k)
43+ case 0 < n && n <= 21:
44+ return sign + digits[:n] + "." + digits[n:]
45+ case -6 < n && n <= 0:
46+ return sign + "0." + strings.Repeat("0", -n) + digits
47+ }
48+ e := n - 1
49+ esign := "+"
50+ if e < 0 {
51+ esign = "-"
52+ e = -e
53+ }
54+ if k == 1 {
55+ return sign + digits + "e" + esign + strconv.Itoa(e)
56+ }
57+ return sign + digits[:1] + "." + digits[1:] + "e" + esign + strconv.Itoa(e)
58+}
@@ -0,0 +1,58 @@
1+// SPDX-License-Identifier: MIT
2+
3+package expr
4+
5+import (
6+ "math"
7+ "strconv"
8+ "strings"
9+)
10+
11+// The helpers below reproduce what the JavaScript reference prints for a
12+// number. The package itself follows the C# runner (numberToCoercedString);
13+// the oracle tests need the JavaScript form to read the recorded tokens.
14+
15+// jsNumberToString is Number.prototype.toString: the shortest decimal that
16+// round-trips, in positional notation for exponents from -7 to 20 and in
17+// exponent notation ("1e+21", "1.5e-7") beyond.
18+func jsNumberToString(x float64) string {
19+ switch {
20+ case math.IsNaN(x):
21+ return "NaN"
22+ case x == 0:
23+ return "0"
24+ case math.IsInf(x, 1):
25+ return "Infinity"
26+ case math.IsInf(x, -1):
27+ return "-Infinity"
28+ }
29+ sign := ""
30+ if x < 0 {
31+ sign = "-"
32+ x = -x
33+ }
34+ s := strconv.FormatFloat(x, 'e', -1, 64) // d.ddde±xx
35+ mant, expStr, _ := strings.Cut(s, "e")
36+ exp, _ := strconv.Atoi(expStr)
37+ digits := strings.Replace(mant, ".", "", 1)
38+ k := len(digits)
39+ n := exp + 1
40+ switch {
41+ case k <= n && n <= 21:
42+ return sign + digits + strings.Repeat("0", n-k)
43+ case 0 < n && n <= 21:
44+ return sign + digits[:n] + "." + digits[n:]
45+ case -6 < n && n <= 0:
46+ return sign + "0." + strings.Repeat("0", -n) + digits
47+ }
48+ e := n - 1
49+ esign := "+"
50+ if e < 0 {
51+ esign = "-"
52+ e = -e
53+ }
54+ if k == 1 {
55+ return sign + digits + "e" + esign + strconv.Itoa(e)
56+ }
57+ return sign + digits[:1] + "." + digits[1:] + "e" + esign + strconv.Itoa(e)
58+}
M expr/json.go +6 -6
@@ -453,12 +453,12 @@ func (w *jsonWriter) value(v Value, depth int) {
453453 w.str("false")
454454 }
455455 case Number:
456- f := float64(x)
457- if math.IsNaN(f) || math.IsInf(f, 0) {
458- w.str("null")
459- } else {
460- w.str(jsNumberToString(f))
461- }
456+ // The runner writes the number as it converts it to a string
457+ // (ToJson.WriteValue: value.ConvertToString()), so NaN and the
458+ // infinities are written as such, and a number has at most 15
459+ // significant digits. JSON.stringify gives null and the shortest
460+ // round-trip form.
461+ w.str(numberToCoercedString(float64(x)))
462462 case String:
463463 w.jsonString(string(x))
464464 case *Array:
@@ -453,12 +453,12 @@ func (w *jsonWriter) value(v Value, depth int) {
453 w.str("false")453 w.str("false")
454 }454 }
455 case Number:455 case Number:
456- f := float64(x)456+ // The runner writes the number as it converts it to a string
457- if math.IsNaN(f) || math.IsInf(f, 0) {457+ // (ToJson.WriteValue: value.ConvertToString()), so NaN and the
458- w.str("null")458+ // infinities are written as such, and a number has at most 15
459- } else {459+ // significant digits. JSON.stringify gives null and the shortest
460- w.str(jsNumberToString(f))460+ // round-trip form.
461- }461+ w.str(numberToCoercedString(float64(x)))
462 case String:462 case String:
463 w.jsonString(string(x))463 w.jsonString(string(x))
464 case *Array:464 case *Array:
M expr/json_test.go +7 -5
@@ -132,12 +132,14 @@ func TestToJSONFormatting(t *testing.T) {
132132 }{
133133 {nil, "null"},
134134 {Null{}, "null"},
135- {Number(math.NaN()), "null"},
136- {Number(math.Inf(1)), "null"},
137- {Number(math.Inf(-1)), "null"},
135+ // The runner writes numbers with ConvertToString (G15), so NaN and
136+ // the infinities are written as such.
137+ {Number(math.NaN()), "NaN"},
138+ {Number(math.Inf(1)), "Infinity"},
139+ {Number(math.Inf(-1)), "-Infinity"},
138140 {Number(math.Copysign(0, -1)), "0"},
139- {Number(1e21), "1e+21"},
140- {Number(1e-7), "1e-7"},
141+ {Number(1e21), "1E+21"},
142+ {Number(1e-7), "1E-07"},
141143 {Number(123456789), "123456789"},
142144 {String("\u0000\u001f\u007f\u0080\u2028"), `"\u0000\u001f` + "\u007f\u0080\u2028" + `"`},
143145 {String("\"\\\b\f\n\r\t"), `"\"\\\b\f\n\r\t"`},
@@ -132,12 +132,14 @@ func TestToJSONFormatting(t *testing.T) {
132 }{132 }{
133 {nil, "null"},133 {nil, "null"},
134 {Null{}, "null"},134 {Null{}, "null"},
135- {Number(math.NaN()), "null"},135+ // The runner writes numbers with ConvertToString (G15), so NaN and
136- {Number(math.Inf(1)), "null"},136+ // the infinities are written as such.
137- {Number(math.Inf(-1)), "null"},137+ {Number(math.NaN()), "NaN"},
138+ {Number(math.Inf(1)), "Infinity"},
139+ {Number(math.Inf(-1)), "-Infinity"},
138 {Number(math.Copysign(0, -1)), "0"},140 {Number(math.Copysign(0, -1)), "0"},
139- {Number(1e21), "1e+21"},141+ {Number(1e21), "1E+21"},
140- {Number(1e-7), "1e-7"},142+ {Number(1e-7), "1E-07"},
141 {Number(123456789), "123456789"},143 {Number(123456789), "123456789"},
142 {String("\u0000\u001f\u007f\u0080\u2028"), `"\u0000\u001f` + "\u007f\u0080\u2028" + `"`},144 {String("\u0000\u001f\u007f\u0080\u2028"), `"\u0000\u001f` + "\u007f\u0080\u2028" + `"`},
143 {String("\"\\\b\f\n\r\t"), `"\"\\\b\f\n\r\t"`},145 {String("\"\\\b\f\n\r\t"), `"\"\\\b\f\n\r\t"`},
M expr/number.go +45 -63
@@ -14,86 +14,68 @@ var (
1414 posInf = math.Inf(1)
1515 )
1616
17-// jsNumberToString is Number.prototype.toString: the shortest decimal that
18-// round-trips, in positional notation for exponents from -7 to 20 and in
19-// exponent notation ("1e+21", "1.5e-7") beyond.
20-func jsNumberToString(x float64) string {
17+// numberToCoercedString is the string form of a number: .NET's
18+// Double.ToString("G15", InvariantCulture), which is what the C# runner uses
19+// for every number-to-string conversion (ExpressionConstants.NumberFormat,
20+// EvaluationResult.ConvertToString, hence format(), join(), toJSON() and the
21+// comparison with strings). The JavaScript reference computes
22+// (+x.toFixed(15)).toString() instead; they agree on ordinary values and
23+// differ on large, small and very precise ones. See PORTING.md.
24+//
25+// G15 rounds to 15 significant digits (ties to even on the exact binary
26+// value), then prints positionally when the decimal exponent X of the rounded
27+// value satisfies -5 < X < 15, and as d.dddE+XX otherwise, with the trailing
28+// zeros of the mantissa removed and at least two exponent digits. NaN is
29+// "NaN" and the infinities are "Infinity" and "-Infinity". Negative zero is
30+// "0": .NET Core prints "-0", which Sdk/Expressions/EvaluationResult.cs
31+// (ConvertToString, "keep our behavior consistent") and the shared fixtures
32+// (coerce_string.json, format('{0}', -0)) undo; the older
33+// DTExpressions2 copy of the runner lacks that fix and prints "-0".
34+func numberToCoercedString(x float64) string {
2135 switch {
2236 case math.IsNaN(x):
2337 return "NaN"
24- case x == 0:
25- return "0"
2638 case math.IsInf(x, 1):
2739 return "Infinity"
2840 case math.IsInf(x, -1):
2941 return "-Infinity"
42+ case x == 0:
43+ return "0"
3044 }
3145 sign := ""
3246 if x < 0 {
3347 sign = "-"
3448 x = -x
3549 }
36- s := strconv.FormatFloat(x, 'e', -1, 64) // d.ddde±xx
50+ s := strconv.FormatFloat(x, 'e', 14, 64) // d.dddddddddddddde±xx, correctly rounded
3751 mant, expStr, _ := strings.Cut(s, "e")
3852 exp, _ := strconv.Atoi(expStr)
39- digits := strings.Replace(mant, ".", "", 1)
40- k := len(digits)
41- n := exp + 1
42- switch {
43- case k <= n && n <= 21:
44- return sign + digits + strings.Repeat("0", n-k)
45- case 0 < n && n <= 21:
46- return sign + digits[:n] + "." + digits[n:]
47- case -6 < n && n <= 0:
48- return sign + "0." + strings.Repeat("0", -n) + digits
49- }
50- e := n - 1
51- esign := "+"
52- if e < 0 {
53- esign = "-"
54- e = -e
55- }
56- if k == 1 {
57- return sign + digits + "e" + esign + strconv.Itoa(e)
58- }
59- return sign + digits[:1] + "." + digits[1:] + "e" + esign + strconv.Itoa(e)
60-}
61-
62-var pow10to15 = new(big.Int).Exp(big.NewInt(10), big.NewInt(15), nil)
63-
64-// numberToCoercedString is NumberData.coerceString of the reference
65-// implementation: (+x.toFixed(15)).toString(). Rounding to 15 decimals
66-// removes binary noise (0.1+0.2 is "0.3") and ties round half up like
67-// toFixed.
68-func numberToCoercedString(x float64) string {
69- if x == 0 {
70- return "0"
71- }
72- if math.IsNaN(x) || math.IsInf(x, 0) {
73- return jsNumberToString(x)
74- }
75- a := math.Abs(x)
76- // toFixed returns Number::toString from 1e21 up, and for integers the
77- // fixed form parses back to the same number.
78- if a >= 1e21 || a == math.Trunc(a) {
79- return jsNumberToString(x)
80- }
81- r := new(big.Rat).SetFloat64(a)
82- r.Mul(r, new(big.Rat).SetInt(pow10to15))
83- q, rem := new(big.Int).QuoRem(r.Num(), r.Denom(), new(big.Int))
84- if rem.Lsh(rem, 1).Cmp(r.Denom()) >= 0 {
85- q.Add(q, big.NewInt(1))
86- }
87- digits := q.String()
88- if len(digits) < 16 {
89- digits = strings.Repeat("0", 16-len(digits)) + digits
53+ digits := strings.TrimRight(strings.Replace(mant, ".", "", 1), "0")
54+ if digits == "" {
55+ digits = "0"
9056 }
91- fixed := digits[:len(digits)-15] + "." + digits[len(digits)-15:]
92- v, _ := strconv.ParseFloat(fixed, 64)
93- if x < 0 {
94- v = -v
57+ switch {
58+ case exp >= 15 || exp <= -5:
59+ out := digits[:1]
60+ if len(digits) > 1 {
61+ out += "." + digits[1:]
62+ }
63+ esign := "+"
64+ if exp < 0 {
65+ esign = "-"
66+ exp = -exp
67+ }
68+ e := strconv.Itoa(exp)
69+ if len(e) < 2 {
70+ e = "0" + e
71+ }
72+ return sign + out + "E" + esign + e
73+ case exp < 0:
74+ return sign + "0." + strings.Repeat("0", -exp-1) + digits
75+ case len(digits) <= exp+1:
76+ return sign + digits + strings.Repeat("0", exp+1-len(digits))
9577 }
96- return jsNumberToString(v)
78+ return sign + digits[:exp+1] + "." + digits[exp+1:]
9779 }
9880
9981 // jsToNumber is Number(string): the StringToNumber algorithm of ECMAScript.
@@ -14,86 +14,68 @@ var (
14 posInf = math.Inf(1)14 posInf = math.Inf(1)
15 )15 )
16 16
17-// jsNumberToString is Number.prototype.toString: the shortest decimal that17+// numberToCoercedString is the string form of a number: .NET's
18-// round-trips, in positional notation for exponents from -7 to 20 and in18+// Double.ToString("G15", InvariantCulture), which is what the C# runner uses
19-// exponent notation ("1e+21", "1.5e-7") beyond.19+// for every number-to-string conversion (ExpressionConstants.NumberFormat,
20-func jsNumberToString(x float64) string {20+// EvaluationResult.ConvertToString, hence format(), join(), toJSON() and the
21+// comparison with strings). The JavaScript reference computes
22+// (+x.toFixed(15)).toString() instead; they agree on ordinary values and
23+// differ on large, small and very precise ones. See PORTING.md.
24+//
25+// G15 rounds to 15 significant digits (ties to even on the exact binary
26+// value), then prints positionally when the decimal exponent X of the rounded
27+// value satisfies -5 < X < 15, and as d.dddE+XX otherwise, with the trailing
28+// zeros of the mantissa removed and at least two exponent digits. NaN is
29+// "NaN" and the infinities are "Infinity" and "-Infinity". Negative zero is
30+// "0": .NET Core prints "-0", which Sdk/Expressions/EvaluationResult.cs
31+// (ConvertToString, "keep our behavior consistent") and the shared fixtures
32+// (coerce_string.json, format('{0}', -0)) undo; the older
33+// DTExpressions2 copy of the runner lacks that fix and prints "-0".
34+func numberToCoercedString(x float64) string {
21 switch {35 switch {
22 case math.IsNaN(x):36 case math.IsNaN(x):
23 return "NaN"37 return "NaN"
24- case x == 0:
25- return "0"
26 case math.IsInf(x, 1):38 case math.IsInf(x, 1):
27 return "Infinity"39 return "Infinity"
28 case math.IsInf(x, -1):40 case math.IsInf(x, -1):
29 return "-Infinity"41 return "-Infinity"
42+ case x == 0:
43+ return "0"
30 }44 }
31 sign := ""45 sign := ""
32 if x < 0 {46 if x < 0 {
33 sign = "-"47 sign = "-"
34 x = -x48 x = -x
35 }49 }
36- s := strconv.FormatFloat(x, 'e', -1, 64) // d.ddde±xx50+ s := strconv.FormatFloat(x, 'e', 14, 64) // d.dddddddddddddde±xx, correctly rounded
37 mant, expStr, _ := strings.Cut(s, "e")51 mant, expStr, _ := strings.Cut(s, "e")
38 exp, _ := strconv.Atoi(expStr)52 exp, _ := strconv.Atoi(expStr)
39- digits := strings.Replace(mant, ".", "", 1)53+ digits := strings.TrimRight(strings.Replace(mant, ".", "", 1), "0")
40- k := len(digits)54+ if digits == "" {
41- n := exp + 155+ digits = "0"
42- switch {
43- case k <= n && n <= 21:
44- return sign + digits + strings.Repeat("0", n-k)
45- case 0 < n && n <= 21:
46- return sign + digits[:n] + "." + digits[n:]
47- case -6 < n && n <= 0:
48- return sign + "0." + strings.Repeat("0", -n) + digits
49- }
50- e := n - 1
51- esign := "+"
52- if e < 0 {
53- esign = "-"
54- e = -e
55- }
56- if k == 1 {
57- return sign + digits + "e" + esign + strconv.Itoa(e)
58- }
59- return sign + digits[:1] + "." + digits[1:] + "e" + esign + strconv.Itoa(e)
60-}
61-
62-var pow10to15 = new(big.Int).Exp(big.NewInt(10), big.NewInt(15), nil)
63-
64-// numberToCoercedString is NumberData.coerceString of the reference
65-// implementation: (+x.toFixed(15)).toString(). Rounding to 15 decimals
66-// removes binary noise (0.1+0.2 is "0.3") and ties round half up like
67-// toFixed.
68-func numberToCoercedString(x float64) string {
69- if x == 0 {
70- return "0"
71- }
72- if math.IsNaN(x) || math.IsInf(x, 0) {
73- return jsNumberToString(x)
74- }
75- a := math.Abs(x)
76- // toFixed returns Number::toString from 1e21 up, and for integers the
77- // fixed form parses back to the same number.
78- if a >= 1e21 || a == math.Trunc(a) {
79- return jsNumberToString(x)
80- }
81- r := new(big.Rat).SetFloat64(a)
82- r.Mul(r, new(big.Rat).SetInt(pow10to15))
83- q, rem := new(big.Int).QuoRem(r.Num(), r.Denom(), new(big.Int))
84- if rem.Lsh(rem, 1).Cmp(r.Denom()) >= 0 {
85- q.Add(q, big.NewInt(1))
86- }
87- digits := q.String()
88- if len(digits) < 16 {
89- digits = strings.Repeat("0", 16-len(digits)) + digits
90 }56 }
91- fixed := digits[:len(digits)-15] + "." + digits[len(digits)-15:]57+ switch {
92- v, _ := strconv.ParseFloat(fixed, 64)58+ case exp >= 15 || exp <= -5:
93- if x < 0 {59+ out := digits[:1]
94- v = -v60+ if len(digits) > 1 {
61+ out += "." + digits[1:]
62+ }
63+ esign := "+"
64+ if exp < 0 {
65+ esign = "-"
66+ exp = -exp
67+ }
68+ e := strconv.Itoa(exp)
69+ if len(e) < 2 {
70+ e = "0" + e
71+ }
72+ return sign + out + "E" + esign + e
73+ case exp < 0:
74+ return sign + "0." + strings.Repeat("0", -exp-1) + digits
75+ case len(digits) <= exp+1:
76+ return sign + digits + strings.Repeat("0", exp+1-len(digits))
95 }77 }
96- return jsNumberToString(v)78+ return sign + digits[:exp+1] + "." + digits[exp+1:]
97 }79 }
98 80
99 // jsToNumber is Number(string): the StringToNumber algorithm of ECMAScript.81 // jsToNumber is Number(string): the StringToNumber algorithm of ECMAScript.
M expr/runner_oracle_test.go +22 -2
@@ -9,6 +9,7 @@ import (
99 "os"
1010 "strings"
1111 "testing"
12+ "unicode/utf8"
1213 )
1314
1415 // testdata/runner.json.gz holds what the C# runner's expression engine
@@ -57,6 +58,25 @@ func loadRunner(tb testing.TB) *runnerFile {
5758 return &rf
5859 }
5960
61+// lossy replaces every lone surrogate (WTF-8) by U+FFFD, which is what a JSON
62+// decoder makes of the \ud800 escapes in the recorded answers.
63+func lossy(s string) string {
64+ if !hasWTF8(s) {
65+ return s
66+ }
67+ var b strings.Builder
68+ for i := 0; i < len(s); {
69+ r, w := decodeRune(s[i:])
70+ if r >= 0xD800 && r < 0xE000 {
71+ b.WriteRune(utf8.RuneError)
72+ } else {
73+ b.WriteString(s[i : i+w])
74+ }
75+ i += w
76+ }
77+ return b.String()
78+}
79+
6080 // goAnswer evaluates e like the runner harness does and returns its answer in
6181 // the runner's format.
6282 func goAnswer(e string, ctxNames []string, ec EvalContext) runnerCase {
@@ -82,8 +102,8 @@ func goAnswer(e string, ctxNames []string, ec EvalContext) runnerCase {
82102 return rc
83103 }
84104 rc.K = v.Kind().String()
85- rc.S = CoerceString(v)
86- rc.J = ToJSON(v)
105+ rc.S = lossy(CoerceString(v))
106+ rc.J = lossy(ToJSON(v))
87107 return rc
88108 }
89109
@@ -9,6 +9,7 @@ import (
9 "os"9 "os"
10 "strings"10 "strings"
11 "testing"11 "testing"
12+ "unicode/utf8"
12 )13 )
13 14
14 // testdata/runner.json.gz holds what the C# runner's expression engine15 // testdata/runner.json.gz holds what the C# runner's expression engine
@@ -57,6 +58,25 @@ func loadRunner(tb testing.TB) *runnerFile {
57 return &rf58 return &rf
58 }59 }
59 60
61+// lossy replaces every lone surrogate (WTF-8) by U+FFFD, which is what a JSON
62+// decoder makes of the \ud800 escapes in the recorded answers.
63+func lossy(s string) string {
64+ if !hasWTF8(s) {
65+ return s
66+ }
67+ var b strings.Builder
68+ for i := 0; i < len(s); {
69+ r, w := decodeRune(s[i:])
70+ if r >= 0xD800 && r < 0xE000 {
71+ b.WriteRune(utf8.RuneError)
72+ } else {
73+ b.WriteString(s[i : i+w])
74+ }
75+ i += w
76+ }
77+ return b.String()
78+}
79+
60 // goAnswer evaluates e like the runner harness does and returns its answer in80 // goAnswer evaluates e like the runner harness does and returns its answer in
61 // the runner's format.81 // the runner's format.
62 func goAnswer(e string, ctxNames []string, ec EvalContext) runnerCase {82 func goAnswer(e string, ctxNames []string, ec EvalContext) runnerCase {
@@ -82,8 +102,8 @@ func goAnswer(e string, ctxNames []string, ec EvalContext) runnerCase {
82 return rc102 return rc
83 }103 }
84 rc.K = v.Kind().String()104 rc.K = v.Kind().String()
85- rc.S = CoerceString(v)105+ rc.S = lossy(CoerceString(v))
86- rc.J = ToJSON(v)106+ rc.J = lossy(ToJSON(v))
87 return rc107 return rc
88 }108 }
89 109
A expr/runner_oracle_wip_test.go +66 -0
@@ -0,0 +1,66 @@
1+//go:build wip
2+
3+package expr
4+
5+import (
6+ "encoding/json"
7+ "os"
8+ "sort"
9+ "testing"
10+)
11+
12+// runnerResidual lists, by reason, the corpus expressions on which this package
13+// knowingly differs from the runner. It is filled by classifyResidual.
14+func classifyResidual(g, r runnerCase) string {
15+ return ""
16+}
17+
18+func TestRunnerOracle(t *testing.T) {
19+ of, ctxValues, ctxNames := loadOracle(t)
20+ rf := loadRunner(t)
21+ if len(rf.Cases) != len(of.Cases) {
22+ t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
23+ }
24+ t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
25+ ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
26+ residual := map[string]int{}
27+ failures := 0
28+ for i, rc := range rf.Cases {
29+ if rc.E != of.Cases[i].E {
30+ t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
31+ }
32+ g := goAnswer(rc.E, ctxNames, ec)
33+ if sameAnswer(g, rc) {
34+ continue
35+ }
36+ if reason := classifyResidual(g, rc); reason != "" {
37+ residual[reason]++
38+ continue
39+ }
40+ failures++
41+ if p := os.Getenv("WIP_OUT"); p != "" {
42+ f, _ := os.OpenFile(p, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
43+ b1, _ := json.Marshal(g)
44+ b2, _ := json.Marshal(rc)
45+ _, _ = f.WriteString(string(b1) + "\n" + string(b2) + "\n")
46+ _ = f.Close()
47+ }
48+ if failures <= 30 {
49+ gj, _ := json.Marshal(g)
50+ rj, _ := json.Marshal(rc)
51+ t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
52+ }
53+ }
54+ reasons := make([]string, 0, len(residual))
55+ for k := range residual {
56+ reasons = append(reasons, k)
57+ }
58+ sort.Strings(reasons)
59+ for _, k := range reasons {
60+ t.Logf("known difference from the runner (%d cases): %s", residual[k], k)
61+ }
62+ t.Logf("%d cases, %d differences from the runner", len(rf.Cases), failures)
63+ if failures > 0 {
64+ t.Fatalf("%d mismatches with the runner", failures)
65+ }
66+}
@@ -0,0 +1,66 @@
1+//go:build wip
2+
3+package expr
4+
5+import (
6+ "encoding/json"
7+ "os"
8+ "sort"
9+ "testing"
10+)
11+
12+// runnerResidual lists, by reason, the corpus expressions on which this package
13+// knowingly differs from the runner. It is filled by classifyResidual.
14+func classifyResidual(g, r runnerCase) string {
15+ return ""
16+}
17+
18+func TestRunnerOracle(t *testing.T) {
19+ of, ctxValues, ctxNames := loadOracle(t)
20+ rf := loadRunner(t)
21+ if len(rf.Cases) != len(of.Cases) {
22+ t.Fatalf("runner answers %d cases, oracle has %d", len(rf.Cases), len(of.Cases))
23+ }
24+ t.Logf("runner: %s @ %s, %s", rf.Reference.Source, rf.Reference.Commit, rf.Reference.Runtime)
25+ ec := EvalContext{Contexts: ctxValues, Funcs: oracleFuncs, MaxResultBytes: -1}
26+ residual := map[string]int{}
27+ failures := 0
28+ for i, rc := range rf.Cases {
29+ if rc.E != of.Cases[i].E {
30+ t.Fatalf("case %d: runner expression %q, oracle %q", i, rc.E, of.Cases[i].E)
31+ }
32+ g := goAnswer(rc.E, ctxNames, ec)
33+ if sameAnswer(g, rc) {
34+ continue
35+ }
36+ if reason := classifyResidual(g, rc); reason != "" {
37+ residual[reason]++
38+ continue
39+ }
40+ failures++
41+ if p := os.Getenv("WIP_OUT"); p != "" {
42+ f, _ := os.OpenFile(p, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
43+ b1, _ := json.Marshal(g)
44+ b2, _ := json.Marshal(rc)
45+ _, _ = f.WriteString(string(b1) + "\n" + string(b2) + "\n")
46+ _ = f.Close()
47+ }
48+ if failures <= 30 {
49+ gj, _ := json.Marshal(g)
50+ rj, _ := json.Marshal(rc)
51+ t.Errorf("%q:\n go %s\n runner %s", clip(rc.E), gj, rj)
52+ }
53+ }
54+ reasons := make([]string, 0, len(residual))
55+ for k := range residual {
56+ reasons = append(reasons, k)
57+ }
58+ sort.Strings(reasons)
59+ for _, k := range reasons {
60+ t.Logf("known difference from the runner (%d cases): %s", residual[k], k)
61+ }
62+ t.Logf("%d cases, %d differences from the runner", len(rf.Cases), failures)
63+ if failures > 0 {
64+ t.Fatalf("%d mismatches with the runner", failures)
65+ }
66+}
M expr/testdata/runner.json.gz +0 -0 binary file: no text diff

binary file: no text diff

A expr/testdata/tojson/big_int.runner +1 -0
@@ -0,0 +1,1 @@
1+1.23456789012346E+19⏎̸
@@ -0,0 +1,1 @@
1+1.23456789012346E+19⏎̸
A expr/testdata/tojson/exp_20.runner +1 -0
@@ -0,0 +1,1 @@
1+1E+20⏎̸
@@ -0,0 +1,1 @@
1+1E+20⏎̸
A expr/testdata/tojson/exp_large.runner +1 -0
@@ -0,0 +1,1 @@
1+1E+21⏎̸
@@ -0,0 +1,1 @@
1+1E+21⏎̸
A expr/testdata/tojson/exp_small.runner +1 -0
@@ -0,0 +1,1 @@
1+1E-07⏎̸
@@ -0,0 +1,1 @@
1+1E-07⏎̸
A expr/testdata/tojson/max_double.runner +1 -0
@@ -0,0 +1,1 @@
1+1.79769313486232E+308⏎̸
@@ -0,0 +1,1 @@
1+1.79769313486232E+308⏎̸
A expr/testdata/tojson/min_double.runner +1 -0
@@ -0,0 +1,1 @@
1+4.94065645841247E-324⏎̸
@@ -0,0 +1,1 @@
1+4.94065645841247E-324⏎̸
A expr/testdata/tojson/overflow.runner +1 -0
@@ -0,0 +1,1 @@
1+Infinity⏎̸
@@ -0,0 +1,1 @@
1+Infinity⏎̸
A expr/testdata/tojson/precision.runner +1 -0
@@ -0,0 +1,1 @@
1+0.3⏎̸
@@ -0,0 +1,1 @@
1+0.3⏎̸

Keyboard shortcuts

Single-key shortcuts are on. ctrl k and esc work whatever this says.

ctrl k :
go to a section, branch, tag, file or repository
?
this list
j k
move the row cursor down / up
enter
open the row under the cursor; on Files, a directory opens or closes in place
↑ ↓ ← → home end
in the file tree (focus on a row): previous / next row, close / open a directory or go to its parent / first entry, first / last row. They never open a file
l h
on Files: open a directory or a file / close a directory or go up to its parent, then to the parent page. Elsewhere: into the side panel (on a merge request, its files) / back to the content; there j k move between its controls
g g G
first / last row
/
filter the list; on Files, find a file anywhere in the repository (esc clears, enter opens the best match)
b
switch branch or tag
g s g f g c g b g m g i g r g a
tabs: summary, files, commits, branches & tags, merge requests, issues, releases, actions (when available)
g h
home
w
wide mode: use the whole width of the screen, or not (remembered)
y
copy a permalink pinned to the commit
d u x
diff: split / unified / collapse or expand all
e E
diff: the whole file in view / every file, with the lines the hunks hide
n
merge request: next note in the diff
esc
close a dialog, leave a field

Branches 1

Tags 3

↑↓ moveenter openesc close