JSON, files & HTTP

This reference describes the implemented native runtime. Historical paths and internal version labels are retained where relevant. Later extension chapters supersede older limitations.

These extend the existing optimized interpreter; they do not replace its scalar strings, numeric/text caches, arrays, function frames or expression evaluator. Existing function name argc, %1, flags and commands remain. The added reserved function/parameter names are args input json map try catch http. No classes, JIT, shell execution or package ecosystem is introduced.

Build and verification

Linux, C++17, GCC 13 (or a compatible compiler), Make, pkg-config, nlohmann JSON headers and libcurl development libraries are required. HTTP requires libcurl with asynchronous DNS and the string protocol options (libcurl 7.85 or newer). The verified local dependencies are Ubuntu libcurl 8.5.0 and nlohmann JSON 3.11.3. Python 3 and the openssl executable are needed for tests; neither is invoked by the interpreter.

sudo apt-get install g++-13 make pkg-config nlohmann-json3-dev libcurl4-openssl-dev libsqlite3-dev python3 openssl
cd /root/vanilla1
make release sanitize
make test
make test-sanitize
python3 tests/run_utilities_demo.py ./build/vanilla1

The Makefile links libcurl directly; there is no runtime shell/curl subprocess. Tests create actual HTTP and self-signed HTTPS listeners on loopback only. Tests use temporary working directories, bounded subprocesses, and no external credentials or services. Both full test targets now require a root harness for the real permission-denial suite, which drops every candidate subprocess to nobody with empty supplementary groups; /usr/bin/python3 and a traversable project path are required. Missing prerequisites fail, not skip. See docs/UTILITIES-SPEC-FOLLOWUP.md for the 16 denial checks, current measurements and remaining performance limitation. docs/utilities-spec-verification.txt contains the final suite output; docs/utilities-spec-build.txt contains rebuild output. The earlier utility run remains in docs/utilities-verification.txt. docs/utilities-demo-output.txt is actual demo stdout. benchmarks/utilities_smoke.py reruns only the fixed 408738-iteration smoke against the preserved pre-utilities binary, on the same available CPU (one warmup plus seven measurements); raw results are in docs/utilities-smoke.json.

1. CLI arguments and keyboard/stdin

vanilla1 [--allow-files] [--allow-network] [--max-steps N] run FILE [-- ARG ...]
args count $dest
args get $dest index
args all $array
input $line $eof

The separator -- must occur after run FILE. Everything following it is a script argument, even --help, --allow-files, run, or another --. Earlier flags retain their old interpretation. Script arguments do not include the executable or source filename. args get is zero-based; function %1 remains one-based. Script arguments are visible from any function. args all creates an ordinary array of strings. Missing/out-of-range argument access is an error, not an empty string. CLI input is capped at 1024 arguments and 1 MiB aggregate argument bytes.

input reads one line from stdin, which may be a keyboard, pipe, or redirected file. It removes LF and an immediately preceding CR; a final CR without LF remains data. Empty lines are data ($line="", $eof="0"). A final unterminated line is data too. On the following read, $line="" and $eof="1"; subsequent reads keep returning EOF. NUL bytes are preserved in line contents. The two destinations must be different and already declared. Destinations are checked before consuming input. There is no prompt parameter: use print first; the standard stdin/stdout tie flushes a prompt before keyboard input.

Reads are truly incremental, with a 65536-byte line limit before stripping CR (LF is excluded) and a 16 MiB cumulative consumed-input budget including delimiters. Oversize input is a fatal resource error, not a truncated line. Stdin may block waiting for a person; instruction limits are not wall-time limits.

2. Maps and typed nested JSON

Ordinary scalars retain exact text spelling and existing binary64 arithmetic behavior. JSON is a separate typed value, including primitive JSON values. JSON arrays are distinct from legacy string arrays. You cannot implicitly print or do math on a JSON value: convert explicitly. Every destination still needs var first.

Keys may be any string, including empty strings, UTF-8 and embedded NUL. Objects serialize in sorted key order. json get requires a structured source; an index must be integral and in bounds. Object keys do not interpret dots, slashes or JSON Pointer syntax. Chain gets for nesting. To change a nested object, get it, change it, then set it back in its parent. JSON arrays can be parsed, traversed, nested and serialized; this minimal API does not add array append/index assignment for typed JSON arrays. Legacy array commands remain unchanged.

var root child value text
json parse $root "{\"settings\":{\"enabled\":true},\"items\":[1,null]}"
map get $child $root settings
json parse $value false
map set $child enabled $value
map set $root settings $child
json stringify $text $root
print $text "\n"

map set ... true stores the string "true". To store a Boolean use json parse $v true, then map set ... $v. Assignments replace old scalar/array/JSON storage and invalidate numeric caches. Alias operations such as map get $x $x key and json stringify $x $x work. Gets copy only the selected subtree, not an unrelated whole source tree. Map mutation is an explicit bounded copy and never creates reference cycles; map set $m child $m stores a snapshot. Later mutation of a selected child does not mutate its original parent. Arguments and returns are still scalar-only; pass serialized JSON to functions when needed.

JSON correctness and resource policy

• nlohmann JSON performs parsing/escaping/UTF-8 and surrogate validation. Trailing non-whitespace, malformed input, nonfinite numbers, and duplicate object keys are rejected. Duplicate detection uses decoded keys, at every nesting level ("a" and "\u0061" collide). Comments and trailing commas are not accepted.
• Integer-valued numbers outside [-9007199254740991, 9007199254740991] are rejected, including integers represented with an exponent, oversized unsigned integers, and integers that the JSON library would otherwise promote to floating point. Safe integer tokens round-trip exactly. This deliberately forbids silently losing integer precision when converting to the interpreter's binary64 arithmetic.
• Fractional/exponent decimals use finite IEEE binary64, not arbitrary-precision decimal. Rounding and underflow-to-zero follow the JSON library. Original numeric spelling is not retained; 1.0 and 1 have the same reported number type but may serialize differently. A floating result that is integral outside the safe range is rejected. Keep money/IDs requiring arbitrary decimal precision as JSON strings. The legacy scalar numeric parser and its underflow behavior are unchanged.
• JSON strings constructed from binary scalar data may contain invalid UTF-8; serialization rejects such strings rather than replacing bytes. Parsed JSON strings are validated. NUL/control characters in valid strings are escaped on output. Library diagnostic excerpts replace invalid/control bytes with ? so errors remain readable.
• Input and serialized output are each capped at 1 MiB. Serialization uses a bounded stream buffer instead of first constructing an unchecked expanded result. Depth is at most 64 (root depth zero), and nodes at most 16384. The conservative 1 MiB structured-storage budget charges 128 bytes per node, string/key capacity, and array capacity; it can reject wide JSON before the node cap. Structured storage counts toward the existing 2 MiB frame budget. Explicit bounded temporary copies are possible; these are accounting limits, not an exact process RSS promise. Resource failures cannot be caught.

3. Recoverable errors

var error result
try
    math $result 1 / 0
catch $error
    print "Recovered: " $error "\n"
end try

Exactly one catch is required per try. The handler destination must already exist when entering the try. On success, the catch body is skipped. A recoverable runtime error transfers to the nearest enclosing active try body, including through function calls. The destination receives source-file:line: error: message, identifying the original failing instruction. Execution continues after end try when the handler finishes. An error inside a catch can be caught by an outer try, not by that same catch. Nested loops inside an abandoned try are unwound; enclosing loops remain active. break, continue and return retain their existing meanings. A try is not a loop. goto cannot cross try boundaries or cross between the protected body and its catch.

Recoverable errors include ordinary type/index/domain failures, missing map keys, invalid JSON input, denied file/network feature gates, file I/O failures, and HTTP transport/TLS/timeouts. Resource exhaustion is explicitly tagged fatal: steps, calls, source/expression/data/output/input/JSON/handle/network budgets cannot be caught to evade their limits. exit is also not catchable and still ends the entire program successfully. Host allocation failures are not converted into recoverable script errors.

Commands, argument counts and structural syntax in every function are validated at load time; these errors occur before any try can execute. The existing math expression evaluator remains lazy for compatibility: malformed expression text is a runtime expression error when evaluated, including its existing diagnostic precedence. Catching is not a transaction: earlier prints, variable assignments, file writes and requests are not rolled back. Streams explicitly opened inside a try remain open until closed or the interpreter ends.

4. Streaming files, directory listing, safe copy/rename

All operations in this section require --allow-files, independently of the network flag. Existing file read/write/append/exists retain their syntax and behavior.

file open $handle path
file readline $line $eof handle
file close handle
file list $array directory
file copy source destination
file rename source destination

open opens an existing regular file for binary reading and returns an opaque decimal handle. Handles are script-wide (can be passed as scalars), monotonically allocated, and never reused during a run. At most 32 are open at once. readline reads from its current position, incrementally, not by reading/splitting the whole file. Its distinct output destinations and LF/CRLF/EOF behavior match input. Each line is limited to 64 KiB; all streaming file readers share a separate cumulative 16 MiB consumed-byte budget. Empty files report EOF on the first read. close releases a handle; double close or use after close is an error. Interpreter destruction closes remaining handles; returning from a function does not implicitly close script-wide handles.

list returns sorted entry names, not full paths, as a legacy string array; no recursion, . or ... Files, directories and symlinks are included. Missing/non-directory paths and iteration errors are reported. Listings are capped at 16384 entries and 1 MiB accounted name data; the frame budget still applies. Listings are snapshots of names, not stable directory handles.

copy and rename never clobber an existing destination, including a dangling symlink or the source itself. There is no overwrite mode:

• Copy opens the source with O_NOFOLLOW|O_NONBLOCK and verifies a regular file using fstat. It creates the destination with O_CREAT|O_EXCL|O_NOFOLLOW, permission mode 0600 subject to umask, and copies in 8192-byte chunks. It preserves bytes, not permissions/timestamps/ownership. Source symlinks and directories are rejected. Copied bytes share the existing cumulative 16 MiB file-write budget. Failed copy can leave a partial newly created destination; it is deliberately not unlinked by pathname because a concurrent process could replace that pathname. Writes/close errors are checked.
• Rename checks a regular non-symlink source and uses Linux renameat2(RENAME_NOREPLACE) for atomic no-clobber destination semantics. It does not fall back to unsafe check-then-overwrite rename or cross-filesystem copy/delete. Cross-filesystem moves, unsupported kernel/filesystem primitives and permission errors fail safely. Rename does not consume the content-write byte budget.

Neither feature flag is a sandbox. Paths can be absolute and can traverse symlinked parent directories. Regular checks for stream opens, legacy operations, and rename sources can race concurrent changes; no race-proof confinement or hostile-filesystem wall-time guarantee is claimed. Existing file write still truncates and file append still appends. Sources/includes are always readable independently of --allow-files.

5. Bounded HTTP/S

http get  $body $status url timeout_ms max_response_bytes
http post $body $status url timeout_ms max_response_bytes request_body

--allow-network is required, separately from --allow-files. Both output destinations must already exist and must differ. GET has no request body. POST sends the provided binary-safe scalar body using libcurl's default POST content type (application/x-www-form-urlencoded); this minimal API has no custom headers or JSON content-type switch. Status is an ordinary decimal scalar. HTTP statuses including 3xx/4xx/5xx are normal successful transport results, not exceptions. Body/status are assigned only after a successful bounded transfer; transport errors preserve their previous values.

Security/bounds policy:

• Only URLs beginning with lowercase http:// or https://, at most 8192 bytes and with no NUL. Libcurl's own allowed-protocol list independently restricts transfers to HTTP/S. URL username/password credentials are forbidden.
• Redirects are disabled, including redirects to HTTP/S. A 302 remains a 302 and its response body can be inspected; a file: redirect is never followed.
• TLS certificate-chain/peer and hostname verification are always on. There is no insecure flag or script-specified CA override; the installed libcurl/system trust store is used. A real self-signed loopback TLS endpoint is rejected by the regression suite. That negative integration test proves rejection, not a complete TLS implementation audit.
• Proxy environment settings are deliberately ignored (CURLOPT_PROXY=""); requests connect directly. No automatic decompression is enabled. HTTP response bodies are returned as bytes, not decoded text. No cookies, authentication management, shell commands or subprocesses.
• Each call requires an integer total timeout in 1..30000 ms; connect timeout is bounded by the same value. Async DNS support is required so name resolution is not an unbounded synchronous phase. The caller's response cap must be 1..1048576 bytes. Headers have an independent cumulative 64 KiB per response cap, including intermediate headers. Oversized body/header responses fail fatally rather than returning partial data.
• Per interpreter run: at most 64 attempted transfers, at most 60000 ms summed requested timeout reservations (even fast successes/failed transfers consume their reservation), and 16 MiB combined request-body/received-body bytes. Header traffic is independently bounded by the per-call header cap and request count. Request bodies remain subject to the scalar 1 MiB limit. A transfer cannot exceed the remaining aggregate response-body allowance.
• Resource budgets cannot be reset/caught. Transport failures/timeouts can be caught. These caps limit library operations; scheduling, kernels, filesystems, stdin, and the entire script are not given a hard process-wide deadline. Keep an external timeout for untrusted workloads.

Opting in allows private networks, loopback, metadata services and arbitrary destinations reachable by the process. This is not SSRF prevention or a sandbox. Run trusted scripts or apply OS/network isolation. Source includes and environment access remain separate existing capabilities.

6. Named function parameters, retaining legacy calls

function add a b
    var result
    math $result $a + %2
    return $result
kill add

Named parameters are bare identifiers, not $dest forms. Each is automatically initialized with its argument value/type (see WORKLOG.md for structured calls); var a thereafter is a no-op. %1/%2 still read the original scalar arguments even if $a/$b are changed. Recursion gets independent parameter variables and positional arguments. Call syntax and exact-arity checks are unchanged: call $result add 2 3 or add 2 3 to discard the return value.

function legacy 2 still declares two positional arguments with no named locals. A declaration with one non-identifier token retains the legacy numeric-argc interpretation and range checks. function noargs is the named form with zero parameters; function noargs 0 also works. main must still take zero function arguments; use the args commands for CLI input.

Duplicate, quoted, invalid or reserved parameter names are rejected before execution, even for unused functions. Named parameters obey the existing identifier and source-token limits. Parameters may be reassigned to scalars, arrays or JSON like other local variables; typed arguments and explicit returnvalue results are now supported by the work-log extension; legacy return remains scalar concatenation.

Full runnable demo

examples/utilities.v is the complete concise script. Run the supplied wrapper:

cd /root/vanilla1
python3 tests/run_utilities_demo.py ./build/vanilla1

It starts a disposable loopback HTTP fixture on a free port, pipes Ada\n to stdin, passes the URL after --, runs the actual script in an empty temporary directory with both opt-in flags, verifies exact stdout and copied/renamed file contents, then removes the temporary directory and stops the fixture. It does not serve or write in the repository's working directory. The underlying interpreter invocation is:

/path/to/vanilla1 --allow-files --allow-network run /root/vanilla1/examples/utilities.v -- http://127.0.0.1:FIXTURE_PORT

Use the wrapper for an executable recipe (the port above is explanatory, not a promised running service). Exact stdout is saved in docs/utilities-demo-output.txt and asserted by the wrapper.