Welcome to the Tsonnet series!
If you're just joining, you can check out how it all started in the first post of the series.
In the previous post, we added support for late binding, and a little bit of Jsonnet's inconsistency rant:
We made array access by index possible, but we are still missing another indexable expr variant: String.
Let's add index access capability to it.
Getting string characters
The simplest test sample:
local name = "Tsonnet";
name[0]
$ jsonnet samples/strings/get_char_zero.jsonnet
"T"
The index out of bounds case:
local name = "Tsonnet";
name[1234]
The non-indexable index value:
local str = "Tsonnet";
local index = false;
str[index]
The Jsonnet error messages hurt my eyes:
$ jsonnet samples/errors/string_index_out_of_bounds.jsonnet
RUNTIME ERROR: Index 1234 out of bounds, not within [0, 7)
samples/errors/string_index_out_of_bounds.jsonnet:2:1-11 $
During evaluation
$ jsonnet samples/errors/string_index_not_int.jsonnet
RUNTIME ERROR: Unexpected type boolean, expected number
samples/errors/string_index_not_int.jsonnet:3:1-11 $
During evaluation
Let's get this working:
$ dune exec -- tsonnet samples/strings/get_char_zero.jsonnet
samples/strings/get_char_zero.jsonnet:2:0 Expected "Array", found "String"
2: name[0]
^^^^^^^
With a bit of copy and paste from the array implementation, it's easily adaptable. It's just a matter of using the String equivalent functions accordingly to get the values given an index:
diff --git a/lib/tsonnet.ml b/lib/tsonnet.ml
index 6431f0a..bda2b9c 100644
--- a/lib/tsonnet.ml
+++ b/lib/tsonnet.ml
@@ -115,7 +115,20 @@ let rec interpret env expr =
else Error.trace ("Index out of bounds. Trying to access index " ^ string_of_int i ^ " but \"" ^ varname ^ "\" length is " ^ string_of_int len) pos >>= error)
| expr' -> Error.trace ("Expected Integer index, got " ^ Ast.string_of_type expr') pos >>= error
)
- | evaluated_expr -> Error.trace ("Expected \"Array\", found \"" ^ string_of_type evaluated_expr ^ "\"") pos >>= error
+ | String (_, s) ->
+ let* (env', idx_expr') = interpret env' index_expr in
+ (match idx_expr' with
+ | Number (_, Int i) ->
+ (let len = String.length s in
+ if i >= 0 && i < len
+ then
+ let char_str = String.make 1 (String.get s i) in
+ ok (env', String (dummy_pos, char_str))
+ else
+ Error.trace ("Index out of bounds. Trying to access index " ^ string_of_int i ^ " but \"" ^ varname ^ "\" length is " ^ string_of_int len) pos >>= error)
+ | expr' -> Error.trace ("Expected Integer index, got " ^ Ast.string_of_type expr') pos >>= error
+ )
+ | evaluated_expr -> Error.trace (string_of_type evaluated_expr ^ " is a non indexable value") pos >>= error
)
~err:(fun err_msg -> Error.trace err_msg pos >>= error)
And now it works like a charm. Simple as that:
$ dune exec -- tsonnet samples/strings/get_char_zero.jsonnet
"T"
$ dune exec -- tsonnet samples/errors/string_index_out_of_bounds.jsonnet
samples/errors/string_index_out_of_bounds.jsonnet:2:0 Index out of bounds. Trying to access index 1234 but "name" length is 7
2: name[1234]
^^^^^^^^^^
$ dune exec -- tsonnet samples/errors/string_index_not_int.jsonnet
samples/errors/string_index_not_int.jsonnet:3:0 Expected Integer index, got Bool
3: str[index]
^^^^^^^^^^
The cram tests:
diff --git a/test/cram/errors.t b/test/cram/errors.t
index 5b9dc86..4c71d4a 100644
--- a/test/cram/errors.t
+++ b/test/cram/errors.t
@@ -59,9 +59,15 @@
[1]
$ tsonnet ../../samples/errors/value_non_indexable.jsonnet
- ../../samples/errors/value_non_indexable.jsonnet:2:0 Expected "Array", found "Int"
+ ../../samples/errors/value_non_indexable.jsonnet:2:0 Int is a non indexable value
2: answer[0]
^^^^^^^^^
[1]
+ $ tsonnet ../../samples/errors/string_index_out_of_bounds.jsonnet
+ ../../samples/errors/string_index_out_of_bounds.jsonnet:2:0 Index out of bounds. Trying to access index 1234 but "name" length is 7
+
+ 2: name[1234]
+ ^^^^^^^^^^
+ [1]
diff --git a/test/cram/strings.t b/test/cram/strings.t
index e506cb1..b23d023 100644
--- a/test/cram/strings.t
+++ b/test/cram/strings.t
@@ -3,3 +3,6 @@
$ tsonnet ../../samples/strings/concat_to_multiple_types.jsonnet
"2 apples, 42.0, true, [ 42 ], { \"answer\": 42 }"
+
+ $ tsonnet ../../samples/strings/get_char_zero.jsonnet
+ "T"
Looking good, but we can make it even better.
The indexable abstraction
There is a lot of duplication that we can encapsulate in a well crafted Indexable module:
(* lib/ast.ml *)
module Indexable = struct
let length (e : expr) =
match e with
| Array (_, exprs) -> Result.ok (List.length exprs)
| String (_, s) -> Result.ok (String.length s)
| evaluated_expr -> Result.error (string_of_type evaluated_expr ^ " is a non indexable value")
let nth (expr : expr) (index : int) =
match expr with
| Array (_, exprs) -> Result.ok (List.nth exprs index)
| String (_, s) -> Result.ok (String (dummy_pos, String.make 1 (String.get s index)))
| evaluated_expr -> Result.error (string_of_type evaluated_expr ^ " is a non indexable value")
let get (index : expr) (expr : expr) : (expr, string) result =
match index with
| Number (_, Int i) ->
let* len = length expr in
if i >= 0 && i < len
then nth expr i
else
Result.error ("Index out of bounds. Trying to access index " ^ string_of_int i ^ " but length is " ^ string_of_int len)
| expr' ->
Result.error ("Expected Integer index, got " ^ (string_of_type expr'))
end
This module provides polymorphic functions (length, nth, and get) that operate on indexable expression types, specifically the Array and String variants of the expr type.
And with that we can remove all the duplicated code in the interpreter:
diff --git a/lib/tsonnet.ml b/lib/tsonnet.ml
index bda2b9c..cba9e74 100644
--- a/lib/tsonnet.ml
+++ b/lib/tsonnet.ml
@@ -55,6 +55,8 @@ let interpret_unary_op (op: unary_op) (evaluated_expr: expr) =
| BitwiseNot, Number (pos, Int i) -> ok (Number (pos, Int (lnot i)))
| _ -> error "Invalid unary operation"
+let error_at pos = fun msg -> Error.trace msg pos >>= error
+
(** [interpret expr] interprets and reduce the intermediate AST [expr] into a result AST. *)
let rec interpret env expr =
match expr with
@@ -72,7 +74,7 @@ let rec interpret env expr =
| Ident (pos, varname) ->
Env.find_var varname env
~succ:(fun env' expr -> interpret env' expr)
- ~err:(fun err_msg -> Error.trace err_msg pos >>= error)
+ ~err:(error_at pos)
| BinOp (pos, op, e1, e2) ->
(let* (env1, e1') = interpret env e1 in
let* (env2, e2') = interpret env1 e2 in
@@ -86,11 +88,10 @@ let rec interpret env expr =
Error.trace "Invalid binary operation" pos >>= error
)
| UnaryOp (pos, op, expr) ->
- (let* (env', expr') = interpret env expr in
+ let* (env', expr') = interpret env expr in
Result.fold (interpret_unary_op op expr')
~ok:(fun expr' -> ok (env', expr'))
- ~error:(fun errmsg -> Error.trace errmsg pos >>= error)
- )
+ ~error:(error_at pos)
| Local (_, vars) ->
let acc_fun env (varname, expr) = Env.Map.add varname expr env in
let env' = List.fold_left acc_fun env vars
@@ -102,35 +103,13 @@ let rec interpret env expr =
| [expr] -> interpret env expr
| (expr :: exprs) -> interpret env expr >>= fun (env', _) -> interpret env' (Seq exprs))
| IndexedExpr (pos, varname, index_expr) ->
- Env.find_var varname env
- ~succ:(fun env' expr ->
- match expr with
- | Array (_, exprs) ->
- let* (env', idx_expr') = interpret env' index_expr in
- (match idx_expr' with
- | Number (_, Int i)->
- (let len = List.length exprs in
- if i >= 0 && i < len
- then ok (env', List.nth exprs i)
- else Error.trace ("Index out of bounds. Trying to access index " ^ string_of_int i ^ " but \"" ^ varname ^ "\" length is " ^ string_of_int len) pos >>= error)
- | expr' -> Error.trace ("Expected Integer index, got " ^ Ast.string_of_type expr') pos >>= error
- )
- | String (_, s) ->
- let* (env', idx_expr') = interpret env' index_expr in
- (match idx_expr' with
- | Number (_, Int i) ->
- (let len = String.length s in
- if i >= 0 && i < len
- then
- let char_str = String.make 1 (String.get s i) in
- ok (env', String (dummy_pos, char_str))
- else
- Error.trace ("Index out of bounds. Trying to access index " ^ string_of_int i ^ " but \"" ^ varname ^ "\" length is " ^ string_of_int len) pos >>= error)
- | expr' -> Error.trace ("Expected Integer index, got " ^ Ast.string_of_type expr') pos >>= error
- )
- | evaluated_expr -> Error.trace (string_of_type evaluated_expr ^ " is a non indexable value") pos >>= error
+ let* (env', index_expr') = interpret env index_expr in
+ Env.find_var varname env'
+ ~succ:(fun env' expr -> Result.fold (Indexable.get index_expr' expr)
+ ~ok:(fun e -> interpret env' e)
+ ~error:(error_at pos)
)
- ~err:(fun err_msg -> Error.trace err_msg pos >>= error)
+ ~err:(error_at pos)
let run (filename: string) : (string, string) result =
let env = Env.Map.empty in
The code is now cleaner and easier to understand, isn't it?
Conclusion
With string indexing now implemented, Tsonnet can handle both arrays and strings as indexable types. The refactoring into an Indexable module not only eliminated code duplication but also created a clean abstraction that will make it easier to add support for other indexable types in the future. The error handling remains consistent and informative, helping developers understand what went wrong when things don't work as expected.
The diff with all the changes can be seen here.
There's still something bothering me about testing, but I will leave it as a surprise for the next post.
Thanks for following along as we build Tsonnet from scratch! Don't let your inbox be a non-indexable value -- subscribe to get more posts about interpreters, compilers, and the occasional rant about inconsistent language design. No out-of-bounds errors, just quality content delivered straight to your inbox!
Photo by Maksym Kaharlytskyi on Unsplash