Swift DynamicJSON

DynamicJSON is a framework for representing, querying, and manipulating generic JSON values. The framework provides:

• A generic representation of JSON values as defined by RFC 8259.
• A natural embedding of functionality for creating and manipulating JSON values into the Swift programming language, including support for reading and writing JSON data and for converting typed and untyped JSON representations.
• An implementation of JSON Pointer as defined by RFC 6901 for locating values within a JSON document.
• An implementation of JSON Path as defined by RFC 9535 for querying JSON data.
• An implementation of JSON Patch as defined by RFC 6902 for mutating JSON data.
• An implementation of JSON Merge Patch as defined by RFC 7396 for merging JSON data with JSON patches.
• An implementation of JSON Schema as defined by the 2020-12 Internet Draft specification for validating JSON data.
• Reading of JSON streams supporting NDJSON/JSON Lines, JSON text sequences as defined by RFC 7464, concatenated JSON, and the elements of large top-level arrays.

Table of contents
1.  Representing JSON Data 2.  Accessing JSON Values    2.1  JSON Location    2.2  JSON Pointer 3.  Queries with JSON Path 4.  Mutating JSON Values    4.1  Mutation API    4.2  JSON Patch 5.  Merging JSON Values    5.1  Symmetrical Merge    5.2  Overriding Merge    5.3  JSON Merge Patch	6.  Validating JSON Data    6.1  Implementation Overview    6.2  Validation API    6.3  Metadata and Defaults 7.  Streaming JSON Values    7.1  Streaming from Web APIs and LLMs    7.2  Writing Streams of JSON Values    7.3  Extracting JSON from LLM Output    7.4  Demo: WikiWatch 8.  Building the Documentation 9.  Requirements 10.  Migrating from the Swift 5 Version

 

Representing JSON Data

All JSON values in framework DynamicJSON are represented with enumeration JSON. Enumeration JSON defines the following cases:

indirect enum JSON: Hashable, Codable, CustomStringConvertible, ... {
  case null
  case boolean(Bool)
  case integer(Int64)
  case float(Double)
  case string(String)
  case array([JSON])
  case object([String : JSON])
  ...
}

JSON values can be easily constructed using Swift literal syntax. Here is an example for the initialization of a small JSON-based data structure:

let json0: JSON = [
  "foo": true,
  "str": "one two",
  "object": [
    "value": nil,
    "arr": [1, 2, 3],
    "obj": [ "x" : 17.6 ]
  ]
]

There are also initializers to convert JSON encoded data in the form of a String or a Data object into a JSON enumeration. The following code initializes a JSON value from a JSON encoded value in a String literal.

let json1 = try JSON(string: """
  {
    "foo": true,
    "str": "one two",
    "object": {
      "value": null,
      "arr": [1, 2, 3],
      "obj": { "x" : 17.6 }
    }
  }
""")

Any encodable type can be converted into a JSON value using the initializer init(encodable:). Alternatively, init() can be used. This is the most generic initializer which can also coerce basic types like Bool, Int, String, etc. into JSON.

struct Person: Codable {
  let name: String
  let age: Int
  let children: [Person]
}
let person = Person(name: "John", age: 34,
                    children: [ Person(name: "Sofia", age: 5, children: []) ])
let json2 = try JSON(encodable: person)
print(json2.description)

Executing this code will print the following JSON-based representation of Person:

{
  "age" : 34,
  "children" : [
    {
      "age" : 5,
      "children" : [],
      "name" : "Sofia"
    }
  ],
  "name" : "John"
}

It is also possible to do the inverse, and convert a JSON-based representation into a strongly typed data structure via method coerce().

let json3: JSON = [
  "name": "Matthew",
  "age": 29,
  "children": []
]
let person2: Person = try json3.coerce()

Accessing JSON Values

A JSON value within a larger JSON document can be identified and accessed using dynamic member lookup as if the data was fully structured, e.g. by representing it as a struct. Here are several examples showcasing the different ways how to access the first element of array arr in object of json1. All expressions return the JSON value 1.

• Dynamic member lookup: json1.object?.arr?[0]
• Keypath lookup: json1[keyPath: \.object?.arr?[0]]
• Subscript lookup: json1["object"]?["arr"]?[0]
• Reference lookup:
  • Using JSON Pointer string: try json1[ref: "/object/arr/0"]
  • Using JSON Path string: try json1[ref: "$.object.arr[0]"]
  • Using implementations of JSONReference, such as JSONPointer and JSONPath: json1[ref: p], where p is an object of type JSONReference

In DynamicJSON, components of a JSON value are identified by implementations of the protocols JSONReference and SegmentableJSONReference. The following code presents the core methods implementing JSON references:

protocol JSONReference: CustomStringConvertible, Sendable {
  // Returns a new JSONReference with the given member selected.
  func select(member: String) -> Self
  // Returns a new JSONReference with the given index selected.
  func select(index: Int) -> Self
  // Retrieve value at which this reference is pointing from JSON document `value`.
  func get(from value: JSON) -> JSON?
  // Replace value at which this reference is pointing with `json` within `value`.
  func set(to json: JSON, in value: JSON) throws -> JSON
  // Mutate value at which this reference is pointing within JSON document `value`
  // with function `proc`.
  func mutate(_ json: inout JSON, with proc: (inout JSON) throws -> Void) throws
}

protocol SegmentableJSONReference: JSONReference, JSONLocationConvertible {
  associatedtype Segment: JSONReferenceSegment
  // An array of segments representing the reference.
  var segments: [Segment] { get }
  // Creates a new `SegmentableJSONReference` on top of this reference.
  func select(segment: Segment) -> Self
  // Decomposes this reference into the top segment selector and its parent.
  var deselect: (Self, Segment)? { get }
}

DynamicJSON currently provides two implementations of SegmentableJSONReference: JSONPointer and JSONLocation, an abstraction that is equivalent to singular JSON Path queries.

JSON Location

JSONLocation is the default implementation for identifying JSON values within a JSON document. It is based on how values are identified in JSON Path and uses a restricted form of JSON Path query syntax.

A JSONLocation value is defined in terms of a sequence of member names and array indices used to navigate through the structure of a JSON document. JSONLocation references refer to at most one value within a JSON document. The following code summarizes how JSONLocation values are represented:

indirect enum JSONLocation: SegmentableJSONReference, Codable, Hashable, Sendable {
  case root
  case member(JSONLocation, String)
  case index(JSONLocation, Int)
  
  enum Segment: JSONReferenceSegment, Codable, Hashable, CustomStringConvertible {
    case member(String)
    case index(Int)
    ...
  }
  ...
}

A JSON location is a path to an element in a JSON structure. Each element of the path is called a segment. The JSON location syntax supports two different forms to express such sequences of segments. Each sequence starts with $ indicating the "root" of a JSON document. The most common form for expressing the segment sequence is using the dot notation:

$.store.book[0].title

While accessing an array index is always done using bracket notation, it is possible to also express the access of members of an object using bracket notation:

$['store']['book'][0]['title']

Is is also possible to mix the dot and bracket notation. Dots are only used before property names and never together with brackets:

$['store'].book[-1].title

The previous example also shows the usage of negative indices, which are interpreted as offsets from the end of arrays with -1 referring to the last element.

The JSONLocation API supports multiple initializers for creating JSON location references:

let r1 = try JSONLocation("$['store']['book'][0]['title']")
let r2 = JSONLocation(segments: [.member("store"),
                                 .member("book"),
                                 .index(0),
                                 .member("title")])

JSONLocation defines the following frequently used methods:

indirect enum JSONLocation: SegmentableJSONReference, ... {
  // The segments defining this `JSONLocation`.
  var segments: [Segment]
  // Returns a new JSONLocation with the given member selected.
  func select(member: String) -> JSONLocation
  // Returns a new JSONLocation with the given index selected.
  func select(index: Int) -> JSONLocation
  // Returns a new JSONLocation by appending the given segment.
  func select(segment: Segment) -> JSONLocation
  // Returns a matching `JSONPointer` reference if possible.
  var pointer: JSONPointer?
  // Returns a matching `JSONPath` query.
  var path: JSONPath
  // Retrieve value at this location within `value`.
  func get(from value: JSON) -> JSON?
  // Replace value at this location within `in` with `value`.
  func set(to json: JSON, in value: JSON) throws -> JSON
  // Mutate value at this location within `value` with function `proc`.
  // `proc` is provided a reference, enabling efficient in-place mutations.
  func mutate(_ json: inout JSON, with proc: (inout JSON) throws -> Void) throws
}

JSON Pointer

JSON Pointer is specified by RFC 6901 and is generally the most established formalism for referring to a JSON value within a JSON document. JSON Pointer is intended to be easily expressed in JSON string values as well as Uniform Resource Identifier (URI) fragment identifiers (see RFC 3986).

Like JSON Locations, each JSON Pointer specifies a path to an element in a JSON structure starting with its root. Each element of the path either refers to an object member or an array index. Syntactically, each path element is prefixed with "/". JSON Pointer uses "~1" to encode "/" in member names and "~0" to encode "~". The empty string refers to the root of the JSON document. Here is an example:

/store/book/0/title

JSON Pointer neither supports forcing an element such as "/0" to refer to an array index, nor does it allow for negative indices (as an offset from the end of the array). All numeric path elements such as "0" above can either match an array and select index 0 or they match an object member "0". Thus, there is no general way to map JSON Location into JSON Pointer or vice versa.

Struct JSONPointer implements the JSON Pointer standard in the following way:

struct JSONPointer: SegmentableJSONReference, Codable, Hashable, Sendable {
  let segments: [ReferenceToken]
  
  enum ReferenceToken: JSONReferenceSegment, Hashable, CustomStringConvertible {
    case member(String)
    case index(String, Int?)
    ...
  }
  ...
}

The JSONPointer API supports multiple initializers for creating JSON Pointer references:

let p1 = try JSONPointer("/store/book/0/title")
let p2 = JSONPointer(components: ["store", "book", "0", "title"])

JSONPointer defines the following frequently used methods:

struct JSONPointer: SegmentableJSONReference, ... {
  // Returns this JSONPointer as an array of reference tokens.
  var segments: [ReferenceToken]
  // Returns a new JSONPointer with the given member selected.
  func select(member: String) -> JSONPointer
  // Returns a new JSONPointer with the given index selected.
  func select(index: Int) -> JSONPointer
  // Constructs a new JSONPointer by appending the given segment to this pointer.
  func select(segment: ReferenceToken) -> JSONPointer
  // Decomposes this JSONPointer into a parent pointer and a selector reference token.
  var deselect: (JSONPointer, ReferenceToken)?
  // The reference tokens defining this `JSONPointer` value.
  var components: [String]
  // Returns all JSON locations corresponding to this `JSONPointer`.
  func locations() -> [JSONLocation]
  // Retrieve value at which this reference is pointing from JSON document `value`.
  func get(from value: JSON) -> JSON?
  // Replace value at which this reference is pointing with `json` within `value`.
  func set(to json: JSON, in value: JSON) throws -> JSON
  // Mutate value at this location within `value` with function `proc`. `proc`
  // is provided a reference, enabling efficient, in-place mutations.
  func mutate(_ json: inout JSON, with proc: (inout JSON) throws -> Void) throws
}

Queries with JSON Path

DynamicJSON supports the full JSON Path standard as defined by RFC 9535. Enum JSONPath represents JSON Path queries. JSON provides query() methods to apply a JSON Path query to a JSON value.

To illustrate the usage of JSON Path queries, the following JSON value is being defined (this is the example from RFC 9535):

let jval = try JSON(string: """
  { "store": {
      "book": [
        { "category": "reference",
          "author": "Nigel Rees",
          "title": "Sayings of the Century",
          "price": 8.95 },
        { "category": "fiction",
          "author": "Evelyn Waugh",
          "title": "Sword of Honour",
          "price": 12.99 },
        { "category": "fiction",
          "author": "Herman Melville",
          "title": "Moby Dick",
          "isbn": "0-553-21311-3",
          "price": 8.99 },
        { "category": "fiction",
          "author": "J. R. R. Tolkien",
          "title": "The Lord of the Rings",
          "isbn": "0-395-19395-8",
          "price": 22.99 }
      ],
      "bicycle": {
        "color": "red",
        "price": 399
      }
    }
  }
  """)

Now a JSON Path query $.store.book[?@.price < 10].title can be defined by using the JSONPath(query:strict:) initializer. Finally, the path can be applied to jval by invoking its query() method. The result is an array of LocatedJSON values matching the query within jval. LocatedJSON combines a location where a value was found with the value at that location into one object.

let path = try JSONPath(query: "$.store.book[?@.price < 10].title")
var results = try value.query(path)
for result in results {
  print(result)
}

This is the output generated from this code. It prints two LocatedJSON objects for the two values within jval matching the query path.

$['store']['book'][0]['title'] => "Sayings of the Century"
$['store']['book'][2]['title'] => "Moby Dick"

If only locations or only values are needed as a result of evaluating a JSON Path query, then it is possible to use the query(locations:) or query(values:) methods of JSON.

The above API supports the default JSON Path query language. JSON Path has a built-in extensibility mechanism that lets one add custom functions, applicable in query filters. This can be achieved by extending class JSONPathEnvironment and overriding method initialize(). Such an extended environment can then be passed to the initializer of struct JSONPathEvaluator, which provides a means to execute queries using the extended environment.

Mutating JSON Values

Mutation API

DynamicJSON represents JSON data with value type JSON. There are a number of methods that mutate such data without copies being created. These are listed in the code snippet below.

enum JSON: Hashable, ... {
  // Mutates this JSON value if it represents either an array or a string by
  // appending the given JSON value `json`. For arrays, `json` is appended as a
  // new element. For strings it is expected that `json` also refers to a string
  // and `json` gets appended as a string. For all other types of JSON values,
  // an error is thrown.
  mutating func append(_ json: JSON) throws
  
  // Mutates this JSON value if it represents either an array or a string by
  // inserting the given JSON value `json`. For arrays, `json` is inserted as a
  // new element at `index`. For strings it is expected that `json` also refers to
  // a string and `json` gets inserted into this string at position `index`. For
  // all other types of JSON values, an error is thrown.
  mutating func insert(_ json: JSON, at index: Int) throws
  
  // Adds a new key/value mapping or updates an existing key/value mapping in this
  // JSON object. If this JSON value is not an object, an error is thrown.
  mutating func assign(_ member: String, to json: JSON) throws
  
  // Replaces the value the location reference `ref` is referring to with `json`.
  // The replacement is done in place, i.e. it mutates this JSON value. `ref` can
  // be implemented by any abstraction implementing the `JSONReference` procotol.
  mutating func update(_ ref: JSONReference, with json: JSON) throws
  
  // Replaces the value the location reference string `ref` is referring to with
  // `json`. The replacement is done in place, i.e. it mutates this JSON value.
  // `ref` is a string representation of either `JSONLocation` or `JSONPointer`
  // references.
  mutating func update(_ ref: String, with json: JSON) throws
  
  // Mutates the JSON value the reference `ref` is referring to with function
  // `proc`. `proc` receives a reference to the JSON value, allowing efficient in
  // place mutations without automatically doing any copying. `ref` can be
  // implemented by any abstraction implementing the `JSONReference` procotol.
  mutating func mutate(_ ref: JSONReference,
                       with proc: (inout JSON) throws -> Void) throws
  
  // Mutates the JSON value the reference `ref` is referring to with function
  // `arrProc` if the value is an array or `objProc` if the value is an object. For
  // all other cases, an error is thrown. This method allows for efficient in place
  // mutations without automatically doing any copying. `ref` can be implemented by
  // any abstraction implementing the `JSONReference` procotol.
  mutating func mutate(_ ref: JSONReference,
                       array arrProc: ((inout [JSON]) throws -> Void)? = nil,
                       object objProc: ((inout [String : JSON]) throws -> Void)? = nil,
                       other proc: ((inout JSON) throws -> Void)? = nil) throws
  
  // Mutates the JSON value the reference string `ref` is referring to with function
  // `proc`. `proc` receives a reference to the JSON value, allowing efficient in
  // place mutations without automatically doing any copying. `ref` is a string
  // representation of either `JSONLocation` or `JSONPointer` references.
  mutating func mutate(_ ref: String, with proc: (inout JSON) throws -> Void) throws
  
  // Mutates the JSON array the reference string `ref` is referring to with function
  // `arrProc` if the value is an array or `objProc` if the value is an object. For
  // all other cases, an error is thrown. This method allows for efficient in place
  // mutations without automatically doing any copying. `ref` is a string
  // representation of either `JSONLocation` or `JSONPointer` references.
  mutating func mutate(_ ref: String,
                       array arrProc: ((inout [JSON]) throws -> Void)? = nil,
                       object objProc: ((inout [String : JSON]) throws -> Void)? = nil,
                       other proc: ((inout JSON) throws -> Void)? = nil) throws
  ...
}

The most generic form of mutation is provided by the following two methods:

mutating func mutate(_ ref: JSONReference,
                     with proc: (inout JSON) throws -> Void) throws
mutating func mutate(_ ref: JSONReference,
                     array arrProc: ((inout [JSON]) throws -> Void)? = nil,
                     object objProc: ((inout [String : JSON]) throws -> Void)? = nil,
                     other proc: ((inout JSON) throws -> Void)? = nil) throws

These methods mutate the JSON value at which the reference ref is referring to via function proc. proc receives a reference to this JSON value, allowing efficient, in place mutations without automatically creating copies.

The second form of the mutate method provides specific functions arrProc for mutating arrays and objProc for mutating objects, again in a way in which no copies are created. For all other values, proc is being called.

JSON Patch

JSON Patch defines a JSON document structure for expressing a sequence of operations to apply to a JSON document. Each operation mutates parts of the JSON document. The supported operations specified by RFC 6902 are implemented by enum JSONPatchOperation:

enum JSONPatchOperation: Codable, Hashable, CustomStringConvertible, CustomDebugStringConvertible {
  // add(path, value): Add `value` to the JSON value at `path`
  case add(JSONPointer, JSON)
  // remove(path): Remove the value at location `path` in a JSON value.
  case remove(JSONPointer)
  // replace(path, value): Replace the value at location `path` with `value`.
  case replace(JSONPointer, JSON)
  // move(path, from): Move the value at `from` to `path`. This is equivalent
  // to first removing the value at `from` and then adding it to `path`.
  case move(JSONPointer, JSONPointer)
  // copy(path, from): Copy the value at `from` to `path`. This is equivalent
  // to looking up the value at `from` and then adding it to `path`.
  case copy(JSONPointer, JSONPointer)
  // test(path, value): Compares value at `path` with `value` and fails if the
  // two are different.
  case test(JSONPointer, JSON)
  ...
}

Struct JSONPatch bundles operations together into a "patch object" providing functionality to apply the patch to JSON values:

struct JSONPatch: Codable, Hashable, CustomStringConvertible, CustomDebugStringConvertible {
  // Sequence of operations.
  let operations: [JSONPatchOperation]
  // Initializer based on a sequence of operations
  init(operations: [JSONPatchOperation]) { ... }
  // Initializer based on a target and source JSON object
  init(from: JSON, to: JSON, via maker: () -> JSONPatchMaker = JSONPatchMaker.init)
  // Decodes the provided data with the given decoding strategies.
  init(data: Data,
       dateDecodingStrategy: JSONDecoder.DateDecodingStrategy = .deferredToDate,
       floatDecodingStrategy: JSONDecoder.NonConformingFloatDecodingStrategy = .throw,
       userInfo: [CodingUserInfoKey : any Sendable]? = nil) throws { ... }
  // Decodes the provided string with the given decoding strategies.
  init(string: String,
       dateDecodingStrategy: JSONDecoder.DateDecodingStrategy = .deferredToDate,
       floatDecodingStrategy: JSONDecoder.NonConformingFloatDecodingStrategy = .throw,
       userInfo: [CodingUserInfoKey : any Sendable]? = nil) throws { ... }
  // Decodes the content at the provided URL with the given decoding strategies.
  init(url: URL,
       dateDecodingStrategy: JSONDecoder.DateDecodingStrategy = .deferredToDate,
       floatDecodingStrategy: JSONDecoder.NonConformingFloatDecodingStrategy = .throw,
       userInfo: [CodingUserInfoKey : any Sendable]? = nil) throws { ... }
  ...
  // Applies this patch object to `json` mutating `json` in place.
  func apply(to json: inout JSON) throws { ... }
  ...
}

The following code shows how to load a JSON patch snippet into a patch object and apply it to a json value:

let jsonstr = """
  [
    { "op": "test", "path": "/a/b/c", "value": "foo" },
    { "op": "remove", "path": "/a/b/c" },
    { "op": "add", "path": "/a/b/c", "value": [ "foo", "bar" ] },
    { "op": "replace", "path": "/a/b/c", "value": 42 },
    { "op": "move", "from": "/a/b/c", "path": "/a/b/d" },
    { "op": "copy", "from": "/a/b/d", "path": "/a/b/e" }
  ]
  """
let patch = try JSONPatch(string: jsonstr)
var json: JSON = ...
try json.apply(patch: patch)

Given two JSON values source and target, the expression source.patch(to: target) returns a JSONPatch object that when applied to source, transforms source into target. Function patch generates JSONPatch objects that perform the required transformation, but they are currently not optimized.

Merging JSON Values

Symmetrical Merge

The method isRefinement(of:) of enum JSON defines a relationship between two JSON values. a.isRefinement(of: b) is true if

1. Both a and b are JSON values of the same type,
2. If a and b are arrays, they have the same length n and a[i].isRefinement(of: b[i]) holds for every i ∈ [0; n[,
3. If a and b are objects, for every member m of b with value b[m], there is a member m of a with value a[m] such that a[m].isRefinement(of: b[m]),
4. For all other types, a and b are the same, i.e. a == b.

This relationship intuitively models that whenever it's possible to read a value at a given location (or JSON pointer) from b, it's also possible to read a value at the same location from a and the value that is read for a is a refinement of the value read from b.

The following example is showcasing this relationship:

let a = try JSON(string: #"""
  {
    "a": [1, { "b": 2 }],
    "c": { "d": [{}] }
  }
"""#)
let b = try JSON(string: #"""
  {
    "a": [1, { "b": 2, "e": 4 }],
    "c": { "d": [{"f": 5}] }
  }
"""#)
b.isRefinement(of: a) ⇒ true

Enum JSON provides a method merging(value:) for merging two JSON values a and b such that the result of the merge a.merging(value: b) is the "smallest" JSON value that is a refinement of both a and b. If such a merged value does not exist, then merging(value:) will return nil. Here is an example:

let c = try JSON(string: #"""
  { "a": [1, {"e": 8}],
    "c": {"f": "hello"},
    "g": 9 }
"""#)
a.merging(value: c) 
⇒
{
  "a": [1, { "b": 2, "e": 8 }],
  "c": { "d": [{}], "f": "hello" },
  "g": 9
}

Intuitively, merging(value:) combines two JSON values by adding all non-existing values to the merged value and merging overlapping values when possible. Whenever it is not possible to merge two values, merging(value:) will fail by returning nil.

Overriding Merge

An alternative method, overriding(with:) merges two JSON values differently, letting the JSON value passed as an argument override values of the receiver whenever merging would fail otherwise. As opposed to method merging(value:), combining arrays does not require the arrays to be of the same length. The resulting array has always the length of the longest of the two arrays and individual elements are combined using overriding(with:) whenever two elements are available.

Here is an example which would fail if merging(value:) would be used instead:

let d = try JSON(string: #"""
  {
    "a": [1, { "e": 2 }, 3],
    "c": { "d": "hello" },
    "f": 5
  }
"""#)
a.overriding(with: d)
⇒
{
  "a": [1, { "b": 2, "e": 2 }, 3],
  "c": { "d": "hello" },
  "f": 5
}

JSON Merge Patch

DynamicJSON provides basic support for JSON Merge Patch as defined by RFC 7396.

A JSON merge patch document describes changes to be made to a target JSON document using a syntax that closely mimics the document being modified. Recipients of a merge patch document determine the exact set of changes being requested by comparing the content of the provided patch against the current content of the target document. If the provided merge patch contains members that do not appear within the target, those members are added. If the target does contain the member, the value is replaced. Null values in the merge patch are given special meaning to indicate the removal of existing values in the target.

The algorithm to apply a merge patch document to a JSON value is implemented by method the merging(patch:) of enum JSON.

// Merges this JSON value with the given JSON value `patch` recursively. Objects are
// merged key by key with values from `patch` overriding values of the object represented
// by this JSON value. All other types of JSON values are not merged and `patch` overrides
// this JSON value.
func merging(patch: JSON) -> JSON { ... }

The implementation for applying a merge patch document to a JSON value is not mutating an existing JSON value. It is constructing a new JSON value from scratch by merging the old value with the merge patch document.

Validating JSON Data

DynamicJSON implements JSON Schema as defined by the 2020-12 Internet Draft specification for validating JSON data. The framework is flexible allowing extensions for future revisions.

Implementation Overview

A JSON schema gets represented by enum JSONSchema. It is possible to load JSON schema values either from a file, decode them from a string, or from a data object. In the context of schema validation, top-level schema values are managed via class JSONSchemaResource which pre-processes and validates schema values and provides an identity for them. Often it's easier to just work with JSONSchemaResource objects directly. JSON schema are identified by JSONSchemaIdentifier values, which are essentially URIs with JSON schema-specific methods. A JSONSchemaIdentifier value is either absolute or relative and it is either a base URI, i.e. it is referring to a top-level schema, or it is a non-base URI and thus refers to a schema nested within another schema via a URI fragment.

The semantics of a schema is defined by their dialect. A schema dialect gets identified by a URI. A schema value provides access to their dialect identifier via property schema. If no identifier is provided, a default is assumed (which is JSONSchemaDialect.draft2020 right now for top-level schema and the dialect of the enclosing schema for nested schema). Schema dialects are represented by implementations of the JSONSchemaDialect protocol. A key responsibility of JSONSchemaDialect implementations is to provide a factory method validator(for:, in:) for creating validator objects for this dialect. Validator objects implement protocol JSONSchemaValidator which provides one validate() method that takes a JSON instance and returns a JSONSchemaValidationResult value. This method gets eventually called to validate a JSON value.

The whole schema validation process gets initiated and controlled by a JSONSchemaRegistry object. JSON schema registries define:

• A set of supported dialects with their corresponding URI identities,
• A default dialect (for schema resources that do not define a dialect themselves),
• A set of known/loaded schema resources with their corresponding identities, and
• JSONSchemaProvider objects, each implementing a method for discovering and loading new schema resources for schema that are not loaded already.

Most of the functionality of class JSONSchemaRegistry is about configuring registry objects by registering supported dialects, inserting available schema resources and setting up schema providers for automatically discovering schema resources. Once a registry is configured, method validator(for:, dialect:) can be called to obtain a JSONSchemaValidator object for the given schema resource and default dialect. This validator object can then be used to validate an arbitrary number of JSON instances.

The following example shows how validation is used in general:

// Create a new schema registry
let registry = JSONSchemaRegistry()
// Register a schema resource from a string literal
try registry.register(resource: JSONSchemaResource(string: #"""
  {
    "$id": "https://example.com/schema/test",
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
      "prop1": {
        ...
      }
    }
  }
"""#))
...
// Load a schema resource from a file
try registry.loadSchema(from: URL(filePath: "/Users/objecthub/foo.json"))
...
// Make JSON schema stored in json files under the given directory discoverable
registry.register(provider:
  StaticJSONSchemaFileProvider(
    directory: URL(filePath: "/Users/objecthub/myschema"),
    base: JSONSchemaIdentifier(string: "http://example.com/schemas")!))
...
// Obtain a validator for a schema
guard let validator = try? registry.validator(for: "https://example.com/schema/test") else {
  // Throw error stating that the schema could not be found
}
// Validate a JSON instance `json`
let result = validator.validate(json)
print("valid = \(result.isValid)")

Schema validators return JSONSchemaValidationResult values. These are containers which provide access to information collected during the validation process. JSONSchemaValidationResult values encapsulate the following information:

• Validation errors
• Format constraints (i.e. format requirements for string values defined by the format keyword)
• Meta tags (i.e. annotations about access and deprecations)
• Default values (i.e. default values defined by the default keyword)

For figuring out whether validation was successful, it is sufficient to use property isValid of JSONSchemaValidationResult. It returns true if no validation errors were found. Otherwise, property errors provides access to the validation errors found. Other annotations that are collected during validation are discussed below.

Validation API

If an application only validates JSON instances against a small number of fixed schema (e.g. provided statically at application startup), it would be overkill to make use of the low-level API introduced above. For such simple use cases, enum JSON provides the following convenience methods:

enum JSON: Hashable, ... {
  // Returns true if this JSON document is valid for the given JSON schema (using
  // `registry` for resolving references to schema referred to from `schema`).
  func valid(for schema: JSONSchema,
             dialect: JSONSchemaDialect? = nil,
             using registry: JSONSchemaRegistry? = nil) -> Bool
  
  // Returns a schema validation result for this JSON document validated against the
  // JSON schema `schema` (using`registry` for resolving references to schema
  // referred to from `schema`).
  func validate(with schema: JSONSchema,
                dialect: JSONSchemaDialect? = nil,
                using registry: JSONSchemaRegistry? = nil) throws -> JSONSchemaValidationResult
  
  // Returns true if this JSON document is valid for the given JSON schema (using
  // `registry` for resolving references to schema referred to from `schema`).
  func valid(for resource: JSONSchemaResource,
             dialect: JSONSchemaDialect? = nil,
             using registry: JSONSchemaRegistry? = nil) -> Bool
  
  // Returns a schema validation result for this JSON document validated against the
  // JSON schema `schema` (using`registry` for resolving references to schema
  // referred to from `schema`).
  func validate(with resource: JSONSchemaResource,
                dialect: JSONSchemaDialect? = nil,
                using registry: JSONSchemaRegistry? = nil) throws  -> JSONSchemaValidationResult
  ...
}

These validation methods are creating new registries on demand if parameter registry is set to nil, with only the provided schema or schema resource getting registered. For using non self-contained schema, it is therefore necessary to set up a suitable registry first and pass it in via the registry parameter. Alternatively, it is possible to use the default registry JSONSchemaRegistry.default if a single, shared, global registry is sufficient.

Metadata and Defaults

Format annotations

format annotations of a JSON schema, i.e. declarations that JSON string values have a particular format, are being collected and made available via the formatConstraints property of JSONSchemaValidationResult values. Each constraint is an instance of Annotation<FormatConstraint> providing access to fields value (a LocatedJSON value), location (within the schema), message.format and message.valid, with message.valid being a value of type Bool?. true refers to valid constraints, false to invalid constraints, and nil to constraints that could not be validated. Here is code that prints out all invalid constraints:

val res: JSONSchemaValidationResult = ...
for constraint in res.formatConstraints where constraint.message.valid == false {
  print("value \(constraint.value) does not match format '\(constraint.message.format)'")
}

If invalid format constraints should result in a validation error, the vocabulary https://json-schema.org/draft/2020-12/meta/format-annotation needs to be enabled. This can be done by creating a custom JSONSchemaDraft2020.Dialect value with a vocabulary of type JSONSchemaDraft2020.Vocabulary whose format property is set to true. Since such a dialect is useful frequently, a preconfigured dialect value is available via JSONSchemaDialect.draft2020Format. Defining a registry with this dialect as its default will always also validate format annotations.

Default annotations

default annotations of a JSON schema, i.e. declarations that properties have a given default value if the property is not defined explicitly, are being collected and made available via the defaults property of JSONSchemaValidationResult values. defaults is providing a map from JSONLocation values to tuples (exists: Bool, values: Set<JSON>). The exists component of the tuple states whether a value exists at this location (and thus, not default needs to be injected). The values component provides a set of JSON values which are all suitable as defaults (a schema can define multiple, alternative defaults). Here is code that prints out default values computed during a validation process:

let schema = try JSONSchema(string: #"""
  {
    "$id": "https://objecthub.com/example/person",
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "title": "person",
    "type": "object",
    "properties": {
      "name": {
        "type": "string",
        "minLength": 1
      },
      "birthday": {
        "type": "string",
        "format": "date"
      },
      "numChildren": {
        "type": "integer",
        "default": 0
      },
      "address": {
        "oneOf": [
          { "type": "string", "default": "12345 Mcity" },
          { "$ref": "#address",
            "default": { "city": "Mcity", "postalCode": "12345" } }
        ]
      },
      "email": {
        "type": "array",
        "maxItems": 3,
        "items": {
          "type": "string",
          "format": "email"
        }
      }
    },
    "required": ["name", "birthday"],
    "$defs": {
      "address": {
        "$anchor": "address",
        "type": "object",
        "properties": {
          "street": {
            "type": "string"
          },
          "city": {
            "type": "string"
          },
          "postalCode": {
            "type": "string",
            "pattern": "\\d{5}"
          }
        },
        "required": ["city", "postalCode"]
      }
    }
  }
"""#)
/// `instance0` is a valid instance of `schema`
let instance0: JSON = [
  "name": "John Doe",
  "birthday": "1983-03-19",
  "numChildren": 2,
  "email": ["john@doe.com", "john.doe@gmail.com"]
]
instance0.valid(for: schema) ⇒ true
/// `instance1` is not a valid instance of `schema`
let instance1: JSON = [
  "name": "John Doe",
  "email": ["john@doe.com", "john.doe@gmail.com"]
]
instance1.valid(for: schema) ⇒ false
/// `instance2` is a valid instance of `schema`
let instance2: JSON = [
  "name": "John Doe",
  "birthday": "1983-03-19",
  "address": "12 Main Street, 17445 Noname"
]
let res2 = try instance2.validate(with: schema)
res2.isValid ⇒ true
for (location, (exists, values)) in res2.defaults {
  if exists {
    print("\(location) exists; defaults: \(values)")
  } else {
    print("\(location) does not exist; defaults: \(values)")
  }
}

The loop at the end of this code prints out the following text:

$['numChildren'] does not exist; defaults: [0]
$['address'] exists; defaults: [
  "12345 Mcity",
  {
    "postalCode" : "12345",
    "city" : "Mcity"
  }
]

Property metadata

Property metadata annotations of a JSON schema such as deprecated, readOnly, and writeOnly, are being collected and made available via the tags property of JSONSchemaValidationResult values. Each location within the validated value with a metadata annotation is included in this array with an entry of type Annotation<MetaTags> providing access to fields value (a LocatedJSON value), location (within the schema), message.deprecated message.readOnly, and message.writeOnly. deprecated, readOnly, and writeOnly are boolean properties.

Streaming JSON Values

Many sources deliver a sequence of JSON values instead of a single document: log files, exports, web service responses, or the output of other processes. JSON.values(from:format:options:) is the single entry point for reading such sequences from any AsyncSequence of bytes (UInt8), one value at a time. The format parameter selects how values are delimited:

Format	Description
.lines	NDJSON and JSON Lines (.ndjson, .jsonl): one value per line; \r\n and blank lines are tolerated
.sequence	JSON text sequences, RFC 7464 (application/json-seq) and RFC 8142 (GeoJSON text sequences): each value is preceded by the ASCII record separator (0x1E)
.concatenated	values follow each other with arbitrary or no whitespace in between, e.g. {"a":1}{"b":2}
.arrayElements	the elements of one huge top-level array are returned incrementally
.automatic	the default: .sequence if the stream starts with a record separator, otherwise .concatenated (which also reads NDJSON)

// Asynchronous streaming from a file, a URL, or a pipe
for try await value in JSON.values(from: url.resourceBytes, format: .lines) {
  print(value)
}
for try await value in JSON.values(contentsOf: url, format: .arrayElements) { ... }

// In-memory data
let values = try JSON.values(from: "{\"a\": 1}\n{\"a\": 2}\n", format: .lines)
for result in JSON.results(from: data, format: .sequence) { ... }

By default, the for try await loop throws a JSON.StreamError (including the byte offset) when it reaches the first malformed value, and no further values are read. Setting JSON.StreamOptions(errors: .skipInvalid) drops malformed values and continues, as recommended by RFC 7464; this works reliably for .lines and .sequence, which resynchronize at the next line or record separator. JSON.results(from:format:options:) returns Result<JSON, JSON.StreamError> elements instead of throwing, and StreamOptions.maxValueSize bounds the memory used for a single value.

Streaming from Web APIs and LLMs

Streaming web APIs, including the APIs of large language models, deliver JSON in two layers: the response body is a stream of server-sent events (text/event-stream), and the JSON carried by these events is often a sequence of fragments of one larger value, such as the arguments of a tool call. DynamicJSON supports both.

Format .serverSentEvents returns the JSON data of each event. Comments and the fields event, id and retry are skipped (use JSON.events(from:) to access them), and the stream ends when an event has the data [DONE] (configurable via StreamOptions.terminator).

JSONPartialParser reads a single JSON value that arrives in fragments and returns the best approximation of the value at any time. It closes open strings, arrays, and objects, and leaves out incomplete keys, values, and numbers, so that each snapshot is a prefix of all later ones:

var parser = JSONPartialParser()
try parser.append(#"{"city": "Par"#)
try parser.snapshot()                  // {"city": "Par"}
try parser.append(#"is", "population": 21"#)
try parser.snapshot()                  // {"city": "Paris"}
try parser.append("00000}")
try parser.finish()                    // {"city": "Paris", "population": 2100000}

The pieces compose. JSON.fragments(from:at:) extracts the text fragments of the events with a JSON pointer, and JSON.partialValues(from:) turns the fragments into a sequence of PartialJSON snapshots (use PartialJSON.patch(from:) to get the changes as a JSONPatch):

let (bytes, _) = try await URLSession.shared.bytes(for: request)
let events = JSON.values(from: bytes, format: .serverSentEvents)
let pointer = try JSONPointer("/choices/0/delta/tool_calls/0/function/arguments")
for try await partial in JSON.partialValues(from: JSON.fragments(from: events, at: pointer)) {
  print(partial.value, partial.isComplete)
}

No provider-specific code is involved; the JSON pointer selects the fragments from the events of the API at hand (e.g. /delta/partial_json for Anthropic tool input).

Writing Streams of JSON Values

The same formats can be created. JSONStreamWriter turns values into the bytes of a stream without doing any I/O: each call returns what belongs to a value, including the framing of the format, and finish() returns the closing bytes (such as the ] of an array). Everything that is written can be read again with JSON.values(from:format:).

var writer = try JSONStreamWriter(format: .lines)     // or .sequence, .concatenated,
var output = Data()                                   // .arrayElements, .serverSentEvents
output.append(try writer.write(["id": 1, "name": "Ada"]))
output.append(try writer.write(["id": 2, "name": "Alan"]))
output.append(try writer.finish())

let data = try JSON.stream(values, format: .sequence) // a whole sequence of values
try JSON.write(values, to: url, format: .arrayElements,
               options: .init(formatting: [.prettyPrinted, .sortedKeys]))

// An asynchronous sequence of values becomes an asynchronous sequence of Data chunks,
// e.g. for the body of a streaming HTTP response
for try await chunk in JSON.stream(values, format: .lines) { ... }

JSON.StreamWriteOptions configure the encoding (output formatting, float and date strategies, line endings, the separator of concatenated values, and a terminator event such as [DONE] for server-sent events). ServerSentEvent.encoded and JSONStreamWriter.write(_:event:id:retry:) create events with names and identifiers; data that consists of several lines becomes several data fields.

Extracting JSON from LLM Output

Language models rarely produce only JSON: answers often wrap the data in prose and Markdown code blocks, and the JSON itself might contain comments, trailing commas, single quotes, Python literals such as None, or be cut off at the length limit. JSON.extract(from:options:) finds the objects and arrays in such a text and parses them leniently. Each result tells where it was found, which range of the text it occupies, and which deviations from JSON (JSONRepair) had to be accepted:

let answer = """
  Sure! Here is the data you asked for:
  ```json
  { "name": "Ada", "languages": ["en", "fr",], }  // trailing commas
  ```
  Let me know if you need anything else.
  """
let found = JSON.extract(from: answer)
found[0].value    // {"name": "Ada", "languages": ["en", "fr"]}
found[0].repairs  // [trailingComma, comment]

JSON.extractFirst(from:options:) returns only the first value, and JSON(lenient:) parses a text that consists of exactly one value (optionally in a code block). Truncated values are completed by closing strings, arrays, and objects and by dropping incomplete members, as JSONPartialParser does. JSONExtractionOptions restricts the search to code blocks, selects the types of values to extract, and limits the number of results.

Demo: WikiWatch

WikiWatch is a small command-line tool (target WikiWatch of the Swift package and of the Xcode project) that puts the streaming support to work on a real stream: Wikimedia's public feed of all edits to its wikis, delivered as server-sent events. It reads the events with JSON.events(from:), validates them with a JSON Schema, selects events with JSON Path filters, and shows a live dashboard in the terminal. Its source code is extensively commented and written to teach how the library is used in practice: start reading at Sources/WikiWatch/main.swift, which has a map of the files and what each of them demonstrates. A recorded sample makes it possible to try it without network access:

swift run WikiWatch                                     # live dashboard
swift run WikiWatch --plain --wiki enwiki --no-bots     # one line per matching event
swift run WikiWatch --replay Examples/WikiWatch/sample-recentchange.sse
swift run WikiWatch --replay Examples/WikiWatch/sample-recentchange.sse --export out.ndjson  # write a stream
swift run WikiWatch --trickle Examples/WikiWatch/sample-recentchange.sse  # incremental JSON

Building the Documentation

The API reference and articles of DynamicJSON are written for DocC. The DocC catalog is located at Sources/DynamicJSON/DynamicJSON.docc; the reference material is generated from the /// comments in the source code.

With Xcode

Open Package.swift (or DynamicJSON.xcodeproj) in Xcode, select the DynamicJSON scheme, and choose Product > Build Documentation (⌃⇧⌘D). The documentation shows up in Xcode's documentation viewer. To export it, right-click the DynamicJSON entry in the documentation navigator and choose Export… to create a .doccarchive bundle.

Without Xcode (Terminal)

With the Xcode command-line tools or a Swift toolchain that includes docc, the documentation can be built from the terminal in two steps. First, emit symbol graphs for the DynamicJSON module (the package also contains the JSONPathTool executable, whose symbol graph must not be mixed in, as a DocC build covers a single module):

rm -rf .build/symbol-graphs && mkdir -p .build/symbol-graphs
swift package dump-symbol-graph
cp .build/out/symbolgraph/DynamicJSON*.symbols.json .build/symbol-graphs/

Then run docc to combine the symbol graphs with the DocC catalog:

xcrun docc convert Sources/DynamicJSON/DynamicJSON.docc \
  --fallback-display-name DynamicJSON \
  --fallback-bundle-identifier org.objecthub.DynamicJSON \
  --additional-symbol-graph-dir .build/symbol-graphs \
  --output-path .build/DynamicJSON.doccarchive

(On platforms without xcrun, invoke docc directly.) The resulting .doccarchive can be opened with Xcode, or previewed in a browser with a local web server:

xcrun docc preview Sources/DynamicJSON/DynamicJSON.docc \
  --fallback-display-name DynamicJSON \
  --fallback-bundle-identifier org.objecthub.DynamicJSON \
  --additional-symbol-graph-dir .build/symbol-graphs

docc preview prints the URL to open (by default http://localhost:8080/documentation/dynamicjson). To publish the documentation on a static web host such as GitHub Pages, add --transform-for-static-hosting --hosting-base-path <path> to the docc convert command.

Alternatively, xcodebuild builds the documentation for the Xcode project from the terminal:

xcodebuild docbuild -scheme DynamicJSON -destination 'generic/platform=macOS'

The build prints the location of the generated .doccarchive bundle (inside Xcode's DerivedData folder, or the folder passed via -derivedDataPath) and reports unresolved symbol links as warnings.

Requirements

The following technologies are needed to build the DynamicJSON framework. The library and the command-line tool can both be built either using Xcode or the Swift Package Manager.

• Xcode 16
• Swift 6
• Swift Package Manager

Migrating from the Swift 5 Version

This version of DynamicJSON is built in the Swift 6 language mode (swift-tools-version:6.0) and needs a Swift 6 toolchain (Xcode 16 or later). Previous versions were built in the Swift 5 language mode. The API is otherwise unchanged, but the move to strict concurrency checking introduced the following source incompatibilities:

• JSONReference requires Sendable. Custom implementations of the protocol (JSONReference, SegmentableJSONReference) have to be Sendable, e.g. value types with Sendable members. JSONLocation and JSONPointer already are.
• userInfo parameters. The parameter userInfo of the initializers and methods that encode or decode JSON data (JSON, JSONPatch, JSONPatchOperation) now has type [CodingUserInfoKey : any Sendable]? instead of [CodingUserInfoKey : Any]?, matching JSONEncoder and JSONDecoder of the current Foundation. A dictionary variable of the old type needs to be changed accordingly; dictionary literals continue to work.
• @Sendable closures. Format validators of a schema dialect (JSONSchemaDraft2020.Vocabulary.formatValidators) and functions added to JSON Path (JSONPathEvaluator.Function) are @Sendable: closures that are registered there must not capture mutable state.
• More types are Sendable. JSONLocation, JSONPointer, JSONPath, JSONSchema and the related schema types are now Sendable. This only matters for code that relied on them not being Sendable, which is rare.
• JSONSchemaRegistry.default is unsafe to share. It is declared nonisolated(unsafe), as it was already documented not to be thread-safe. Do not use it from multiple tasks at the same time; create separate registries instead.

Copyright

Author: Matthias Zenger (matthias@objecthub.com)
Copyright © 2024-2026 Matthias Zenger. All rights reserved.