Spotlight on dwsJSON

dwsJSON is a unit that supports JSON parsing/creating, it’s part of DWScript but relatively “standalone”, in that if you add it in your Delphi (or FPC) projects, it won’t pull the whole of DWScript library, and thus can be used anywhere you need.

The JSON parser in dwsJSON generates an object tree based on TdwsJSONValue (and its subclasses), you can also create such a tree from scratch and use it to spit out JSON. There is also a low-level JSON writer for fast, write-only streaming. All classes support full Unicode (in Delphi) as well as very long strings (like a base64 streamed image).

dwsJSON is rather fast, especially compared to Delphi’s DBXJSON, don’t be surprised if it parses data 15 to 30 times faster (f.i. on JSON.org examples). And the ratio can go even higher on larger JSON datasets. It is also more resilient to bad input, less buggy ( 😉 ), and should use less memory in just about any usage scenario. It also is about twice faster than the last version (as of this writing) of SuperObject, which is itself quite efficient.

Parsing JSON

Let’s suppose you have the following JSON data

{
   "Hello" : [
      "here",
      {
         "There" : 456;
      }
   ],
   "World" : 123
}

and you loaded it in the rawJSON variable, you can parse it with

var json : TdwsJSONValue;
...
json := TdwsJSONValue.ParseString(rawJSON);

That’s all.

Accessing the JSON data

You can access the data using explicit methods and properties (see the source and unit tests), or use the short forms:

WriteLn('Hello has ', json['Hello'].ElementCount, ' elements');

WriteLn('The first element has a ValueType of ', GetEnumName(TypeInfo(TdwsJSONValueType), Ord(json['Hello'][0].ValueType)));
WriteLn('The second element class name is ', json['Hello'][1].ClassName);

WriteLn('The first element has a first field named "', json['Hello'][0].Names[0], '"');
WriteLn('and its JSON representation is ', json['Hello'][0]['here'].ToString);

WriteLn('The value of the DoesNotExists field is "', json['DoesNotExists'].ToString, '"');
WriteLn('and its type is ', GetEnumName(TypeInfo(TdwsJSONValueType), Ord(json['DoesNotExists'].ValueType)));

WriteLn('Also World = ', json['World'].ToString);

will give the following output:

Hello has 2 elements
The first element has a ValueType of jvtObject
The second element class name is TdwsJSONImmediate
The first element has a first field named "here"
and its JSON representation is {"There":456}
The value of the DoesNotExists field is ""
and its type is jvtUndefined
Also World = 123

One thing of note is that TdwsJSONValue allows to navigate in nodes that do NOT exist in the JSON (ie. nil objects in Delphi), they’ll have a ValueType of jvtUndefined, if you query their value, you’ll get an empty string, a zero, etc.

This can be particularly convenient when the JSON has optional portions (common in JSON), as it allows to drastically cut down on the “if” testing. For instance you can write

json['DoesNotExists'].ElementCount

even if json[‘DoesNotExists’] is nil, ie. if your array is missing in the JSON, ElementCount will be zero, and if you write

json['DoesNotExists'][2]['Field']

it will return a nil TdwsJSONValue, whose ValueType will be jvtUndefined.

Altering or writing JSON

You can alter an existing (or newly generated) JSON tree by using the AddXxxx methods and assign values using the AsXxxx. You remove elements through the DetachChild method, which can be used to delete elements, or to split a monolithic JSON into simpler sub-trees.

Another option to generate JSON is to use the lower-level TdwsJSONWriter, which is a simple streamer (so the output JSON will only be correct if you write everything needed). It has a TdwsJSONBeautifiedWriter subclass, which instead of generating a “tight” JSON will introduces tabs, spaces and new lines to make the output more palatable when it’s intended to be human-readable.

The TdwsJSONWriter provides a set of WriteXxxx methods, along with Begin/End Object/Array methods, so can be easily adapted to most serialization frameworks.

Licensing

Like the rest of DWScript, dwsJSON is under MPL 1.1, ie. free to use in both open-source and commercial software, as long as you honor the license terms: the DWScript project or dwsJSON must be “conspicuously” mentioned, and you have to publish modifications to the code (if you make any, you don’t have to publish anything if you use it verbatim). If you’re interested in using it under other terms, feel free to contact me.

13 thoughts on “Spotlight on dwsJSON

  1. Nice to see more publicity on dwsJSON))

    What about Oxygene support? Did this stuff include some quirks that prevent to use it in Oxygene-based languages?

  2. What about Oxygene support? Did this stuff include some quirks that prevent to use it in Oxygene-based languages?

    No idea, at the moment, I’m focusing more on getting everything up & stable under FreePascal.

  3. The only abstraction I’m missing in this, is that dwsJSON doesn’t offer a stand-alone tokenizer. The way it is right now, I cannot avoid instantiating lots and lots of instances, even when only a small part of the input is what I want, or when I just want to scan for some value, or whatnot…

  4. @Eric
    That is the best thing that you are doing. I am truly interested in FPC based version of DWScript.

    Thanks for your initiative.

    Does this mean that I will be able to use DWScript in D2007 also once it is converted to FPC?

  5. when only a small part of the input is what I want, or when I just want to scan for some value, or whatnot…

    Like what SAX is to XML? It’s somewhere in the future pipeline.

    Does this mean that I will be able to use DWScript in D2007 also once it is converted to FPC?

    That’s unlikely, as I’m aiming for the very latest FPC (2.7.1) with generics & Unicode support. The distance to pre-Unicode, pre-Generics compilers is just too great to bridge (not without sacrificing lots of things, and making the support pointless).

  6. There is a bug in dwsJSON parser. Just try to parse this simple valid json string:
    {
    “Bug”: {}
    }

    EdwsJSONParseError exception is raised.

  7. Hi

    When trying to use dwsJSON.pas, and having copied across the following files to my Lazarus project too :

    dws.inc
    dwsUtils.pas
    dwsXPlatform.pas

    I get the following errors when trying to compile from program:

    dwsXPlatform.pas(142,74) Error: Incompatible type for arg no. 5: Got “PWideChar”, expected “PChar”
    dwsXPlatform.pas(204,17) Error: Incompatible types: got “Char” expected “Array[0..260] Of WideChar”
    dwsXPlatform.pas(205,67) Error: Incompatible type for arg no. 4: Got “Array[0..260] Of WideChar”, expected “PChar”

    I am using Lazarus 1.1, FPC 2.6.1, SVN Rev 37432 on Windows 7 Pro, 64-Bit and dwscript-2.2 stable.

    Any thoughts? I am probably doing something wrong? Ideally I’d like to package the JSON functionality into my Lazarus project but saw no lpk file to do so?

  8. Hi,
    I’m very interested in using JSON as a streaming replacement for our client-server application.
    Can you give me some information about how fast objects are streamed using JSON compared to a simple Delphi TMemoryStream (the objects are streamed in binary here, not in any textual representation)?
    Thank you.

  9. @TED
    Dunno about FPC 2.6.1, there are too many things missing, so I only tested with FPC 2.7.1, and I don’t get those errors with it. Though Unicode support in FPC is still quite incomplete, and the unit tests don’t pass under FPC yet.

    I don’t know about .lpk, the JSON functionality isn’t visual though, so a regular ‘uses’ is enough.

    @Norbert
    Will depend on what your objects contain, there will be a hit when converting numbers to their string representations, or when encoding the strings, how much of a hit will depend on your data structures.

Comments are closed.