JsonValue Feature Details

The current WCF serialization stack in the framework does not provide a DOM-based approach for working with JSON data. When it comes to XML data, we have classes such as XElement, which we can use in this scenario, but there is no analog for JSON. JsonValue and its associated types provide a mechanism for weakly-typed access to JSON data.

Scenarios

1. Working with JSON objects where it is impractical to build a matching type

The existing DataContractJsonSerializer requires that the developer build a CLR type that the JSON object maps to before deserialization can take place. When working with third-party services that return JSON, frequently the JSON objects returned by those services are documented in human-readable form, or not documented at all. There is no universally-adopted JSON schema format. So there are situations where it is impractical to build a type to deserialize into (for example if the JSON object returned has a ton of members and/or deep nesting), where a DOM-based approach is needed. A reader-based approach would also work, however it is significantly less usable.

2. Selectively obtaining a few pieces of data out of a JSON object

It is common that clients will send a request to a HTTP-based service (for example to get a weather forecast for given ZIP code), and only extract one or two data members out of the response (for example the current temperature and chance of rain). In this case the user may choose to use the serializer and build a CLR type that contains just the values they are looking for. If the values are nested inside the JSON structure, they may have to build a hierarchy of nested types just to get to the fields that interest them, which is cumbersome and impractical. In this scenario the user can use a DOM-based approach to only get the values they are interested in without any extra overhead. A reader-based approach would also work, however it is significantly less usable.

3. Working with JSON in environments where user is not aware of types

In more dynamic language environments (such as the new ASP.NET Razor view engine), types are not first-class. For example the new Microsoft.Data APIs return dynamic types, compared to strong types, which is what Entity Framework returns. Users do not expect to write classes to access data, so the natural experience for working with JSON data is a weakly-typed dynamic object.

4. Relaying data between JavaScript clients

In a scenarios where multiple JavaScript clients are communicating (via a protocol such as WebSockets), messages originate at one client and are received by another. The server can pass the messages through, but may wish to parse the messages to validate that they contained valid JSON data. It may also look for some control information that the server can use to decide how to route the message. Since the messages originate at the clients, the server may not have types to deserialize the arbitrary JSON objects into.

Details

JsonValue (and its derived types JsonPrimitive, JsonArray, and JsonObject) provide a weakly-typed DOM-based abstraction for dealing with JSON in the framework. These APIs have already shipped in Silverlight and are documented here. This document presents proposed improvements over this existing API.

1. Casting improvements

Much of the added value over just using Dictionary<string, object> is the ability to intelligently cast in and out of JsonValue, to provide a seamless user experience. We have a set of explicit casts into common CLR primitives: that is fairly straightforward. We also introduced a more fluent way of doing a cast using the ReadAs<T> method. The behavior of that method is identical to the matching cast, which also means that if the value cannot be cast, then the method will throw an exception (usually InvalidCastException). This is not ideal in situations where the user has to handle unexpected data, and they don’t want to write logic based on try/catch. To satisfy this case, we also added the ability to return a fallback value instead of throwing, as well as a TryReadAs<T> method.

  1. public class JsonValue : IEnumerable, IDynamicMetaObjectProvider
  2. {
  3.     // added for more fluent casting experience
  4.     public virtual T ReadAs<T>();
  5.     // added for more fluent casting experience
  6.     public virtual T ReadAs<T>(T fallback);
  7.     // added for a safe casting experience
  8.     public virtual bool TryReadAs<T>(out T value);
  9. }

The focus on gracefully handling casting errors will become clearer in the next requirement.

2. Ability to parse CLR primitives out of JSON strings

Because of JSON’s limited type hierarchy, some complex types of information are frequently presented as regular strings. Examples of that are dates, Uris, guids, and others. Consider the following typical JSON object

  1. {
  2.     "ID" : "538a868a-c575-4fc9-9a3e-e1e1e68c70c5",
  3.     "Name" : "John Doe",
  4.     "DOB" : "1962-09-23T03:58Z"
  5. }

(Another aspect that is not discussed here is that we have developed a way to parse form-urlencoded notation into JsonValue. When using JsonValue this way, all values are represented as JsonType.String and there is no JsonType.Number or JsonType.Boolean. Therefore, the set of types that can now be “hidden” inside a string is now expanded to all number types and Booleans, which makes this feature significantly more compelling.)

To satisfy this requirement, we now allow casts/ReadAs calls from a JsonPrimitive with JsonType.String, to all common primitives. Previously that would have resulted in an InvalidCastException. Instead of having to hunt around for Parse/TryParse methods on the appropriate types, the user can now use the familiar casting/ReadAs semantic.

  1. DateTime date = jo["DOB"].ReadAs<DateTime>();
  2. Guid id = jo["ID"].ReadAs<Guid>();

Here are the details of how parsing works:

JsonType

Cast

Behavior

Boolean (JsonPrimitive)
(actual internal value type is bool)

bool (bool)value
string Call ToString on the internal value
others InvalidCastException

Number (JsonPrimitive)
(actual internal value type could be any of byte, decimal, double,  short, int, long, float, sbyte, ushort, uint, ulong)

byte If the internal type is the same as (cast) then return it. If not then call Convert.ChangeType, which will almost always work, but it may throw OverflowException.
decimal
double
short
int
long
float
sbyte
ushort
uint
ulong
string Call ToString on the internal value
others InvaildCastException

String (JsonPrimitive)
(actual internal value type could be string, char, DateTime, DateTimeOffset, Guid, Uri)

always As a first step of all casts, take the internal type and get it JSON representation, then unescape it and remove the quotes from the beginning and end. Then...
string ...return the result.
byte ... first call (cast).TryParse. If that doesn’t work it could be that the string still contains a valid number (say “1.1”), but the cast (for example “int”) expects a different format. In that case we try calling TryParse on long (only if characters “.”/“e”/“E” are present), decimal, and then double in that order, to get some numeric value out of the string, and then call Convert.ChangeType to get it to type {cast} (might throw OverflowException). If all that fails we return InvalidFormatException.
decimal
double
short
int
long
float
sbyte
ushort
uint
ulong
DateTime

... return DateTime/DateTimeOffset.TryParse with the following formats.

UTC time of event (also honor the time zone where the client machine is located if it is provided):

  • ddd, d MMM yyyy HH:mm:ss UTC
  • ddd, d MMM yyyy HH:mm:ss GMT
  • yyyy-MM-ddTHH:mm:ssK
  • yyyy-MM-ddTHH:mm:ss.fffK

Local time at server machine (assume client and server are collocated):

  • yyyy-MM-ddTHH:mm:sszzz
  • yyyy-MM-ddTHH:mm:ss
  • yyyy-MM-dd
  • HH:mm:ss
  • HH:mm

This might throw InvalidDateFormatException.

DateTimeOffset
Uri ... return Uri.TryCreate, which might throw InvalidUriFormatException.
char

… return (cast).TryParse, which might throw InvaldFormatException.

bool
Guid
others InvalidCastException

Object (JsonObject)

always InvalidCastException

Array (JsonArray)

always InvalidCastException

3. Colloquial JSON

JSON is a format largely driven by the community, so it is important to support convention that is commonly adopted, even if it is not part of the standard. To accomplish this, we have made changes around the handling of certain types:

  1. DateTime: Previously we would only honor the ASP.NET AJAX DateTime format
    "\/Date(894427200000)\/", which is not very widely adopted. Now on deserialization we accept common formats supported natively by today’s browsers, such as the format "Thu, 8 Jul 2010 03:10:03 UTC" and the ISO 8601 format "1994-11-05T08:15:30-05:00". On serialization, we use the ISO 8601 format due to its ability to express UTC time and timezones.
  2. DateTimeOffset: Previously we would use a custom JSON object which looks like {"date": "\/Date(894427200000)\/", "offset": 1234}. Now we use the standard ISO 8601 format
  3. TimeSpan: We no longer support TimeSpan because there is no useful representation of it in JSON, that browsers understand.

4. Safe indexers

In scenarios where the user is trying to select a value inside a JsonObject, where the value is potentially nested deeply, the existing array and string indexers on JsonArray and JsonObject don’t work because they may throw an exception or return null. To ensure a safe selection operation, the user has to wrap their logic in a try/catch block that looks for NullReferenceException and KeyNotFoundException.

We do offer a safe way to check if a property is there on an object: the user can call JsonObject.ContainsKey before they try to access the key, or they can use the TryGetValue method. Similarly for arrays, they can check the Count before they try to access a given index. However both of these are quite cumbersome and require multiple lines per member, so it becomes very hard if the desired value is nested deeply.

To address this, we are adding a special set of string and integer indexers called ValueOrDefault that return the value or a new JsonType.Default if the value is not there. You can keep calling ValueOrDefault on the JsonType.Default, and you will not get any errors.

  1. public class JsonValue : IEnumerable, IDynamicMetaObjectProvider
  2. // class is no longer abstract, but has an internal constructor
  3. {
  4.     internal JsonValue(); // we use this to instantiate “default” instances
  5.     public virtual JsonType JsonType; // now returns JsonType.Default if not overriden
  6.     public virtual int Count; // now returns 0 if JsonType.Default
  7.     public virtual bool ContainsKey(string key); // now returns false if JsonType.Default
  8.     private static JsonValue DefaultInstance; // the default object is a singleton
  9.     public override string ToString(); // now returns “Default” if JsonType.Default
  10.  
  11.     public virtual JsonValue ValueOrDefault(string key); // returns default instance
  12.  
  13.     public virtual JsonValue ValueOrDefault(int index); // returns default instance
  14. }
  15. public sealed class JsonArray : JsonValue, IList<JsonValue>
  16. {
  17.     public override JsonValue ValueOrDefault(int index); // returns value or calls base
  18. }
  19. public sealed class JsonObject : JsonValue, IDictionary<string, JsonValue>
  20. {
  21.     public override JsonValue ValueOrDefault(string key);  // returns value or calls base
  22. }

This enables you to work with JsonObject in the following way, without being afraid of unexpected exceptions you have to handle:

  1. string city = jo.ValueOrDefault("Friends").ValueOrDefault(1).ValueOrDefault("Address").ValueOrDefault("City").ReadAs<string>("City not available");

5. Support for the “dynamic” keyword

In dynamic language environments, users expect to be able to cast JsonValue to dynamic and use dots to index into the type instead of using indexers/TryGetValue/ValueOrDefault.

  1. public class JsonValue : IEnumerable, IDynamicMetaObjectProvider
  2. // implementing this interface gives us dynamic support
  3. {
  4.     public dynamic AsDynamic; // returns (dynamic)this
  5.  
  6.     DynamicMetaObject IDynamicMetaObjectProvider.GetMetaObject(Expression parameter)
  7.     {
  8.         return new DynamicIndexerMetaObject(parameter, this);
  9.     }
  10.     [EditorBrowsable(EditorBrowsableState.Never)] // hide in IntelliSense
  11.     public virtual JsonValue GetValueByKey(string key); // calls ValueOrDefault(key)
  12.     [EditorBrowsable(EditorBrowsableState.Never)] // hide in IntelliSense
  13.     public virtual JsonValue GetValueByIndex(int index); // calls VaueOrDefault(index)
  14.     [EditorBrowsable(EditorBrowsableState.Never)] // hide in IntelliSense
  15.     public virtual JsonValue SetValueByKey(string key, object value); // calls [key] = value
  16.     [EditorBrowsable(EditorBrowsableState.Never)] // hide in IntelliSense
  17.     public virtual JsonValue SetValueByIndex(int index, object value); //calls [index] = value
  18.     private class DynamicIndexerMetaObject : DynamicMetaObject { };
  19. }

Note that the dynamic implementation relies on the new safe indexers.

6. Improvements in LINQ support

LINQ support over JsonValue allows customers to use LINQ over JSON, just like they currently do over XML. Much in the same spirit as the safe indexers mentioned above, we want to ensure that it is easy for customers to write safe LINQ queries that will return an empty result set instead of throwing exceptions.

Let’s take a common example of a LINQ query over JsonValue.

  1. var match = from person in people
  2.             where person["Name"].ReadAs<string>().StartsWith("S")
  3.                   && person["Age"].ReadAs<int>() > 20
  4.             select person;

There are multiple places where this expression could throw an exception:

  1. We expect the person object to be a JsonObject, however that might not be the case. It could be a JsonPrimitive or a JsonArray instead, which would cause the use of the [string] indexer to throw NotImplementedException
  2. person[“Name”] might throw KeyNotFoundException or return null, which will result in a NullReferenceException
  3. ReadAs<int> might throw if the value cannot be easily cast to an integer

Consider rewriting this expression as follows:

  1. var match = from person in people
  2.             where person.Value.AsDynamic.Name.ReadAs<string>("").StartsWith("S")
  3.                   && person.Value.AsDynamic.Age.ReadAs<int>(0) > 20
  4.             select person;

This expression will never throw an exception. Let’s see how we have addressed the concerns listed above.

  1. JsonValue and all its derived types now implement IEnumerable<KeyValuePair<string, JsonValue>>. The enumerator on all of the types will never return null and will return an empty collection in the worst case. This guarantees that the user can always safely write a where clause against a KeyValuePair<string, JsonValue>.
  2. Note that we have switched to using the dynamic support described in the previous requirement. The dynamic implementation uses the safe indexers underneath. We could have used the safe indexers directly, by writing something like person.Value.ValueOrDefault(“Name”).
  3. We are now using the ReadAs<T> overload, which takes a default value

The behavior for the new enumerator is as follows:

  • JsonObject – the Key will be the name of a child JsonValue, and the Value will be the value itself.
  • JsonArray – the Key will be a string containing the array index of the value, and the Value will be the value itself.
  • JsonPrimitive – empty enumerator.
  • JsonValue (JsonType.Default) – empty enumerator.

Last edited Nov 15, 2010 at 6:33 PM by yavorg, version 9

Comments

riles01 Oct 30, 2010 at 5:23 PM 
By the way, I absolutely love the LINQ support. I'm thrilled to see language-integrated Maybe monads allowing chainable computations in getting desired results. This alone would make me want to switch from any other web service platform.

riles01 Oct 30, 2010 at 5:21 PM 
Has any thought been given to an implementation of IQueryable over JsonValue? A few quick thoughts of the utility of such an implementation would be to reduce deserialization to only those objects that were wanted and/or to shape a given result into the desired object instances for the given application.