In this article, we’re going to explore the different flavors of serialization and deserialization in C#.
Let’s dive into it.
What Is Serialization and Deserialization in C#?
Serialization is the process of converting the state of an object into a form (string, byte array, or stream) that can be persisted or transported.
Deserialization is the process of converting the serialized stream of data into the original object state. This ensures that the original state is not altered and is recreated when we need it.
Why Do We Need Serialization and Deserialization?
Serialization is useful for the storage and exchange of object states. Within the application memory, the state of an object resides in complicated data structures which is unsuitable for storage or exchange over the network.
So, serialization converts the object into a shareable format.
With serialization, we can transfer objects:
- Between client and server via REST APIs or GRPC
- Over the network for messaging systems like Kafka or RabbitMQ
- Through firewalls as JSON or XML strings
- To other data stores like SQL or NoSQL databases for persistence
Deserialization is typically useful in reconstructing the object to its original state at its destination for further processing.
Demo Web API
Let’s use a Web API application that performs simple create and read operations on a list of employees:
[HttpGet]
[ProducesResponseType(typeof(IEnumerable<Employee>), StatusCodes.Status200OK)]
public IActionResult GetAllEmployees()
{
return Ok(_repo.GetAll());
}
[HttpPost]
[ProducesResponseType(typeof(Employee), StatusCodes.Status201Created)]
public IActionResult CreateEmployee(Employee employee)
{
_repo.Create(employee);
return Created($"/employees/{employee.Id}", employee);
}
In this example, the Web API exposes two endpoints that get a list of employees and create a new employee respectively.
This Web API supports content transfer in multiple formats (XML, JSON, or Protobuf). The server chooses the data transaction format at runtime depending on the Accept and Content-Type header values in the request:
builder.Services.AddControllers()
.AddProtobufFormatters()
.AddXmlSerializerFormatters();
The AddXmlSerializerFormatters() method adds both input and output formatters, so we can serialize objects to and from XML. ASP.NET Core Web API adds the JSON formatter by default. So, there is no need to add it explicitly.
These concepts are part of the broader topic called content negotiation.
We are using the NuGet package WebApiContrib.Core.Formatter.Protobuf to support the Protobuf formatter here:
dotnet add package WebApiContrib.Core.Formatter.Protobuf
Accordingly, the data contract model contains the protobuf related attributes:
[ProtoContract]
public class Employee
{
[ProtoMember(1)]
public Guid Id { get; set; }
[ProtoMember(2)]
public string Name { get; set; }
[ProtoMember(3)]
public Address? Address { get; set; }
}
[ProtoContract]
public class Address
{
[ProtoMember(1)]
public string AddressLine1 { get; set; }
[ProtoMember(2)]
public string? AddressLine2 { get; set; }
[ProtoMember(3)]
public string City { get; set; }
[ProtoMember(4)]
public string Country { get; set; }
}
The EmployeeRepository is responsible for interaction with a persistent store for the employees:
public class EmployeeRepository : IEmployeeRepository
{
private readonly Dictionary<Guid, Employee> _employees = new();
private Random _rnd = new Random();
public EmployeeRepository()
{
InitializeEmployeeStore();
}
public List<Employee> GetAll()
{
return _employees.Values.ToList();
}
public void Create(Employee employee)
{
if (employee == null)
{
return;
}
_employees[employee.Id] = employee;
}
...
}
For simplicity, let’s use an in-memory dictionary as a persistent store for the employees. However, the repositories in an enterprise application will not use an in-memory dictionary. In such cases, it will interact with a relational or non-relational data source.
Let’s send a request to the API endpoint /api/Employees that returns the list of employees. We set the Accept header value as application/json:
curl -X 'GET' \ 'https://localhost:7181/api/Employees' \ -H 'accept: application/json'
Hence, we get the response in JSON format:
[
{
"id": "4d0b2bd1-f611-453b-be0c-b7b04578893d",
"name": "TestEmployee1",
"address": {
"addressLine1": "Street #1",
"addressLine2": "District #1",
"city": "Sample City3",
"country": "Sample Country1"
}
},
{
"id": "1e071d0e-b1d7-4a06-ad3d-2fa8ad3640fb",
"name": "TestEmployee2",
"address": {
"addressLine1": "Street #2",
"addressLine2": "District #2",
"city": "Sample City4",
"country": "Sample Country3"
}
},
...
...
]
Similarly, we set the Accept header as application/xml in the request header:
curl -X 'GET' \ 'https://localhost:7181/api/Employees' \ -H 'accept: application/xml'
Consequently, we get the response in XML format:
<ArrayOfEmployee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<Employee>
<Id>4d0b2bd1-f611-453b-be0c-b7b04578893d</Id>
<Name>TestEmployee1</Name>
<Address>
<AddressLine1>Street #1</AddressLine1>
<AddressLine2>District #1</AddressLine2>
<City>Sample City3</City>
<Country>Sample Country1</Country>
</Address>
</Employee>
<Employee>
<Id>1e071d0e-b1d7-4a06-ad3d-2fa8ad3640fb</Id>
<Name>TestEmployee2</Name>
<Address>
<AddressLine1>Street #2</AddressLine1>
<AddressLine2>District #2</AddressLine2>
<City>Sample City4</City>
<Country>Sample Country3</Country>
</Address>
</Employee>
...
...
</ArrayOfEmployee>
Finally, let’s see how the response looks when we set the Accept header as application/x-protobuf:
curl -X 'GET' \ 'https://localhost:7181/api/Employees' \ -H 'accept: application/x-protobuf'
However, please note that the response in protobuf format is not human-readable:
\
�+M�;E���Ex�=
TestEmployee17
Street #1District #1Sample City3"Sample Country1
\
ױJ�=/��6@�
TestEmployee27
Street #2District #2Sample City4"Sample Country3
\
T��,ϮE��%���'
TestEmployee37
Street #3District #3Sample City2"Sample Country3
\
�e#*l�dF��-�m@�
TestEmployee47
Street #4District #4Sample City3"Sample Country3
\
Ɨu���KL�.��ֹX
TestEmployee57
Street #5District #5Sample City2"Sample Country3
Consuming the API
Let’s use a .NET console application as a consumer of the Web API described in the previous section.
The client application selects JSON, XML, or Protobuf for the exchange of data with the web API based on user input:
IClient client = format switch
{
"json" => new JsonClient(),
"cache" => new CachedJsonClient(),
"xml" => new XmlClient(),
"proto" => new ProtobufClient(),
_ => new JsonClient()
};
Depending on the functional requirements, we may need to serialize a POCO (Plain Old CLR Object)Â into a string or an array of bytes. For example, we may need to serialize a complex object into a JSON string and store it into a NoSQL DB like MongoDB or Redis.
There can also be a scenario where we use RabbitMQ for asynchronous communication between two service components. RabbitMQ doesn’t allow plain strings or complex types in the message body. In this case, we need the transfer object to be converted to byte[].
For such use cases, the client code supports string as well as byte array serialization and deserialization. This is indicated by the interfaces IStringSerializer and IByteSerilizer.
The IStringSerilizer implementation serializes a POCO to a string. Also, it can de-serialize a string to a POCO:
internal interface IStringSerializer
{
T Deserialize<T>(string text);
string Serialize(object data);
}
Similarly, the IByteSerializer serializes a POCO to an array of bytes and vice-versa:
internal interface IByteSerializer
{
T Deserialize<T>(byte[] buffer);
byte[] Serialize(object data);
}
We will explore the concrete implementations later on.
Irrespective of the format chosen by the client, we always see two lists of employees in the output. One before and the other after adding a new employee which includes the newly added employee:
Enter a format name(json, cache, xml, proto): json Employee list before adding an employee. -------------------------------------- Name:TestEmployee1 Address:Street #1, District #1, Sample City3, Sample Country1 Name:TestEmployee2 Address:Street #2, District #2, Sample City4, Sample Country3 Name:TestEmployee3 Address:Street #3, District #3, Sample City2, Sample Country3 Name:TestEmployee4 Address:Street #4, District #4, Sample City3, Sample Country3 Name:TestEmployee5 Address:Street #5, District #5, Sample City2, Sample Country3 New employee TestEmployee6 has been added Employee list after adding an employee. -------------------------------------- Name:TestEmployee1 Address:Street #1, District #1, Sample City3, Sample Country1 Name:TestEmployee2 Address:Street #2, District #2, Sample City4, Sample Country3 Name:TestEmployee3 Address:Street #3, District #3, Sample City2, Sample Country3 Name:TestEmployee4 Address:Street #4, District #4, Sample City3, Sample Country3 Name:TestEmployee5 Address:Street #5, District #5, Sample City2, Sample Country3 Name:TestEmployee6 Address:Street #6, District #6, Sample City30, Sample Country22
JSON Serialization
JSON serialization converts the public properties of an object into a string, byte array, or stream that conforms to the RFC 8259 JSON specification
In this demo, the JsonStringSerializer class implements the IStringSerializer interface. This wrapper class uses the JsonSerializer available in System.Text.Json to serialize and deserialize an object:
internal class JsonStringSerializer : IStringSerializer
{
public T Deserialize<T>(string text)
=> JsonSerializer.Deserialize<T>(text, _options);
public string Serialize(object data)
=> JsonSerializer.Serialize(data, _options);
}
JsonStringSerializer also supports the passing of different serialization options through a parameterized constructor:
private readonly JsonSerializerOptions _options;
public JsonStringSerializer()
: this(GetDefaultJsonSerializerOptions())
{
}
public JsonStringSerializer(JsonSerializerOptions options)
=> _options = options;
Let’s now see how this JsonStringSerializer is used in the JsonClient.
The GetAllEmployees() method uses the Deserialize() method in JsonStringSerializer class to convert the Web API response to a List:
public async Task<List<Employee>> GetAllEmployees()
{
var url = $"{BaseUrl}/api/employees";
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await _client.SendAsync(request);
var content = await response.Content.ReadAsStringAsync();
return _serializer.Deserialize<List<Employee>>(content);
}
We set the Accept header as application/json here to instruct the server to return the response data in a format that conforms to JSON specifications. We use the ReadAsStringAsync() method to convert the HttpContent instance to a JSON string. Now, the code deserializes the resulting JSON to the intended C# object.
As the name suggests, the CreateEmployee() method sends an employee object to the Web API to create a new employee at the server:
public async Task<Employee> CreateEmployee(Employee employee)
{
var url = $"{BaseUrl}/api/employees";
var request = new HttpRequestMessage(HttpMethod.Post, url);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
request.Content = new StringContent(_serializer.Serialize(employee), Encoding.UTF8,"application/json");
var response = await _client.SendAsync(request);
if (response.IsSuccessStatusCode)
{
var newEmployee = _serializer.Deserialize<Employee>(await response.Content.ReadAsStringAsync());
return newEmployee;
}
return null;
}
Firstly, the Content-Type header is set as application/json to indicate the format used by the object in the request payload for content negotiation between the client and the server. Then, the Serialize() method serializes the new Employee POCO to a JSON string which is then used as StringContent in the HttpRequestMessage object.
Using JSON Serialization in Caching
Let’s explore a use case where the caching mechanism uses JSON Serialization.
The IDistributedCache interface available in Microsoft.Extensions.Caching.Distributed package is used to read and write data from an underlying cache-store. The GetAsync() and SetAsync() methods in the interface only support reading and writing a byte to the underlying cache. This is where we will be using the JsonByteSerializer.
The JsonByteSerializer class implements the IByteSerializer. This class also uses the same JsonSerializer but with minor tweaks to serialize and deserialize POCOs to byte[] instead of string and vice versa:
internal class JsonByteSerializer : IByteSerializer
{
private readonly JsonSerializerOptions _options;
public T Deserialize<T>(byte[] buffer)
{
using var stream = new MemoryStream(buffer);
return JsonSerializer.Deserialize<T>(stream, _options);
}
public byte[] Serialize(object data)
{
return JsonSerializer.SerializeToUtf8Bytes(data, _options);
}
}
The CachedJsonClient uses the JsonByteSerializer to get a list of employees and also create a new employee. But, it also uses a caching mechanism to improve performance:
public async Task<List<Employee>> GetAllEmployees()
{
// Find cached item
byte[] objectFromCache = await _distributedCache.GetAsync(AllEmployeesKey);
if (objectFromCache != null)
{
var employees = _serializer.Deserialize<List<Employee>>(objectFromCache);
return employees;
}
return await RefreshCache();
}
The GetAllEmployees() method reads the list of employees from the cache. If it can read the data from the cache, it deserializes the byte[] retrieved from cache to a List using the Deserialize() method from JsonByteSerializer.
private async Task<List<Employee>> RefreshCache()
{
// If not found, then recalculate response
var content = await GetResponseFromApi();
var cacheEntryOptions = new DistributedCacheEntryOptions()
.SetSlidingExpiration(TimeSpan.FromSeconds(10))
.SetAbsoluteExpiration(TimeSpan.FromSeconds(30));
// Cache it
var resultAsByteArray = _serializer.Serialize(await content.ReadAsStringAsync());
await _distributedCache.SetAsync(AllEmployeesKey, resultAsByteArray, cacheEntryOptions);
return _serializer.Deserialize<List<Employee>>(resultAsByteArray);
}
private async Task<HttpContent> GetResponseFromApi()
{
var url = $"{BaseUrl}/api/employees";
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var response = await _client.SendAsync(request);
return response.Content;
}
If there is a cache miss, the GetAllEmployees() method gets the data from the Web API and saves it in the cache. The Serialize() method from JsonByteSerializer converts the JSON string response from Web API to a byte[] before saving it in the cache.
XML Serialization
Similar to JSON serialization, XML serialization and deserialization is the process of conversion of C# objects to XML and vice versa. The custom XmlSerializer class used in this demo implements the IStringSerializer interface. It is a wrapper over the System.Xml.Serialization.XmlSerializer provided by .NET out of the box:
internal class XmlSerializer : IStringSerializer
{
public T Deserialize<T>(string text)
{
var serializer = new System.Xml.Serialization.XmlSerializer(typeof(T));
using (var stream = new StringReader(text))
{
return (T)serializer.Deserialize(stream);
}
}
public string Serialize(object data)
{
var serializer = new System.Xml.Serialization.XmlSerializer(data.GetType());
using (var stream = new Utf8StringWriter())
{
serializer.Serialize(stream, data);
return stream.ToString();
}
}
}
The Serialize() method uses the StringWriter class from the System.IO namespace in conjunction with the out-of-the-box XML serializer to create an XML representation of the object as a string. Similarly, the Deserialize() method uses the StringReader to create an object by reading the string representation of an XML.
The XmlClient is very similar to the JsonClient. However, here, the Accept and Content-Type headers are set as application/xml instead of application/json.
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/xml"));
request.Content = new StringContent(_serializer.Serialize(employee), Encoding.UTF8, "application/xml");
Protobuf Serialization
Google developed Protocol Buffers or ProtoBuf for communication between their service components. It is a method of encoding and sending out structured data in an efficient manner. Protobuf is language-neutral, platform-neutral, and highly performant.
In this demo, the ProtobufSerializer class implements the IByteSerializer and acts as a wrapper over the Serializer class provided by the protobuf-net assembly. The protobuf-net is a contract-based serializer for the .NET code. This was installed along with the WebApiContrib.Core.Formatter.Protobuf package that we have introduced in the Demo Web API section.
The native protobuf serializer only serializes to and deserializes from the .NET Stream type. Hence, the ProtobufSerializer methods use a MemorStream in conjunction with the out of the protobuf Serializer to convert POCOs to byte[] and vice-versa:
internal class ProtobufSerializer : IByteSerializer
{
public T Deserialize<T>(byte[] buffer)
{
using (var stream = new MemoryStream(buffer))
{
return Serializer.Deserialize<T>(stream);
}
}
public byte[] Serialize(object data)
{
using (var stream = new MemoryStream())
{
Serializer.Serialize(stream, data);
stream.Flush();
return stream.ToArray();
}
}
}
Since the data encoded in protobuf format is not human readable, the IByteSerializer is a better fit in this case compared to the IStringSerializer. We could have used MemoryStream directly as well in place of byte[], But we are maintaining consistency here by implementing the IByteSerializer.
In the next section, we will learn why Protobuf has become popular in recent times over the standard formatting like JSON or XML.
Let’s see how the ProtobufSerializer is used in the ProtobuClient:
public async Task<List<Employee>> GetAllEmployees()
{
var url = $"{BaseUrl}/api/employees";
var request = new HttpRequestMessage(HttpMethod.Get, url);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-protobuf"));
var response = await _client.SendAsync(request);
var content = await response.Content.ReadAsByteArrayAsync();
return _serializer.Deserialize<List<Employee>>(content);
}
Similar to the JsonClient, The GetAllEmployees() method in ProtobufClient uses the Deserialize() method in the ProtobufSerializer class to convert the Web API response to a List. The only differences are that the Accept header is set as application/x-protobuf as part of content negotiation with the Web API. Also, the HttpConetent instance is read as byte[] from the response.
The ProtobufClient also has a CreateEmployee() method to create a new employee:
public async Task<Employee> CreateEmployee(Employee employee)
{
var url = $"{BaseUrl}/api/employees";
var request = new HttpRequestMessage(HttpMethod.Post, url);
request.Headers.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-protobuf"));
request.Content = new ByteArrayContent(_serializer.Serialize(employee));
request.Content.Headers.Add("Content-Type", "application/x-protobuf");
var response = await _client.SendAsync(request);
if (response.IsSuccessStatusCode)
{
var newEmployee = _serializer.Deserialize<Employee>(await response.Content.ReadAsByteArrayAsync());
return newEmployee;
}
return null;
}
Here, the Content-Type header is set as application/x-protobuf to indicate the format used by the object in the request payload for communication between the client and the server. The Serialize() method from the ProtobufSerializer serializes the new employee POCO to a byte[] which is then used as ByteArrayContent in the HttpRequestMessage object.
Why Is Protobuf Serialization Gaining Popularity
To date, JSON remains the most popular format to communicate over HTTP. JSON has many obvious advantages as a data-interchange format. It is human readable, well understood, and typically performs well. But, in recent times protocol buffers have become increasingly popular.
Text-based serializers like JSON and XML have certain disadvantages:
- They require complex parsing code to conform to human-readable options
- They need more bandwidth as the property names also need to be part of the payload
Protocol buffers use a binary format. Furthermore, the message does not include the metadata defining the message content. For example, if our message has a property named EmployeeName then this name is not part of the message. In XML and JSON we will include EmployeeName as a string for each occurrence of the property in the message. Hence, protocol buffer messages are more compact in comparison to the same messages in XML or JSON.
Protobuf is considered ideal for internal API communications between two server components within a distributed architecture where it helps in optimizing performance over low bandwidth networks.
There are situations when JSON/XML is a better fit than something like Protocol Buffers, including situations where:
- The data needs to be human-readable
- A web browser consumes response data from a service
- Our server application uses JavaScript
Each serialization format has its advantages and disadvantages. It depends on the context it is used.
For example, in an on-premises data center with low bandwidth, services can communicate using protobuf. Whereas, we cannot dispense with JSON when we build our web client application using JS frameworks like Angular/React. We may also need XML serialization to make a system backward compatible with a client that is in a legacy platform.
Conclusion
In this article, we have covered the different use cases where serialization and deserialization in C# play a pivotal role.
It is important to understand that we cannot consider serialization as a one-size-fits-all approach. It depends on the usage and the different components that are involved in the systems that communicate among each other.