Data Shaping in ASP.NET Core Web API

In this article, we are going to talk about a neat concept called data shaping and how to implement it in ASP.NET Core Web API. To achieve that, we are going to use similar tools as we did in the sorting article. Data shaping is not something that every API needs, but it can be very useful in some cases.

If you want to follow along with the article, you can use the start branch and if you want to get the final solution or if you get stuck, switch to the final branch.

NOTE: Some degree of previous knowledge is needed to follow this article. It relies heavily on the ASP.NET Core Web API series on Code Maze, so if you are not sure how to set up the database or how the underlying architecture works, we strongly suggest you go through the series.

Let’s start by learning what data shaping is exactly.

What is Data Shaping

Data shaping is a great way to reduce the amount of traffic sent from the API to the client. It enables the consumer of the API to select (shape) the data by choosing the fields through the query string.

What we mean by this is something like:
https://localhost:5001/api/owner?fields=name,dateOfBirth

This would tell the API to return a list of owners with ONLY the Name and DateOfBirth fields of the owner entity. In our case, the owner entity doesn’t have too many fields to choose from, but you can see how this can be very helpful when an entity has a lot of fields in it. It’s not that uncommon.

By giving the consumer a way to select just the fields it needs, we can potentially reduce the stress on the API. On the other hand, this is not something that every API needs, so we need to think carefully and decide whether we should implement it since its implementation has a bit of reflection going on.

And we know for a fact that reflection takes its toll and slows our application down.

Finally, as always, data-shaping should work well together with the concepts we’ve covered so far – paging, filtering, searching, and sorting.

Let’s get to work.

How to Implement Data Shaping in ASP.NET Core Web API

First things first, we need to extend our QueryStringParameters class since we are going to add a new feature to our query string and we want it to be available for any entity:

public abstract class QueryStringParameters
{
	const int maxPageSize = 50;
	public int PageNumber { get; set; } = 1;

	private int _pageSize = 10;
	public int PageSize
	{
		get
		{
			return _pageSize;
		}
		set
		{
			_pageSize = (value > maxPageSize) ? maxPageSize : value;
		}
	}

	public string OrderBy { get; set; }

	public string Fields { get; set; }
}

We’ve added the Fields property and now we can use fields as a query string parameter.

Next on, similarly to what we did with sorting, we are going to do here. But, we’ll make this generic to start with.

We’ll make the IDataShaper.cs and DataShaper.cs in the Helpers folder of the Entities project:

First, let’s create the IDataShaper interface:

public interface IDataShaper<T>
{
	IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string fieldsString);
	ExpandoObject ShapeData(T entity, string fieldsString);
}

The IDataShaper defines two methods that should be implemented, one for the single entity, and one for the collection of entities. Both are named ShapeData but they have different signatures.

Notice how we use the ExpandoObjecttype as a return type. We need to do that in order to shape our data how we want it.

And now, let’s see the actual implementation:

public class DataShaper<T> : IDataShaper<T>
{
	public PropertyInfo[] Properties { get; set; }

	public DataShaper()
	{
		Properties = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
	}

	public IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string fieldsString)
	{
		var requiredProperties = GetRequiredProperties(fieldsString);

		return FetchData(entities, requiredProperties);
	}

	public ExpandoObject ShapeData(T entity, string fieldsString)
	{
		var requiredProperties = GetRequiredProperties(fieldsString);

		return FetchDataForEntity(entity, requiredProperties);
	}

	private IEnumerable<PropertyInfo> GetRequiredProperties(string fieldsString)
	{
		var requiredProperties = new List<PropertyInfo>();

		if (!string.IsNullOrWhiteSpace(fieldsString))
		{
			var fields = fieldsString.Split(',', StringSplitOptions.RemoveEmptyEntries);

			foreach (var field in fields)
			{
				var property = Properties.FirstOrDefault(pi => pi.Name.Equals(field.Trim(), StringComparison.InvariantCultureIgnoreCase));

				if (property == null)
					continue;

				requiredProperties.Add(property);
			}
		}
		else
		{
			requiredProperties = Properties.ToList();
		}

		return requiredProperties;
	}

	private IEnumerable<ExpandoObject> FetchData(IEnumerable<T> entities, IEnumerable<PropertyInfo> requiredProperties)
	{
		var shapedData = new List<ExpandoObject>();

		foreach (var entity in entities)
		{
			var shapedObject = FetchDataForEntity(entity, requiredProperties);
			shapedData.Add(shapedObject);
		}

		return shapedData;
	}

	private ExpandoObject FetchDataForEntity(T entity, IEnumerable<PropertyInfo> requiredProperties)
	{
		var shapedObject = new ExpandoObject();

		foreach (var property in requiredProperties)
		{
			var objectPropertyValue = property.GetValue(entity);
			shapedObject.TryAdd(property.Name, objectPropertyValue);
		}

		return shapedObject;
	}
}

Let’s break this class down.

Implementation – Step by Step

We have one public property in this class – Properties. It’s an array of PropertyInfo’s that we’re going to pull out of the input type, whatever it is, Account or Owner in our case.

public PropertyInfo[] Properties { get; set; }

public DataShaper()
{
	Properties = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
}

So here it is, on class instantiation, we get all the properties of an input class.

Next on, we have the implementation of our two public methods ShapeData:

public IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string fieldsString)
{
	var requiredProperties = GetRequiredProperties(fieldsString);

	return FetchData(entities, requiredProperties);
}

public ExpandoObject ShapeData(T entity, string fieldsString)
{
	var requiredProperties = GetRequiredProperties(fieldsString);

	return FetchDataForEntity(entity, requiredProperties);
}

They both look similar and rely on the GetRequiredProperties method to parse the input string that contains the fields we want to fetch.

The GetRequiredProperties method is a method that does the magic. It parses the input string and returns just the properties we need to return to the controller:

private IEnumerable<PropertyInfo> GetRequiredProperties(string fieldsString)
{
	var requiredProperties = new List<PropertyInfo>();

	if (!string.IsNullOrWhiteSpace(fieldsString))
	{
		var fields = fieldsString.Split(',', StringSplitOptions.RemoveEmptyEntries);

		foreach (var field in fields)
		{
			var property = Properties.FirstOrDefault(pi => pi.Name.Equals(field.Trim(), StringComparison.InvariantCultureIgnoreCase));

			if (property == null)
				continue;

			requiredProperties.Add(property);
		}
	}
	else
	{
		requiredProperties = Properties.ToList();
	}

	return requiredProperties;
}

As you can see, there’s nothing special about it. If the fieldsString is not empty, we split it and check if the fields match the properties in our entity. If they do, we add them to the list of required properties.

On the other hand, if the fieldsString is empty, we consider all properties are required.

Now, FetchData and FetchDataForEntity are the private methods to extract the values from these required properties we’ve prepared:

FetchDataForEntity does it for a single entity:

private ExpandoObject FetchDataForEntity(T entity, IEnumerable<PropertyInfo> requiredProperties)
{
	var shapedObject = new ExpandoObject();

	foreach (var property in requiredProperties)
	{
		var objectPropertyValue = property.GetValue(entity);
		shapedObject.TryAdd(property.Name, objectPropertyValue);
	}

	return shapedObject;
}

As you can see, we loop through the requiredProperties and then using a bit of reflection, we extract the values and add them to our ExpandoObject. ExpandoObject implements IDictionary<string,object> so we can use the TryAdd method to add our property using its name as a key, and the value as a value for the dictionary.

This way we dynamically add just the properties we need to our dynamic object.

The FetchData method is just an implementation for multiple objects. It utilizes the FetchDataForEntity method we’ve just implemented:

private IEnumerable<ExpandoObject> FetchData(IEnumerable<T> entities, IEnumerable<PropertyInfo> requiredProperties)
{
	var shapedData = new List<ExpandoObject>();

	foreach (var entity in entities)
	{
		var shapedObject = FetchDataForEntity(entity, requiredProperties);
		shapedData.Add(shapedObject);
	}

	return shapedData;
}

Nothing special here, just looping through the list of entities and returning a collection of shaped entities as a result.

That’s it for the implementation, let’s see how do we connect all this into our existing solution.

Connecting the Dots

Now that we’ve implemented the logic we can inject our data shaper in the repositories as we did with ISortHelper:

private IDataShaper<Owner> _dataShaper;

public OwnerRepository(RepositoryContext repositoryContext, 
	ISortHelper<Owner> sortHelper,
	IDataShaper<Owner> dataShaper)
	: base(repositoryContext)
{
	_sortHelper = sortHelper;
	_dataShaper = dataShaper;
}

And then apply data shaping in the GetOwners method:

public PagedList<ExpandoObject> GetOwners(OwnerParameters ownerParameters)
{
	var owners = FindByCondition(o => o.DateOfBirth.Year >= ownerParameters.MinYearOfBirth &&
								o.DateOfBirth.Year <= ownerParameters.MaxYearOfBirth);

	SearchByName(ref owners, ownerParameters.Name);

	_sortHelper.ApplySort(owners, ownerParameters.OrderBy);
	var shapedOwners = _dataShaper.ShapeData(owners, ownerParameters.Fields);

	return PagedList<ExpandoObject>.ToPagedList(shapedOwners,
		ownerParameters.PageNumber,
		ownerParameters.PageSize);
}

Create another GetOwnerById method that shapes the data since we still need our regular method for validation checks in the controller actions:

public SerializableExpando GetOwnerById(Guid ownerId, string fields)
{
	var owner = FindByCondition(owner => owner.Id.Equals(ownerId))
		.DefaultIfEmpty(new Owner())
		.FirstOrDefault();

	return _dataShaper.ShapeData(owner, fields);
}

And don’t forget to modify the IOwnerRepository interface to reflect these changes:

public interface IOwnerRepository : IRepositoryBase<Owner>
{
	PagedList<ExpandoObject> GetOwners(OwnerParameters ownerParameters);
	ExpandoObject GetOwnerById(Guid ownerId, string fields);
	Owner GetOwnerById(Guid ownerId);
	void CreateOwner(Owner owner);
	void UpdateOwner(Owner dbOwner, Owner owner);
	void DeleteOwner(Owner owner);
}

You can try changing AccountRepository on your own for practice. If you get stuck, check out the finished project.

And of course, since we’ve modified the repository classes constructors, we need to modify our RepositoryWrapper too:

public class RepositoryWrapper : IRepositoryWrapper
{
	private RepositoryContext _repoContext;
	private IOwnerRepository _owner;
	private IAccountRepository _account;
	private ISortHelper<Owner> _ownerSortHelper;
	private ISortHelper<Account> _accountSortHelper;
	private IDataShaper<Owner> _ownerDataShaper;
	private IDataShaper<Account> _accountDataShaper;

	public IOwnerRepository Owner
	{
		get
		{
			if (_owner == null)
			{
				_owner = new OwnerRepository(_repoContext, _ownerSortHelper, _ownerDataShaper);
			}

			return _owner;
		}
	}

	public IAccountRepository Account
	{
		get
		{
			if (_account == null)
			{
				_account = new AccountRepository(_repoContext, _accountSortHelper, _accountDataShaper);
			}

			return _account;
		}
	}

	public RepositoryWrapper(RepositoryContext repositoryContext,
		ISortHelper<Owner> ownerSortHelper,
		ISortHelper<Account> accountSortHelper,
		IDataShaper<Owner> ownerDataShaper,
		IDataShaper<Account> accountDataShaper)
	{
		_repoContext = repositoryContext;
		_ownerSortHelper = ownerSortHelper;
		_accountSortHelper = accountSortHelper;
		_ownerDataShaper = ownerDataShaper;
		_accountDataShaper = accountDataShaper;
	}

	public void Save()
	{
		_repoContext.SaveChanges();
	}
}

Using dependency injection means we need to register our data shapers in the ConfigureRepositoryWrapper method of the ServiceExtensions class order to resolve them:

public static void ConfigureRepositoryWrapper(this IServiceCollection services)
{
	services.AddScoped<ISortHelper<Owner>, SortHelper<Owner>>();
	services.AddScoped<ISortHelper<Account>, SortHelper<Account>>();

	services.AddScoped<IDataShaper<Owner>, DataShaper<Owner>>();
	services.AddScoped<IDataShaper<Account>, DataShaper<Account>>();

	services.AddScoped<IRepositoryWrapper, RepositoryWrapper>();
}

And because we’ve done such an awesome job, we don’t need to change our GetOwners action in the OwnerController, but we do need to make a slight change to validation in the GetOwnerById action:

[HttpGet("{id}", Name = "OwnerById")]
public IActionResult GetOwnerById(Guid id, [FromQuery] string fields)
{
	var owner = _repository.Owner.GetOwnerById(id, fields);

	if (owner == default(ExpandoObject))
	{
		_logger.LogError($"Owner with id: {id}, hasn't been found in db.");
		return NotFound();
	}

	return Ok(owner);
}

As you can see the changes aren’t that drastic.

Everything is set up now. We just need to run this and it will work for sure. Or will it?

As a matter of fact, it won’t 🙂

Resolving Json Serialization Problems

Now, if we try to run the application as it is right now, we will get a big fat InvalidCastException. The reason for that is that System.Text doesn’t support the casting of ExpandoObject to IDictionary, which is what we need to happen when we return the result from the controller.

To avoid this, we need to add Microsoft.AspNetCore.Mvc.NewtonsoftJson NuGet package and then in the ConfigureServices method of our Startup.cs class find the services.AddControllers() line and add .AddNewtonsoftJson()method to the end of it:

public void ConfigureServices(IServiceCollection services)
{
	services.ConfigureCors();

	services.ConfigureIISIntegration();

	services.ConfigureLoggerService();

	services.ConfigureMySqlContext(Configuration);

	services.ConfigureRepositoryWrapper();

	services.AddControllers(config =>
	{
		config.RespectBrowserAcceptHeader = true;
		config.ReturnHttpNotAcceptable = true;
	}).AddXmlDataContractSerializerFormatters()
	.AddNewtonsoftJson();
}

What this does is replace the default System.Text.Json serializer with Newtonsoft.Json one. The exception should be gone now and the application should work normally again.

While we are at it, we are going to configure the application to respect browser headers, return 406 Not Acceptable if an unknown media type is requested, and use a data contract XML serializer because we want to support content negotiation.

This solves the problem with the json serialization.

But let’s test our solution, to see if it actually does what we specified.

Testing Our Solution

Before we test our solution, let’s quickly take a look at how our Owner table looks like right now:

Great, these entries should do it.

First, let’s send a plain GET request to our owners’ endpoint:
https://localhost:5001/api/owner

We should get a full response back:

[
    {
        "Id": "261e1685-cf26-494c-b17c-3546e65f5620",
        "Name": "Anna Bosh",
        "DateOfBirth": "1974-11-14T00:00:00",
        "Address": "27 Colored Row"
    },
    {
        "Id": "9c362f85-5581-4182-ac96-b7b88a74dda7",
        "Name": "Anna Bosh",
        "DateOfBirth": "1964-11-14T00:00:00",
        "Address": "24 Crescent Street"
    },
    {
        "Id": "24fd81f8-d58a-4bcc-9f35-dc6cd5641906",
        "Name": "John Keen",
        "DateOfBirth": "1980-12-05T00:00:00",
        "Address": "61 Wellfield Road"
    },
    {
        "Id": "f98e4d74-0f68-4aac-89fd-047f1aaca6b6",
        "Name": "Martin Miller",
        "DateOfBirth": "1983-05-21T00:00:00",
        "Address": "3 Edgar Buildings"
    },
    {
        "Id": "66774006-2371-4d5b-8518-2177bcf3f73e",
        "Name": "Nick Somion",
        "DateOfBirth": "1998-12-15T00:00:00",
        "Address": "North sunny address 102"
    },
    {
        "Id": "a3c1880c-674c-4d18-8f91-5d3608a2c937",
        "Name": "Sam Query",
        "DateOfBirth": "1990-04-22T00:00:00",
        "Address": "91 Western Roads"
    }
]

So our data shaping functionality hasn’t changed the default application behavior.

Now, let’s see data shaping in action:
https://localhost:5001/api/owner?fields=name,dateOfBirth

The API should return the shaped data:

[
    {
        "Name": "Sam Query",
        "DateOfBirth": "1990-04-22T00:00:00"
    },
    {
        "Name": "Nick Somion",
        "DateOfBirth": "1998-12-15T00:00:00"
    },
    {
        "Name": "Martin Miller",
        "DateOfBirth": "1983-05-21T00:00:00"
    },
    {
        "Name": "John Keen",
        "DateOfBirth": "1980-12-05T00:00:00"
    },
    {
        "Name": "Anna Bosh",
        "DateOfBirth": "1974-11-14T00:00:00"
    },
    {
        "Name": "Anna Bosh",
        "DateOfBirth": "1964-11-14T00:00:00"
    }
]

And now to top it off, let’s see if it works with paging, filtering, searching, and sorting:
https://localhost:5001/api/owner?fields=name,dateOfBirth&pageSize=2&pageNumber=1&orderBy=name asc,dateOfBirth desc&maxYearOfBirth=1970

This query should return only one result. Can you guess which one? If you’ve guessed, leave us a comment with what you think the result is.

That’s it, we’ve tested our API successfully.

One more thing to test. Let’s try to request an XML result.

Resolving XML Serialization Problems

Let’s change the Accept header to application/xml and send a request. We want to test out if our content negotiation works.

We are going to send a simple request this time:
https://localhost:5001/api/owner/a3c1880c-674c-4d18-8f91-5d3608a2c937

And the response looks like this:

<ArrayOfKeyValueOfstringanyType xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
    <KeyValueOfstringanyType>
        <Key>Id</Key>
        <Value xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/" i:type="d3p1:guid">a3c1880c-674c-4d18-8f91-5d3608a2c937</Value>
    </KeyValueOfstringanyType>
    <KeyValueOfstringanyType>
        <Key>Name</Key>
        <Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:string">Sam Query</Value>
    </KeyValueOfstringanyType>
    <KeyValueOfstringanyType>
        <Key>DateOfBirth</Key>
        <Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:dateTime">1990-04-22T00:00:00</Value>
    </KeyValueOfstringanyType>
    <KeyValueOfstringanyType>
        <Key>Address</Key>
        <Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:string">91 Western Roads</Value>
    </KeyValueOfstringanyType>
</ArrayOfKeyValueOfstringanyType>

As you can see that looks pretty ugly and unreadable. But that’s how our XmlDataContractSerializerOutputFormatter serializes our ExpandoObject by default.

So do we want to fix this and how do we do it?

This is a topic that’s quite outside of the scope of this article, but you can find the solution in the finished project source code.

A simple explanation is that we need to create our own dynamic object and define XML serialization rules for it.

So we need to create something like this:

public class Entity : DynamicObject, IXmlSerializable, IDictionary<string, object>
{
	//...
	//implementation
	//...
}

The main thing to notice here is that we inherit from DynamicObject that will make our object dynamic, IXmlSerializable interface, which we need to implement custom serialization rules and IDictionary<string, object> because of the Add method that is needed for XML serialization.

All that’s left is to replace the ExpandoObject type with the Entity type throughout our project.

Now, we should get a response like this one:

<Entity xmlns="http://schemas.datacontract.org/2004/07/Entities.Models">
    <Id>a3c1880c-674c-4d18-8f91-5d3608a2c937</Id>
    <Name>Sam Query</Name>
    <DateOfBirth>4/22/1990 12:00:00 AM</DateOfBirth>
    <Address>91 Western Roads</Address>
</Entity>

That looks much nicer, doesn’t it?

If the XML serialization is not important to you, you can keep using ExpandoObject, but if you want a nicely formatted XML response, this is a way to go.

Ok, let’s summarize what we’ve done so far.

Conclusion

Data shaping is an exciting and neat little feature that can really make our APIs flexible and reduce our network traffic. If we have a high volume traffic API, data shaping should work just fine. On the other hand, it’s not a feature that we should use lightly because it utilizes reflection and dynamic typing to get things done.

As with all other functionalities, we need to be careful when and if we should implement data shaping. Performance tests might come in handy even if we do implement it.

In this article we’ve covered:

• What data shaping is
• How to implement a generic data shaping solution in ASP.NET Core Web API
• How to fix our json serialization problems with ExpandoObject
• Testing of our solution by sending some simple requests
• Formatting our XML responses

If you found some parts unclear, we suggest taking a quick look at the other parts of this mini-series: paging, filtering, searching, and sorting. The paging article is especially important since we set up the infrastructure for the whole series in that article.

Hopefully, you’ve learned something new and interesting this time. In the next article, we’re going to cover HATEOAS implementation.