Updated on
Data shaping lets the client decide which fields come back. A request for /api/owners?fields=name,dateOfBirth returns owners carrying those two properties and nothing else.
The pattern travels under several names. JSON:API calls it sparse fieldsets, Google’s API design guide calls it a partial response, and plenty of teams just call it field selection. They all describe the same thing this article calls data shaping.
It is not free. The response stops being a known type, so we build it at runtime with reflection, which is why the second half of this article is about what that costs and when it is worth paying.
NOTE: Some degree of previous knowledge is needed to follow this article. It relies heavily on the ASP.NET Core Web API series on Code Maze, so if you are not sure how to set up the database or how the underlying architecture works, we strongly suggest you go through the series.
Let’s start by learning what data shaping is exactly.
What Is Data Shaping in a Web API?
Data shaping lets the client choose which fields come back, by naming them in the query string. A request for /api/owners?fields=name,dateOfBirth returns owners carrying only those two properties.
The pattern has other names. JSON:API calls it sparse fieldsets, Google’s API design guide calls it a partial response, and GraphQL makes it the entire point of the query language. They all solve one problem: a fixed response shape sends every consumer the same payload, whether or not it needs every field.
The cost is that the response is no longer a known type. We build it at runtime from a dictionary of property names and values, so the compiler cannot check it, the generated OpenAPI document can only call it an untyped object, and every request pays for reflection over the entity’s properties.
That trade is what decides whether data shaping belongs in an API. Wide entities and bandwidth-sensitive clients make it pay. A handful of small DTOs does not.
By giving the consumer a way to select just the fields it needs, we can potentially reduce the stress on the API. On the other hand, this is not something that every API needs, so we need to think carefully and decide whether we should implement it since its implementation has a bit of reflection going on.
And we know for a fact that reflection takes its toll and slows our application down.
Finally, as always, data-shaping should work well together with the concepts we’ve covered so far, paging, filtering, searching, and sorting.
Let’s get to work.
How Do We Add a fields Query Parameter?
Three pieces do the work. A Fields property on the query-string parameters class carries the raw string. A generic DataShaper<T> turns that string into a list of PropertyInfo and reads their values off each entity. The controller passes the parameter through and returns whatever comes back.
The shaper caches nothing about the request and everything about the type. typeof(T).GetProperties() runs once per shaper instance, and matching a requested field name against that array is a case-insensitive string comparison.
Unrecognised field names are skipped rather than rejected. A request for ?fields=name,nope returns the name and ignores the rest, which stops a client’s typo from becoming a 400 it cannot act on.
An empty fields value means every property. That keeps the endpoint’s default behaviour identical to what it was before shaping existed, so adding the feature breaks no client that never asks for it.
First things first, we need to extend our QueryStringParameters class since we are going to add a new feature to our query string and we want it to be available for any entity:
public abstract class QueryStringParameters
{
const int MaxPageSize = 50;
private int _pageNumber = 1;
private int _pageSize = 10;
public int PageNumber
{
get => _pageNumber;
set => _pageNumber = Math.Max(value, 1);
}
public int PageSize
{
get => _pageSize;
set => _pageSize = Math.Clamp(value, 1, MaxPageSize);
}
public string? OrderBy { get; set; }
public string? Fields { get; set; }
}
We’ve added the Fields property and now we can use fields as a query string parameter.
Next on, similarly to what we did with sorting, we are going to do here. But, we’ll make this generic to start with.
We’ll make the IDataShaper.cs and DataShaper.cs in the Helpers folder of the Entities project:
First, let’s create the IDataShaper interface:
public interface IDataShaper<T>
{
IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string? fieldsString);
ExpandoObject ShapeData(T entity, string? fieldsString);
}
The IDataShaper defines two methods that should be implemented, one for the single entity, and one for the collection of entities. Both are named ShapeData but they have different signatures.
Notice how we use the ExpandoObject type as a return type. We need to do that in order to shape our data how we want it.
And now, let’s see the actual implementation:
public class DataShaper<T> : IDataShaper<T>
{
private readonly PropertyInfo[] _properties;
public DataShaper()
{
_properties = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
}
public IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string? fieldsString)
{
var requiredProperties = GetRequiredProperties(fieldsString);
return entities.Select(entity => FetchDataForEntity(entity, requiredProperties)).ToList();
}
public ExpandoObject ShapeData(T entity, string? fieldsString)
{
var requiredProperties = GetRequiredProperties(fieldsString);
return FetchDataForEntity(entity, requiredProperties);
}
private IEnumerable<PropertyInfo> GetRequiredProperties(string? fieldsString)
{
if (string.IsNullOrWhiteSpace(fieldsString))
return _properties;
var requiredProperties = new List<PropertyInfo>();
var fields = fieldsString.Split(',', StringSplitOptions.RemoveEmptyEntries);
foreach (var field in fields)
{
var property = _properties.FirstOrDefault(pi =>
pi.Name.Equals(field.Trim(), StringComparison.OrdinalIgnoreCase));
if (property is null)
continue;
requiredProperties.Add(property);
}
return requiredProperties;
}
private static ExpandoObject FetchDataForEntity(T entity, IEnumerable<PropertyInfo> requiredProperties)
{
var shapedObject = new ExpandoObject();
foreach (var property in requiredProperties)
shapedObject.TryAdd(property.Name, property.GetValue(entity)!);
return shapedObject;
}
}
Let’s break this class down.
Building the Generic Data Shaper
We have one private field in this class, _properties. It’s an array of PropertyInfo‘s that we pull out of the input type, whatever it is, Account or Owner in our case:
private readonly PropertyInfo[] _properties;
public DataShaper()
{
_properties = typeof(T).GetProperties(BindingFlags.Public | BindingFlags.Instance);
}
So here it is, on class instantiation, we get all the properties of an input class. The field is readonly on purpose, and that single keyword is what makes the performance advice at the end of this article safe to follow.
Next on, we have the implementation of our two public ShapeData methods:
public IEnumerable<ExpandoObject> ShapeData(IEnumerable<T> entities, string? fieldsString)
{
var requiredProperties = GetRequiredProperties(fieldsString);
return entities.Select(entity => FetchDataForEntity(entity, requiredProperties)).ToList();
}
public ExpandoObject ShapeData(T entity, string? fieldsString)
{
var requiredProperties = GetRequiredProperties(fieldsString);
return FetchDataForEntity(entity, requiredProperties);
}
They both look similar and rely on the GetRequiredProperties method to parse the input string that contains the fields we want to fetch.
The GetRequiredProperties method is a method that does the magic. It parses the input string and returns just the properties we need to return to the controller:
private IEnumerable<PropertyInfo> GetRequiredProperties(string? fieldsString)
{
if (string.IsNullOrWhiteSpace(fieldsString))
return _properties;
var requiredProperties = new List<PropertyInfo>();
var fields = fieldsString.Split(',', StringSplitOptions.RemoveEmptyEntries);
foreach (var field in fields)
{
var property = _properties.FirstOrDefault(pi =>
pi.Name.Equals(field.Trim(), StringComparison.OrdinalIgnoreCase));
if (property is null)
continue;
requiredProperties.Add(property);
}
return requiredProperties;
}
As you can see, there’s nothing special about it. If the fieldsString is not empty, we split it and check if the fields match the properties in our entity. If they do, we add them to the list of required properties. A field name that matches nothing is skipped by that continue, so a typo costs the client one missing field rather than a failed request.
The comparison is OrdinalIgnoreCase because property names are programmatic identifiers rather than user-facing text, and ordinal is both the documented choice for identifiers and the faster one.
On the other hand, if the fieldsString is empty, we consider all properties are required, which is why an endpoint behaves exactly as it did before shaping was added.
Now, FetchDataForEntity is the private method that extracts the values from these required properties we’ve prepared:
private static ExpandoObject FetchDataForEntity(T entity, IEnumerable<PropertyInfo> requiredProperties)
{
var shapedObject = new ExpandoObject();
foreach (var property in requiredProperties)
shapedObject.TryAdd(property.Name, property.GetValue(entity)!);
return shapedObject;
}
As you can see, we loop through the requiredProperties and then using a bit of reflection, we extract the values and add them to our ExpandoObject. ExpandoObject implements IDictionary<string, object> so we can use the TryAdd method to add our property using its name as a key, and the value as a value for the dictionary.
This way we dynamically add just the properties we need to our dynamic object.
For a collection, ShapeData is a single Select over that method, so the whole per-entity cost of shaping is one GetValue() call per requested field.
That’s it for the implementation, let’s see how do we connect all this into our existing solution.
Connecting the Dots
Now that we’ve implemented the logic we can inject our data shaper in the repositories as we did with ISortHelper:
private readonly ISortHelper<Owner> _sortHelper;
private readonly IDataShaper<Owner> _dataShaper;
public OwnerRepository(RepositoryContext repositoryContext,
ISortHelper<Owner> sortHelper,
IDataShaper<Owner> dataShaper)
: base(repositoryContext)
{
_sortHelper = sortHelper;
_dataShaper = dataShaper;
}
And then apply data shaping in the GetOwners method:
public async Task<PagedList<ExpandoObject>> GetOwners(OwnerParameters ownerParameters)
{
var owners = FindAll();
if (ownerParameters.MinYearOfBirth is { } minYear)
owners = owners.Where(o => o.DateOfBirth >= new DateTime(minYear, 1, 1));
if (ownerParameters.MaxYearOfBirth is { } maxYear)
owners = owners.Where(o => o.DateOfBirth < new DateTime(maxYear + 1, 1, 1));
owners = owners.Search(ownerParameters.SearchTerm);
var sortedOwners = _sortHelper.ApplySort(owners.OrderBy(o => o.Name), ownerParameters.OrderBy);
var pagedOwners = await PagedList<Owner>.ToPagedListAsync(sortedOwners,
ownerParameters.PageNumber,
ownerParameters.PageSize);
var shapedOwners = _dataShaper.ShapeData(pagedOwners, ownerParameters.Fields).ToList();
return new PagedList<ExpandoObject>(shapedOwners,
pagedOwners.TotalCount,
pagedOwners.CurrentPage,
pagedOwners.PageSize);
}
The order of those last three statements is the whole point, and it is worth reading twice. Everything up to ToPagedListAsync is still an IQueryable<Owner>, so the filter, the search, the sort and the page all travel to the database as one query. Only the ten rows that come back are shaped. Shaping first would force the entire matching set into memory before a single row was discarded.
The paging metadata for the X-Pagination header comes off pagedOwners, the paged query, rather than off the shaped list, so the counts stay correct no matter what the client asked for in fields.
Create another GetOwnerById method that shapes the data since we still need our regular method for validation checks in the controller actions:
public ExpandoObject GetOwnerById(Guid ownerId, string? fields) =>
_dataShaper.ShapeData(GetOwnerById(ownerId), fields);
And don’t forget to modify the IOwnerRepository interface to reflect these changes:
public interface IOwnerRepository : IRepositoryBase<Owner>
{
Task<PagedList<ExpandoObject>> GetOwners(OwnerParameters ownerParameters);
ExpandoObject GetOwnerById(Guid ownerId, string? fields);
Owner GetOwnerById(Guid ownerId);
void CreateOwner(Owner owner);
void UpdateOwner(Owner dbOwner, Owner owner);
void DeleteOwner(Owner owner);
}
You can try changing AccountRepository on your own for practice. If you get stuck, check out the finished project linked at the top of this article.
And of course, since we’ve modified the repository classes constructors, we need to modify our RepositoryWrapper too:
public class RepositoryWrapper : IRepositoryWrapper
{
private readonly RepositoryContext _repoContext;
private readonly ISortHelper<Owner> _ownerSortHelper;
private readonly ISortHelper<Account> _accountSortHelper;
private readonly IDataShaper<Owner> _ownerDataShaper;
private readonly IDataShaper<Account> _accountDataShaper;
private IOwnerRepository? _owner;
private IAccountRepository? _account;
public RepositoryWrapper(RepositoryContext repositoryContext,
ISortHelper<Owner> ownerSortHelper,
ISortHelper<Account> accountSortHelper,
IDataShaper<Owner> ownerDataShaper,
IDataShaper<Account> accountDataShaper)
{
_repoContext = repositoryContext;
_ownerSortHelper = ownerSortHelper;
_accountSortHelper = accountSortHelper;
_ownerDataShaper = ownerDataShaper;
_accountDataShaper = accountDataShaper;
}
public IOwnerRepository Owner => _owner ??= new OwnerRepository(_repoContext, _ownerSortHelper, _ownerDataShaper);
public IAccountRepository Account => _account ??= new AccountRepository(_repoContext, _accountSortHelper, _accountDataShaper);
public void Save() => _repoContext.SaveChanges();
}
Using dependency injection means we need to register our data shapers in the ConfigureRepositoryWrapper method of the ServiceExtensions class in order to resolve them:
public static void ConfigureRepositoryWrapper(this IServiceCollection services)
{
services.AddScoped<ISortHelper<Owner>, SortHelper<Owner>>();
services.AddScoped<ISortHelper<Account>, SortHelper<Account>>();
services.AddSingleton<IDataShaper<Owner>, DataShaper<Owner>>();
services.AddSingleton<IDataShaper<Account>, DataShaper<Account>>();
services.AddScoped<IRepositoryWrapper, RepositoryWrapper>();
}
The shapers are singletons rather than scoped, and the last section of this article explains why that is the registration that matters most here.
And because we’ve done such an awesome job, we don’t need to change our GetOwners action in the OwnerController, but we do need to make a slight change to validation in the GetOwnerById action:
[HttpGet("{id}", Name = "OwnerById")]
public IActionResult GetOwnerById(Guid id, [FromQuery] string? fields)
{
var dbOwner = _repository.Owner.GetOwnerById(id);
if (dbOwner.IsEmptyObject())
{
_logger.LogError($"Owner with id: {id}, hasn't been found in db.");
return NotFound();
}
return Ok(_repository.Owner.GetOwnerById(id, fields));
}
We look the owner up as a typed Owner first, decide whether it exists, and only then ask for the shaped version. Validating a shaped object is awkward for the same reason shaping is useful: the client may have asked for a payload that does not include the id.
As you can see the changes aren’t that drastic.
One thing this article used to ask for is no longer needed. Older versions switched the whole application to Newtonsoft.Json to serialise ExpandoObject, because System.Text.Json once threw on it. That is no longer necessary: System.Text.Json handles ExpandoObject on its own, as a single object and as a collection, with no configuration and no extra package. If you find that instruction in an older tutorial or an old answer online, skip it.
One note before we run it. The finished sample in the repository uses a custom Entity type in place of ExpandoObject everywhere, for the XML reason the “Resolving XML Serialization Problems” section below explains. Everything above compiles and behaves identically with either type; the swap is the last step of the article, not the first.
Everything is set up now, so let’s run it.
Testing Our Solution
Our Owner table holds five rows after the setup script runs, which is enough to see shaping and paging interact.
First, let’s send a plain GET request to our owners’ endpoint:
https://localhost:5001/api/owners
We should get a full response back:
[
{
"Id": "261e1685-cf26-494c-b17c-3546e65f5620",
"Name": "Anna Bosh",
"DateOfBirth": "1974-11-14T00:00:00",
"Address": "27 Colored Row"
},
{
"Id": "24fd81f8-d58a-4bcc-9f35-dc6cd5641906",
"Name": "John Keen",
"DateOfBirth": "1980-12-05T00:00:00",
"Address": "61 Wellfield Road"
},
{
"Id": "f98e4d74-0f68-4aac-89fd-047f1aaca6b6",
"Name": "Martin Miller",
"DateOfBirth": "1983-05-21T00:00:00",
"Address": "3 Edgar Buildings"
},
{
"Id": "66774006-2371-4d5b-8518-2177bcf3f73e",
"Name": "Nick Somion",
"DateOfBirth": "1998-12-15T00:00:00",
"Address": "North sunny address 102"
},
{
"Id": "a3c1880c-674c-4d18-8f91-5d3608a2c937",
"Name": "Sam Query",
"DateOfBirth": "1990-04-22T00:00:00",
"Address": "91 Western Roads"
}
]
So our data shaping functionality hasn’t changed the default application behavior.
Now, let’s see data shaping in action:
https://localhost:5001/api/owners?fields=name,dateOfBirth
The API should return the shaped data:
[
{
"Name": "Anna Bosh",
"DateOfBirth": "1974-11-14T00:00:00"
},
{
"Name": "John Keen",
"DateOfBirth": "1980-12-05T00:00:00"
},
{
"Name": "Martin Miller",
"DateOfBirth": "1983-05-21T00:00:00"
},
{
"Name": "Nick Somion",
"DateOfBirth": "1998-12-15T00:00:00"
},
{
"Name": "Sam Query",
"DateOfBirth": "1990-04-22T00:00:00"
}
]
Notice the keys came back PascalCase, while every unshaped endpoint in the same API answers in camelCase. That is not a bug in our shaper, and there is a one-line fix for it further down.
A field name we do not recognise is simply dropped. Asking for ?fields=name,nope returns objects carrying Name and nothing else, rather than a 400.
And now to top it off, let’s see if it works with paging, filtering, searching, and sorting:
https://localhost:5001/api/owners?fields=name,dateOfBirth&pageSize=2&pageNumber=1&orderBy=name asc,dateOfBirth desc&maxYearOfBirth=1974
This query returns a single result:
[
{
"Name": "Anna Bosh",
"DateOfBirth": "1974-11-14T00:00:00"
}
]
That’s it, we’ve tested our API successfully.
One more thing to test. Let’s try to request an XML result.
Resolving XML Serialization Problems
Let’s change the Accept header to application/xml and send a request. We want to test out if our content negotiation works.
We are going to send a simple request this time:
https://localhost:5001/api/owners/a3c1880c-674c-4d18-8f91-5d3608a2c937
And the response looks like this:
<ArrayOfKeyValueOfstringanyType xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
<KeyValueOfstringanyType>
<Key>Id</Key>
<Value xmlns:d3p1="http://schemas.microsoft.com/2003/10/Serialization/" i:type="d3p1:guid">a3c1880c-674c-4d18-8f91-5d3608a2c937</Value>
</KeyValueOfstringanyType>
<KeyValueOfstringanyType>
<Key>Name</Key>
<Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:string">Sam Query</Value>
</KeyValueOfstringanyType>
<KeyValueOfstringanyType>
<Key>DateOfBirth</Key>
<Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:dateTime">1990-04-22T00:00:00</Value>
</KeyValueOfstringanyType>
<KeyValueOfstringanyType>
<Key>Address</Key>
<Value xmlns:d3p1="http://www.w3.org/2001/XMLSchema" i:type="d3p1:string">91 Western Roads</Value>
</KeyValueOfstringanyType>
</ArrayOfKeyValueOfstringanyType>
As you can see that looks pretty ugly and unreadable. But that’s how our XmlDataContractSerializerOutputFormatter serializes our ExpandoObject by default: as a dictionary, because that is what an ExpandoObject is.
So do we want to fix this and how do we do it?
A full explanation is quite outside of the scope of this article, but the finished sample linked at the top implements it. The short version is that we need our own dynamic object with XML serialization rules of its own.
So we need to create something like this:
public class Entity : DynamicObject, IXmlSerializable, IDictionary<string, object>
{
//...
//implementation
//...
}
The main thing to notice here is that we inherit from DynamicObject that will make our object dynamic, IXmlSerializable interface, which we need to implement custom serialization rules and IDictionary<string, object> because of the Add method that is needed for XML serialization.
All that’s left is to replace the ExpandoObject type with the Entity type throughout our project, which is exactly what the sample in the repository does.
Now, we should get a response like this one:
<Entity xmlns="http://schemas.datacontract.org/2004/07/Entities.Models">
<Id>a3c1880c-674c-4d18-8f91-5d3608a2c937</Id>
<Name>Sam Query</Name>
<DateOfBirth>4/22/1990 12:00:00 AM</DateOfBirth>
<Address>91 Western Roads</Address>
</Entity>
That looks much nicer, doesn’t it?
One detail to be aware of before shipping this: Entity writes each value with ToString(), which is why the date arrives as 4/22/1990 12:00:00 AM rather than the ISO form the dictionary serializer produced. Clean element names cost us a culture-dependent date, so a consumer parsing that XML outside the same culture needs a round-trip format written into the serialization code.
If the XML serialization is not important to you, you can keep using ExpandoObject, but if you want a nicely formatted XML response, this is a way to go.
Does Data Shaping Slow Down Our API?
Reflection is the cost, and most of it is avoidable. GetProperties() is the expensive call, and its result depends only on T, so it should run once for the lifetime of the process rather than once per request.
Register the shaper as a singleton rather than a scoped service. DataShaper<T> holds no per-request state once its property array is read-only, so a single instance can serve every request for that type.
GetValue() still runs once per requested field per entity, and that is the part that scales. Shape after paging, never before, so the reflection runs across one page of rows instead of the whole result set.
Measure before assuming. On a narrow entity the shaping cost disappears next to the database round trip. On a wide entity across a large page, it does not.
| Approach | JSON via System.Text.Json | XML via the DataContract formatter | Our own reflection | Use when |
|---|---|---|---|---|
ExpandoObject | Serializes correctly; keys stay PascalCase | Verbose ArrayOfKeyValueOfstringanyType | One GetValue() per field, per row | The project already uses it, or something downstream needs dynamic |
Dictionary<string, object?> | Identical output to ExpandoObject | Identical verbose form | Same | Default choice, same result with no DLR |
Entity : DynamicObject, IXmlSerializable | Identical output | Clean <Entity> elements, but dates written with ToString() | Same | XML responses matter (this article builds it) |
| Anonymous type | Serializes, keys camelCased | 406 Not Acceptable | None | The field set is fixed at compile time |
JsonObject | Serializes correctly | 500, the formatter cannot write it | None | JSON only, and the response is assembled by hand |
The PascalCase keys in the shaped responses above are the one surprise worth understanding. PropertyNamingPolicy, which ASP.NET Core sets to camelCase, renames properties. A shaped object is a dictionary, and dictionary keys are governed by the separate DictionaryKeyPolicy, which is unset by default. So the moment an API adds data shaping, its shaped endpoints answer in PascalCase while every other endpoint answers in camelCase.
Setting DictionaryKeyPolicy = JsonNamingPolicy.CamelCase in AddJsonOptions() brings shaped responses back in line with the rest of the API.
The two code changes this section asks for are coupled and must travel together. Making _properties a readonly field is what allows a single shaper instance to be shared safely across concurrent requests; registering a singleton while that array is still publicly settable would let one request swap the properties out from under every other request in flight.
Everything else here is measurement rather than advice, and the site’s own write-up on reflection in C# is the place to start if you want to know why GetProperties() costs what it does.
The Ultimate ASP.NET Core Web API course builds the same feature on a service-layer architecture and benchmarks it, which is the part this article can only point at.
Conclusion
Data shaping is an exciting and neat little feature that can really make our APIs flexible and reduce our network traffic. If we have a high volume traffic API, data shaping should work just fine. On the other hand, it’s not a feature that we should use lightly because it utilizes reflection and dynamic typing to get things done.
As with all other functionalities, we need to be careful when and if we should implement data shaping.
In this article we’ve covered:
- What data shaping is, and the other names the pattern goes by
- How to implement a generic data shaping solution in ASP.NET Core Web API
- Testing of our solution by sending some simple requests
- Formatting our XML responses
- What data shaping costs, and how to keep that cost down
If you found some parts unclear, we suggest taking a quick look at the other parts of this mini-series: paging, filtering, searching, and sorting. The paging article is especially important since we set up the infrastructure for the whole series in that article.
Hopefully, you’ve learned something new and interesting this time. In the next article, we’re going to cover HATEOAS implementation.
Tested with .NET 10.0.10, EF Core 10.0.11, and SQL Server Express LocalDB 13.0.4001.0.
