In this article, we are going to use the popular open-source search and analytics engine – Elasticsearch to power a simple search system in ASP.NET Core.
Before we jump into code, in the first section letâs spend a moment explaining Elasticsearch and some common use cases.
What is Elasticsearch?
Elasticsearch is a free, open-source search database based on the Lucene search library.Â
Some key features include:
- Distributed and scalable, including the ability for sharding and replicas
- Documents stored as JSON
- All interactions over a RESTful HTTP API
- Handy companion software called Kibana which allows interrogation and analysis of data
- A wealth of client-side libraries for all popular languages
In a previous article, we discussed CQRS and how sometimes weâd like to split out the read system into a separate database. Elasticsearch can fit this situation perfectly, as itâs optimized for the read scenarios and provides near real-time search functionality because of the way the engine is designed.
In the next section, letâs look at how to connect to our local Elasticsearch cluster in an ASP.NET Core application.
Adding Data to Elasticsearch
There are a number of ways to add data to Elasticsearch, but a simple way for our purposes is to make use of the Bulk REST API, which allows us to send simple curl requests to Elasticsearch. Documents in Elasticsearch are stored in âindexesâ, which can be thought of as âtablesâ in a relational database.
In this article, we are going to use âBooksâ as sample data. You can download the sample data file here.
Letâs open up a command prompt and run the command (change the port as necessary):
curl -XPOST localhost:9200/books/book/_bulk --data-binary @sample-data.json -H "Content-Type: application/json"
The command simply reads the sample JSON file, and adds the data to a new index called âbooksâ. Itâs worth mentioning here that instead of allowing Elasticsearch to create the document mappings for us (what we are doing here), we can have more control over this behavior. However, thatâs a more advanced topic so here we are just accepting the defaults.
We should see lots of output in the command window signifying all the successful index operations. To confirm everything worked, letâs open up a browser and go to http://localhost:9200/_search
:
We can see that our sample data was inserted successfully into Elasticsearch, which is fantastic!
In the next section, weâll see how to connect to Elasticsearch in ASP.NET.
Connecting to Elasticsearch in ASP.NET Core
To demonstrate how to use Elasticsearch in ASP.NET Core, weâre going to create a simple web application with a text field input. When the user enters some text and clicks a button, weâll send that search query to Elasticsearch.
Creating the ASP.NET Core MVC application
Letâs get started by creating a standard ASP.NET Core application in Visual Studio, choosing Web Application (MVC) as the type:
Creating the model
We will need a class to deserialize the JSON results from Elasticsearch, so letâs add the following class called âBookâ to the Models folder:
public class Book { public string Title { get; set; } public string Isbn { get; set; } public int PageCount { get; set; } public string ThumbnailUrl { get; set; } public string ShortDescription { get; set; } public string LongDescription { get; set; } public string Status { get; set; } public string Authors { get; set; } public string Categories { get; set; } }
Creating an Elasticsearch connection
To connect to Elasticsearch, weâll use the official NEST library developed by the team at Elastic.
Letâs open up the Package Manager Console and install NEST:
PM> install-package NEST
Next, letâs open up Startup.cs and create the connection in the ConfigureServices method:
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200")); var settings = new ConnectionSettings(pool) .DefaultIndex(âbooksâ); var client = new ElasticClient(settings); services.AddSingleton(client);
There are a lot more configuration options available when creating the connection, but letâs keep things very simple here and keep all the defaults.
Creating the Search interface
Next, letâs open up Index.cshtml and add the following HTML:
@model Nest.ISearchResponse<Book> <form asp-controller="Home" asp-action="Index" method="post"> <input type="text" name="query" /> <input type="submit" value="Search" /> </form> @if (Model.Documents.Any()) { <h3>Search Results</h3> <ul> @foreach (var book in Model.Documents) { <li>Title: @book.Title (ISBN: @book.Isbn)</li> } </ul> }
Letâs discuss our HTML:
- First, we strongly type the view to ISearchResponse<Book>. This is the type of response from Elasticsearch calls
- Then we add an HTML form which will allow us to post the search query back to the server, which we will then pass along to Elasticsearch and render the results.
- Finally, we add a simple loop that iterates through the matching documents and renders the title and ISBN.
Now that weâve got the foundation set up, in the next section weâll explore the foundation of searches in Elasticsearch, âQueriesâ.
Elasticsearch Queries
Now that we have our data in Elasticsearch, we want to be able to query it. Elasticsearch offers a very powerful DSL to perform numerous types of search operations, and the NEST library offers two approaches to use that DSL: object initializer syntax, and a Fluent API. In this article, we are going to use the Fluent API.
MatchAll query
The simplest query of all is the MatchAll query, which as the name suggests returns all the documents in an index. It can be likened to a âSELECT *â query in a relational database.
Letâs open up HomeController and firstly assign a local instance of ElasticClient:
private readonly ElasticClient _client; public HomeController(ILogger<HomeController> logger, ElasticClient client) { _logger = logger; _client = client; }
Next, letâs modify the Index() method:
public IActionResult Index() { var results = _client.Search<Book>(s => s .Query(q => q .MatchAll() ) ); return View(results); }
Here weâre using the Search method on the client, specifying our Book model as the type parameter. This tells NEST how to deserialize the results coming back from Elasticsearch.
Weâre then calling the MatchAll() method on the Query() method to do the search and return the View with the results.
If we run our app, we should see the following:
These results are the first ten books from the index.Â
Term query
Probably the most common query in Elasticsearch is the Term query. This allows us to find documents matching an exact query, which is great for scenarios like searching by ID or a simple value.
Letâs modify our Index() method again:
public IActionResult Index(string query) { ISearchResponse<Book> results; if (!string.IsNullOrWhiteSpace(query)) { results = _client.Search<Book>(s => s .Query(q => q .Term(t => t .Field(f => f.Isbn) .Value(query) ) ) ); } else { results = _client.Search<Book>(s => s .Query(q => q .MatchAll() ) ); } return View(results); }
Weâve amended our code to accept a parameter called âqueryâ into the method. This corresponds to the text field submitted in the form post, which if supplied uses the Term method to search upon the ISBN field.
Letâs run our application again, and if we enter one of the ISBNs and hit search, the page should be refreshed with the expected result:
Match query
Often with search systems, we want the user to enter in some text, and we want that text to match any part of the content in the document. Thatâs where the Match query comes in.
Letâs modify the first if statement in the Index method again:
results = _client.Search<Book>(s => s .Query(q => q .Match(t => t .Field(f => f.Title) .Query(query) ) ) );
The syntax is very similar to the Term query. This time weâre searching upon the Title field, which has multiple words, which Elasticsearch analyses in order for us to query upon.
Letâs run our application again, and this time enter the text android and hit Search:
There are a few key points here worth mentioning:
- We match multiple documents
- All matching documents contain the text âandroidâ in the title
- Even though we specified android in lowercase, it still matched documents with Android (title case). This behavior can be overridden with custom analyzers, but thatâs an advanced topic for another time.
There are a lot of other queries supported by Elasticsearch, but the Term and Match queries form the basic use cases.
In the next section, weâll go over another Elasticsearch feature called Aggregations.
Elasticsearch Aggregations
When we have a large data set, often we want to summarise or âaggregateâ that data, to serve functionality like:
- Summary page
- Paging or counts
- Faceted navigation
- Tag bubbles
This is where we can use aggregations to quickly compute results.
Range Aggregation for PageCount
Since we are dealing with books, it might be interesting to see some statistics on the number of pages. For this, we can make use of the Range aggregation.
Letâs modify our MatchAll query to include this aggregation:
results = _client.Search<Book>(s => s .Query(q => q .MatchAll() ) .Aggregations(a => a .Range("pageCounts", r => r .Field(f => f.PageCount) .Ranges(r => r.From(0), r => r.From(200).To(400), r => r.From(400).To(600), r => r.From(600) ) ) ) );
Here in addition to our MatchAll query, we are using the Aggregations method to specify some aggregations, in our case the âRangeâ aggregation.
Letâs open our Index.cshtml and add some HTML to render our new aggregation:
@if (Model.Aggregations != null) { <h3>Aggregations</h3> @if (Model.Aggregations.ContainsKey("pageCounts")) { <h4>PageCounts (Range)</h4> <ul> @foreach (var bucket in Model.Aggregations.Range("pageCounts").Buckets) { <li>@bucket.Key: @bucket.DocCount</li> } </ul> } }
Similar to the previous HTML, weâre simply looping through each âbucketâ (or group) in the pageCounts
aggregations, and rendering the information about it.
Letâs run our app again, and we should see the results:
Terms Aggregation for Categories
We mentioned previously the use case of tag bubbles or faceted UI systems. This is where the Terms aggregation comes in. Itâs similar to a âGROUP BYâ clause in a relational database, where we can get statistics on various words (or âtermsâ) across documents.
Letâs modify our Aggregations code again:
results = _client.Search<Book>(s => s .Query(q => q .MatchAll() ) .Aggregations(a => a .Range("pageCounts", r => r .Field(f => f.PageCount) .Ranges(r => r.From(0), r => r.From(200).To(400), r => r.From(400).To(600), r => r.From(600) ) ) .Terms("categories", t => t .Field("categories.keyword") ) ) );
Notice weâve added the âTermsâ aggregation in addition to the existing Range aggregation. An interesting part here is the field value set to the string âcategories.keywordâ. The reason for this is that Terms aggregation is an expensive operation, and isnât usually done on âtextâ fields. However, all text fields by default have backing âkeywordâ sub-fields, which are optimized for this use case.Â
This approach shows how we can do the terms aggregation on text fields. Another approach would be to explicitly map the field as âkeywordâ, but then we lose the full-text search functionality like âMatchâ. So this approach gives us the best of both worlds.
Letâs modify our Index.cshtml for our new aggregation:
@if (Model.Aggregations != null) { <h3>Aggregations</h3> @if (Model.Aggregations.ContainsKey("pageCounts")) { <h4>PageCounts (Range)</h4> <ul> @foreach (var bucket in Model.Aggregations.Range("pageCounts").Buckets) { <li>@bucket.Key: @bucket.DocCount</li> } </ul> } @if (Model.Aggregations.ContainsKey("categories")) { <h4>Categories (Terms)</h4> <ul> @foreach (var bucket in Model.Aggregations.Terms("categories").Buckets) { <li>@bucket.Key: @bucket.DocCount</li> } </ul> } }
Letâs run our app again:
As expected we now have the number of documents matching each of the categories.
This approach shows how we can do the terms aggregation on text fields. Another approach would be to explicitly map the field as âkeywordâ, but then we lose the full-text search functionality like âMatchâ. So this approach gives us the best of both worlds.
Conclusion
In this article, we provided a very basic introduction to Elasticsearch in ASP.NET Core. We touched on some basic queries and aggregations that are the starting point for most developers. Elasticsearch is a very mature database with lots more features that have been introduced here, but hopefully, this puts you in the right direction.