Transforming Result Documents - Ultimate Solr Guide

Document Transformers modify the information returned about documents in the results of a query.

Using Document Transformers

When executing a request, a document transformer can be used by including it in the fl parameter using square brackets, for example:

fl=id,name,score,[shard]

Some transformers allow, or require, local parameters which can be specified as key value pairs inside the brackets:

fl=id,name,score,[explain style=nl]

As with regular fields, you can change the key used when a Transformer adds a field to a document via a prefix:

fl=id,name,score,my_val_a:[value v=42 t=int],my_val_b:[value v=7 t=float]

The sections below discuss exactly what these various transformers do.

Available Transformers

[value] – ValueAugmenterFactory

Modifies every document to include the exact same value, as if it were a stored field in every document:

q=*:*&fl=id,greeting:[value v='hello']&wt=xml

The above query would produce results like the following:

<result name="response" numFound="32" start="0">
  <doc>
    <str name="id">1</str>
    <str name="greeting">hello</str></doc>
  </doc>
  ...

By default, values are returned as a String, but a t parameter can be specified using a value of int, float, double, or date to force a specific return type:

q=*:*&fl=id,my_number:[value v=42 t=int],my_string:[value v=42]

In addition to using these request parameters, you can configure additional named instances of ValueAugmenterFactory, or override the default behavior of the existing [value] transformer in your solrconfig.xml file:

<transformer name="mytrans2" class="org.apache.solr.response.transform.ValueAugmenterFactory" >
  <int name="value">5</int>
</transformer>
<transformer name="value" class="org.apache.solr.response.transform.ValueAugmenterFactory" >
  <double name="defaultValue">5</double>
</transformer>

The value option forces an explicit value to always be used, while the defaultValue option provides a default that can still be overridden using the v and t local parameters.

[explain] – ExplainAugmenterFactory

Augments each document with an inline explanation of its score exactly like the information available about each document in the debug section:

q=features:cache&fl=id,[explain style=nl]

Supported values for style are text, html, and nl which returns the information as structured data. Here is the output of the above request using style=nl:

{ "response":{"numFound":2,"start":0,"docs":[
      {
        "id":"6H500F0",
        "[explain]":{
          "match":true,
          "value":1.052226,
          "description":"weight(features:cache in 2) [DefaultSimilarity], result of:",
          "details":[{
}]}}]}}

A default style can be configured by specifying an args parameter in your solrconfig.xml configuration:

<transformer name="explain" class="org.apache.solr.response.transform.ExplainAugmenterFactory" >
  <str name="args">nl</str>
</transformer>

[child] – ChildDocTransformerFactory

This transformer returns all descendant documents of each parent document matching your query in a flat list nested inside the matching parent document. This is useful when you have indexed nested child documents and want to retrieve the child documents for the relevant parent documents for any type of search query.

fl=id,[child parentFilter=doc_type:book childFilter=doc_type:chapter limit=100]

Note that this transformer can be used even though the query itself is not a Block Join query.

When using this transformer, the parentFilter parameter must be specified unless the schema declares _nest_path_. It works the same as in all Block Join Queries. Additional optional parameters are:childFilterA query to filter which child documents should be included. This can be particularly useful when you have multiple levels of hierarchical documents. The default is all children. This query supports a special syntax to match nested doc patterns so long as _nest_path_ is defined in the schema and the query contains a / preceding the first :. Example: childFilter=/comments/content:recipe Further details of this are experimental.limitThe maximum number of child documents to be returned per parent document. The default is 10.fl

The field list which the transformer is to return. The default is the top level fl).

There is a further limitation in which the fields here should be a subset of those specified by the top level fl parameter.

[shard] – ShardAugmenterFactory

This transformer adds information about what shard each individual document came from in a distributed request.

ShardAugmenterFactory does not support any request parameters, or configuration options.

[docid] – DocIdAugmenterFactory

This transformer adds the internal Lucene document id to each document – this is primarily only useful for debugging purposes.

DocIdAugmenterFactory does not support any request parameters, or configuration options.

[elevated] and [excluded]

These transformers are available only when using the Query Elevation Component.

[elevated] annotates each document to indicate if it was elevated or not.
[excluded] annotates each document to indicate if it would have been excluded – this is only supported if you also use the markExcludes parameter.

fl=id,[elevated],[excluded]&excludeIds=GB18030TEST&elevateIds=6H500F0&markExcludes=true

{ "response":{"numFound":32,"start":0,"docs":[
      {
        "id":"6H500F0",
        "[elevated]":true,
        "[excluded]":false},
      {
        "id":"GB18030TEST",
        "[elevated]":false,
        "[excluded]":true},
      {
        "id":"SP2514N",
        "[elevated]":false,
        "[excluded]":false},
]}}

[json] / [xml]

These transformers replace a field value containing a string representation of a valid XML or JSON structure with the actual raw XML or JSON structure instead of just the string value. Each applies only to the specific writer, such that [json] only applies to wt=json and [xml] only applies to wt=xml.

fl=id,source_s:[json]&wt=json

[subquery]

This transformer executes a separate query per transforming document passing document fields as an input for subquery parameters. It’s usually used with {!join} and {!parent} query parsers, and is intended to be an improvement for [child].

It must be given an unique name: fl=*,children:[subquery]
There might be a few of them, e.g., fl=*,sons:[subquery],daughters:[subquery].
Every [subquery] occurrence adds a field into a result document with the given name, the value of this field is a document list, which is a result of executing subquery using document fields as an input.
Subquery will use the /select search handler by default, and will return an error if /select is not configured. This can be changed by supplying foo.qt parameter.

Here is how it looks like using various formats:XML

  <result name="response" numFound="2" start="0">
      <doc>
         <int name="id">1</int>
         <arr name="title">
            <str>vdczoypirs</str>
         </arr>
         <result name="children" numFound="1" start="0">
            <doc>
               <int name="id">2</int>
               <arr name="title">
                  <str>vdczoypirs</str>
               </arr>
            </doc>
         </result>
      </doc>
  ...

JSON

{ "response":{
    "numFound":2, "start":0,
    "docs":[
      {
        "id":1,
        "subject":["parentDocument"],
        "title":["xrxvomgu"],
        "children":{
           "numFound":1, "start":0,
           "docs":[
              { "id":2,
                "cat":["childDocument"]
              }
            ]
      }}]}}

SolrJ

 SolrDocumentList subResults = (SolrDocumentList)doc.getFieldValue("children");

Subquery Result Fields

To appear in subquery document list, a field should be specified in both fl parameters: in the main fl (despite the main result documents have no this field), and in subquery’s fl (e.g., foo.fl).

Wildcards can be used in one or both of these parameters. For example, if field title should appear in categories subquery, it can be done via one of these ways:

fl=...title,categories:[subquery]&categories.fl=title&categories.q=...
fl=...title,categories:[subquery]&categories.fl=*&categories.q=...
fl=...*,categories:[subquery]&categories.fl=title&categories.q=...
fl=...*,categories:[subquery]&categories.fl=*&categories.q=...

Subquery Parameters Shift

If a subquery is declared as fl=*,foo:[subquery], subquery parameters are prefixed with the given name and period. For example:

q=*:*&fl=*,**foo**:[subquery]&**foo.**q=to be continued&**foo.**rows=10&**foo.**sort=id desc

Document Field as an Input for Subquery Parameters

It’s necessary to pass some document field values as a parameter for subquery. It’s supported via an implicit row.fieldname parameter, and can be (but might not only) referred via local parameters syntax:

q=name:john&fl=name,id,depts:[subquery]&depts.q={!terms f=id v=$row.dept_id}&depts.rows=10

Here departments are retrieved per every employee in search result. We can say that it’s like SQL join ON emp.dept_id=dept.id.

Note, when a document field has multiple values they are concatenated with a comma by default. This can be changed with the local parameter foo:[subquery separator=' '], this mimics {!terms} to work smoothly with it.

To log substituted subquery request parameters, add the corresponding parameter names, as in: depts.logParamsList=q,fl,rows,row.dept_id

Cores and Collections in SolrCloud

Use foo:[subquery fromIndex=departments] to invoke subquery on another core on the same node. This is what {!join} does for non-SolrCloud mode. But with SolrCloud, just (and only) explicitly specify its native parameters like collection, shards for subquery, for example:

q=*:*&fl=\*,foo:[subquery]&foo.q=cloud&foo.collection=departments

[geo] – Geospatial formatter

Formats spatial data from a spatial field using a designated format type name. Two inner parameters are required: f for the field name, and w for the format name. Example: geojson:[geo f=mySpatialField w=GeoJSON].

Normally you’ll simply be consistent in choosing the format type you want by setting the format attribute on the spatial field type to WKT or GeoJSON – see the section Spatial Search for more information. If you are consistent, it’ll come out the way you stored it. This transformer offers a convenience to transform the spatial format to something different on retrieval.

In addition, this feature is very useful with the RptWithGeometrySpatialField to avoid double-storage of the potentially large vector geometry. This transformer will detect that field type and fetch the geometry from an internal compact binary representation on disk (in docValues), and then format it as desired. As such, you needn’t mark the field as stored, which would be redundant. In a sense this double-storage between docValues and stored-value storage isn’t unique to spatial but with polygonal geometry it can be a lot of data, and furthermore you’d like to avoid storing it in a verbose format (like GeoJSON or WKT). (maxnovahealthcare.com)

[features] – LTRFeatureLoggerTransformerFactory

The “LTR” prefix stands for Learning To Rank. This transformer returns the values of features and it can be used for feature extraction and feature logging.

fl=id,[features store=yourFeatureStore]

This will return the values of the features in the yourFeatureStore store.

fl=id,[features]&rq={!ltr model=yourModel}

If you use [features] together with an Learning-To-Rank reranking query then the values of the features in the reranking model (yourModel) will be returned.

This is it for today, stay tuned for more informative posts in future.