Query builder performance

Question

Hi
I'm running into very big performance problems with QueryBuilder.
I'm using asset viewer to browse my CQ DAM images. We have loaded more than 100.000 images... all of them are tagged.
Launching an http request like the following one:
http://localhost:port/bin/querybuilder.json
?_charset_=utf-8
&_dc=1332289410422
&fulltext=
&1_group.property.3_value=tag1
&1_group.property.29_value=tag2
&1_group.property.30_value=tag3
&1_group.property.31_value=tag4
&1_group.property.32_value=tag5
&1_group.property.33_value=tag6
&1_group.property.34_value=tag7
&0_group.property=jcr%3Acontent%2Fmetadata%2Fcq%3Atags
&0_group.p.or=true
&1_group.property=jcr%3Acontent%2Fmetadata%2Fcq%3Atags
&1_group.p.or=true
&p.limit=24
&2_group.0_path=%2Fcontent%2Fdam%2Fmyfolder%2Fimage%2Fprodotto
&3_group.0_type=dam%3AAsset
&mainasset=true
&p.hitwriter=full
&p.nodedepth=4
&orderby=path
&p.offset=0
it is going to take a *lot* of time (22 minutes!). This query has 36.000 results... but they are paginated... It's not possible to wait 22 minutes to get the first page of results... so I think I'm making some mistakes in the query, but cannot find anything to make it better following the documentation.
How can I improve performances?
Thanks
Ignazio

Answer 1

Hi Alex.
Yes, the order by may be consuming and I already removed it in the past. I have removed it again, but it is not the dominant.
It's difficult for me to explain to a client that with asset viewer (using querybuilder) he has to wait for a very long time to get the first 24 results of a set of 20.000 matches in a 120.000 image repository.
I can understand I need to wait a lot of time to get all the 20.000 matches, but I'm asking only for the first 24 (the first page) setting p.limit = 24.
Here the debug for the query I'm referring to:
21.03.2012 13:18:41.613 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl executing query (URL):
0_group.p.or=true&0_group.property=jcr%3acontent%2fmetadata%2fcq%3atags&0_group.property.0 _value=immaginegenerale&1_group.p.or=true&1_group.property=jcr%3acontent%2fmetadata%2fcq%3 atags&2_group.0_path=%2fcontent%2fdam%2fimage%2fprodotto&3_group.0_type=dam%3aAsset&fullte xt=&fulltext.relPath=jcr%3acontent%2fmetadata%2f%40dc%3amarketingdescription&mainasset=tru e&p.hitwriter=full&p.limit=24&p.nodedepth=4&p.offset=0
21.03.2012 13:18:41.613 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl executing query (predicate tree):
ROOT=group: limit=24, hitwriter=full, nodedepth=4, offset=0[
    {fulltext=fulltext: relPath=jcr:content/metadata/@dc:marketingdescription, fulltext=}
    {mainasset=mainasset: mainasset=true}
    {0_group=group: or=true[
        {property=property: property=jcr:content/metadata/cq:tags, 0_value:immaginegenerale}
    {1_group=group: or=true[
        {property=property: property=jcr:content/metadata/cq:tags}
    {2_group=group: [
        {0_path=path: path=/content/dam/image/prodotto}
    {3_group=group: [
        {0_type=type: type=dam:Asset}
21.03.2012 13:18:41.615 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl xpath query: /jcr:root/content/dam/image/prodotto//element(*, dam:Asset)[((jcr:content/metadata/@cq:tags = 'immaginegenerale'))]
21.03.2012 13:18:44.078 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl xpath query took 2465 ms
21.03.2012 13:18:44.078 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl filtering predicates: {mainasset=mainasset: mainasset=true}, , , ,
21.03.2012 13:22:16.394 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl >> xpath query returned 20117 results (counted)
21.03.2012 13:22:16.395 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl filtering took 212317 ms
21.03.2012 13:22:16.395 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl >> after filtering there are 20117 results
21.03.2012 13:22:16.397 *INFO* [172.31.2.2 [1332332321612] GET /bin/querybuilder.json HTTP/1.1] com.day.cq.search.impl.builder.QueryImpl entire query execution took 214784 ms
As you can see all the time is spent on the filtering predicates.
Is there a way to cut down significantly this time? or is it possible to distribute this effort only during pagination?
in a db-like approach I would have done a sql  SELECT COUNT followed by a SELECT LIMIT 0,24... for sure taking a portion of this time.
Thanks
Ignazio