Term, nested documents and must_not query incompatible in ElasticSearch?

I have trouble combining term, must_not queries on nested documents.

Sense example can be found here : http://sense.qbox.io/gist/be436a1ffa01e4630a964f48b2d5b3a1ef5fa176

Here my mapping :

{
"mappings": {
"docs" : {
"properties": {
"tags" : {
"type": "nested",
"properties" : {
"type": {
"type": "string",
"index": "not_analyzed"
}
}
},
"label" : {
"type": "string"
}
}
}
}
}


with two documents in this index :

{
"tags" : [
{"type" : "POST"},
{"type" : "DELETE"}
],
"label" : "item 1"
},
{
"tags" : [
{"type" : "POST"}
],
"label" : "item 2"
}


When I query this index like this :

{
"query": {
"nested": {
"path": "tags",
"query": {
"bool": {
"must": {
"term": {
"tags.type": "DELETE"
}
}
}
}
}
}
}


I've got one hit (which is correct)

When I want to get documents WHICH DON'T CONTAIN the tag "DELETE", with this query :

{
"query": {
"nested": {
"path": "tags",
"query": {
"bool": {
"must_not": {
"term": {
"tags.type": "delete"
}
}
}
}
}
}
}


I've got 2 hits (which is incorrect).
This issue seems very close to this one (Elasticsearch array must and must_not) but it's not...

Can you give me some clues to resolve this issue ?

Thank you

2 Answers
2

This should fix your problem: http://sense.qbox.io/gist/f4694f542bc76c29624b5b5c9b3ecdee36f7e3ea

Two most important things:

include_in_root on "tags.type". This will tell ES to index tag types as "doc.tags.types" : ['DELETE', 'POSTS'], so you can access an array of those values "flattened" on the root doc . This means you no longer need a nested query (see #2)

"doc.tags.types" : ['DELETE', 'POSTS']

Drop the nested query.

 

{
"mappings": {
"docs" : {
"properties": {
"tags" : {
"type": "nested",
"properties" : {
"type": {
"type": "string",
"index": "not_analyzed"
}
},
"include_in_root": true
},
"label" : {
"type": "string"
}
}
}
}
}


 

{
"query": {
"bool": {
"must_not": {
"term": {
"tags.type": "DELETE"
}
}
}
}
}


Thanks, it works. Can you explain why you did that way ?
– user3393203
Mar 10 '14 at 11:52

Simply because it was not possible without "include_in_root". Instead of having a complex nested filter/query system, you can now just treat all "tags.type" as an array. That way, you can say something like "give me all the docs without 'DELETE' in the tag types array". It's not easy to verbalize this stuff sometimes, but hopefully that makes sense!
– Ben at Qbox.io
Mar 10 '14 at 16:48

It doesn't work for me. That's both weird and sad. Does it maybe because I use terms and integer?
– Aminah Nuraini
Mar 23 '16 at 4:35

terms

integer

Your original query would search in each individual nested object and eliminate the objects that don't match, but if there are some nested objects left, they do match with your query and so you get your results. This is because nested objects are indexed as a hidden separate document

Original code:

{
"query": {
"nested": {
"path": "tags",
"query": {
"bool": {
"must_not": {
"term": {
"tags.type": "delete"
}
}
}
}
}
}
}


The solution is then quite simple really, you should bring the bool query outside the nested documents. Now all the documents are discarded who have a nested object with the "DELETE" type. Just what you wanted!

The solution:

{
"query": {
"bool": {
"must_not": {
"nested": {
"path": "tags",
"query": {
"term": {
"tags.type": "DELETE"
}
}
}
}
}
}
}


NOTE: Your strings are "not analyzed" and you searched for "delete" instead of "DELETE". If you want to search case insensitive, make your strings analyzed

If I do it this way, data which has no tags field is not included to the result
– Aminah Nuraini
Mar 23 '16 at 3:23

Are you sure about that? Which Elasticsearch version do you use?
– rvheddeg
Mar 23 '16 at 15:49

ElasticSearch 2.2.1. After rechecking, a lot of query works fine in my laptop, but in the server. The Elasticsearch version may play a role.
– Aminah Nuraini
Mar 24 '16 at 3:13

I think you mean not in the server. Which ES version do you use there? And did ALL the queries work on the laptop? If they did, I think my solution still stands. In fact, I use this solution everyday in production...
– rvheddeg
Mar 24 '16 at 8:25

It doesn't work on the server, and the server uses version 2.2.1 like I told in the previous comment. I haven't tried it on my own laptop
– Aminah Nuraini
Mar 25 '16 at 3:14

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.