XPath tips from the web scraping trenches

In the context of web scraping, XPath is a nice tool to have in your belt, as it allows you to write specifications of document locations more flexibly than CSS selectors.

In case you're looking for a tutorial, here is a tutorial with XPath tips and examples.

In this post, we'll show you some tips we found valuable when using XPath in the trenches, using Scrapy Selector API for our examples.

Avoid using contains(.//text(), 'search text') in your XPath conditions.

Use contains(., 'search text') instead.

Here is why: the expression .//text() yields a collection of text elements -- a node-set. And when a node-set is converted to a string, which happens when it is passed as argument to a string function like contains() or starts-with(), results in the text for the first element only.

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> from scrapy import Selector

>>> sel = Selector(text='Click here to go to the Next Page')

>>> xp = lambda x: sel.xpath(x).extract() # let's type this only once

>>> xp('//a//text()') # take a peek at the node-set

[u'Click here to go to the ', u'Next Page']

>>> xp('string(//a//text())') # convert it to a string

[u'Click here to go to the ']

>>> from scrapy import Selector >>> sel = Selector(text='Click here to go to the Next Page') >>> xp = lambda x: sel.xpath(x).extract() # let's type this only once >>> xp('//a//text()') # take a peek at the node-set [u'Click here to go to the ', u'Next Page'] >>> xp('string(//a//text())') # convert it to a string [u'Click here to go to the ']

1\>>> from scrapy import Selector
2>>> sel = Selector(text='<a href="#">Click here to go to the <strong>Next Page</strong></a>')
3>>> xp = lambda x: sel.xpath(x).extract() # let's type this only once
4>>> xp('//a//text()') # take a peek at the node-set
5   \[u'Click here to go to the ', u'Next Page'\]
6>>> xp('string(//a//text())')  # convert it to a string
7   \[u'Click here to go to the '\]

A node converted to a string, however, puts together the text of itself plus of all its descendants:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp('//a[1]') # selects the first a node

[u'Click here to go to the Next Page']

>>> xp('string(//a[1])') # converts it to string

[u'Click here to go to the Next Page']

>>> xp('//a[1]') # selects the first a node [u'Click here to go to the Next Page'] >>> xp('string(//a[1])') # converts it to string [u'Click here to go to the Next Page']

1>>> xp('//a\[1\]') # selects the first a node
2\[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>'\]
3>>> xp('string(//a\[1\])') # converts it to string
4\[u'Click here to go to the Next Page'\]

So, in general:

GOOD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("//a[contains(., 'Next Page')]")

[u'Click here to go to the Next Page']

>>> xp("//a[contains(., 'Next Page')]") [u'Click here to go to the Next Page']

1\>>> xp("//a\[contains(., 'Next Page')\]")
2\[u'<a href="#">Click here to go to the <strong>Next Page</strong></a>'\]

BAD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("//a[contains(.//text(), 'Next Page')]")

[]

>>> xp("//a[contains(.//text(), 'Next Page')]") []

1\>>> xp("//a\[contains(.//text(), 'Next Page')\]")
2\[\]

GOOD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("substring-after(//a, 'Next ')")

[u'Page']

>>> xp("substring-after(//a, 'Next ')") [u'Page']

1\>>> xp("substring-after(//a, 'Next ')")
2\[u'Page'\]

BAD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("substring-after(//a//text(), 'Next ')")

[u'']

>>> xp("substring-after(//a//text(), 'Next ')") [u'']

1\>>> xp("substring-after(//a//text(), 'Next ')")
2\[u''\]

You can read more detailed explanations about string values of nodes and node-sets in the XPath spec.

Beware of the difference between //node[1] and (//node)[1]

//node[1] selects all the nodes occurring first under their respective parents.

(//node)[1] selects all the nodes in the document, and then gets only the first of them.

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> from scrapy import Selector

>>> sel=Selector(text="""

....:

....:

• 1

....:

• 2

....:

• 3

....:

....:

....:

• 4

....:

• 5

....:

• 6

....:

""")

>>> xp = lambda x: sel.xpath(x).extract()

>>> xp("//li[1]") # get all first LI elements under whatever it is its parent

[u'

1

', u'

4

']

>>> xp("(//li)[1]") # get the first LI element in the whole document

[u'

1

']

>>> xp("//ul/li[1]") # get all first LI elements under an UL parent

[u'

1

', u'

4

']

>>> xp("(//ul/li)[1]") # get the first LI element under an UL parent in the document

[u'

1

']

>>> from scrapy import Selector >>> sel=Selector(text=""" ....:

....:
• 1
....:
• 2
....:
• 3
....:

....:

....:
• 4
....:
• 5
....:
• 6
....:

""") >>> xp = lambda x: sel.xpath(x).extract() >>> xp("//li[1]") # get all first LI elements under whatever it is its parent [u'

1

', u'

4

'] >>> xp("(//li)[1]") # get the first LI element in the whole document [u'

1

'] >>> xp("//ul/li[1]") # get all first LI elements under an UL parent [u'

1

', u'

4

'] >>> xp("(//ul/li)[1]") # get the first LI element under an UL parent in the document [u'

1

']

1\>>> from scrapy import Selector
2>>> sel=Selector(text="""
3....:     <ul class="list">
4....:         <li>1</li>
5....:         <li>2</li>
6....:         <li>3</li>
7....:     </ul>
8....:     <ul class="list">
9....:         <li>4</li>
10....:         <li>5</li>
11....:         <li>6</li>
12....:     </ul>""")
13>>> xp = lambda x: sel.xpath(x).extract()
14>>> xp("//li\[1\]") # get all first LI elements under whatever it is its parent
15\[u'<li>1</li>', u'<li>4</li>'\]
16>>> xp("(//li)\[1\]") # get the first LI element in the whole document
17\[u'<li>1</li>'\]
18>>> xp("//ul/li\[1\]")  # get all first LI elements under an UL parent
19\[u'<li>1</li>', u'<li>4</li>'\]
20>>> xp("(//ul/li)\[1\]") # get the first LI element under an UL parent in the document
21\[u'<li>1</li>'\]

Also,

//a[starts-with(@href, '#')][1] gets a collection of the local anchors that occur first under their respective parents.

(//a[starts-with(@href, '#')])[1] gets the first local anchor in the document.

When selecting by class, be as specific as necessary

If you want to select elements by a CSS class, the XPath way to do that is the rather verbose:

*[contains(concat(' ', normalize-space(@class), ' '), ' someclass ')]

Let's cook up some examples:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> sel = Selector(text='

Someone

Some content

')

>>> xp = lambda x: sel.xpath(x).extract()

>>> sel = Selector(text='

Someone

Some content

') >>> xp = lambda x: sel.xpath(x).extract()

1\>>> sel = Selector(text='<p class="content-author">Someone</p><p class="content text-wrap">Some content</p>')
2>>> xp = lambda x: sel.xpath(x).extract()

BAD: doesn't work because there are multiple classes in the attribute

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("//*[@class='content']")

[]

>>> xp("//*[@class='content']") []

1\>>> xp("//\*\[@class='content'\]")
2\[\]

BAD: gets more than we want

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("//*[contains(@class,'content')]")

[u'

Someone

']

>>> xp("//*[contains(@class,'content')]") [u'

Someone

']

1\>>> xp("//\*\[contains(@class,'content')\]")
2\[u'<p class="content-author">Someone</p>'\]

GOOD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> xp("//*[contains(concat(' ', normalize-space(@class), ' '), ' content ')]")

[u'

Some content

']

>>> xp("//*[contains(concat(' ', normalize-space(@class), ' '), ' content ')]") [u'

Some content

']

1\>>> xp("//\*\[contains(concat(' ', normalize-space(@class), ' '), ' content ')\]")
2\[u'<p class="content text-wrap">Some content</p>'\]

And many times, you can just use a CSS selector instead, and even combine the two of them if needed:

ALSO GOOD:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

>>> sel.css(".content").extract()

[u'

Some content

']

>>> sel.css('.content').xpath('@class').extract()

[u'content text-wrap']

>>> sel.css(".content").extract() [u'

Some content

'] >>> sel.css('.content').xpath('@class').extract() [u'content text-wrap']

1\>>> sel.css(".content").extract()
2\[u'<p class="content text-wrap">Some content</p>'\]
3>>> sel.css('.content').xpath('@class').extract()
4\[u'content text-wrap'\]

Read more about what you can do with Scrapy's Selectors here.

XPath tips: Learn to use all the different axes

It is handy to know how to use the axes, you can follow through the examples given in the tutorial to quickly review this.

In particular, you should note that following and following-sibling are not the same thing, this is a common source of confusion.

The same goes for preceding and preceding-sibling, and also ancestor and parent.

Useful trick to get text content

Want even more XPath tips? Here is another trick that you may use to get the interesting text contents:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

//*[not(self::script or self::style)]/text()[normalize-space(.)]

//*[not(self::script or self::style)]/text()[normalize-space(.)]

1//\*\[not(self::script or self::style)\]/text()\[normalize-space(.)\]

This excludes the content from script and style tags and also skip whitespace-only text nodes.

Source: http://stackoverflow.com/a/19350897/2572383

Do you have other XPath tips?

Please, leave us a comment with your tips or questions. 🙂

And for everybody who contributed tips and reviewed this article, a big thank you!