Mgiyuk

Scrapy can't find form on page I'm trying to write a spider that will automatically log in to this website. However, when I try using scrapy.FormRequest.from_response in the shell I get the error: scrapy.FormRequest.from_response No <form> element found in <200 https://www.athletic.net/account/login/?ReturnUrl=%2Fdefault.aspx> No <form> element found in <200 https://www.athletic.net/account/login/?ReturnUrl=%2Fdefault.aspx> I can definitely see the form when I inspect element on the site, but it just did not show up in Scrapy when I tried finding it using response.xpath() either. Is it possible for the form content to be hidden from my spider somehow? If so, how do I fix it? response.xpath() 1 Answer 1 The form is created using Javascript, it's not part of the static HTML source code. Scrapy does not parse Javascript, thus it cannot be found. The relevant part o...

remove characters in xpath / scrapy I use scrapy to extract data and it generates the field (typeFacture) with (''), I want to extract the text and delete ('') to insert it into a database, I want to do that to help from XPATH HTML code: <td class="tNorm tSmall-xs"> <b>FACTURE</b> <br> '' Commission '' </td> my code: item['typeFacture'] = [item.strip() for item in sel.xpath('//tbody/tr/td[5]/text()').extract()] result: 'typeFacture': ['', '', 'Commission', '', '', 'Commission', '', '', 'Commission', '', '', 'Commission', '', '', 'Abonnement']} Don...

XPath to get all including text except div I know how to exclude, but I don't know how to exclude a node but get everything else including the text. So I use this XPath: //div[@class="web_description"]/*[not(div)] on the HTML below. I want also the text outside <b> tag. How can I do that? //div[@class="web_description"]/*[not(div)] <b> <h5>Description</h5> The target is red-hot, currently scaling and <b>BRANDED AMAZON 5-STAR RETAILER IN THE</b> <b>PET AND ACCESSORIES </b>segment. This brand is realizing <b>over 200%YOY growth for 2018</b>, and boasts multiple Amazon’s Choice SKUs in the pets category – all of this bound up in a massively attractive and low-overhead FBA platform with broad appeal to all the major verticals in the pet space means this is one of <b>the most viable pet retail acquisition opportunities of 2018.</b><br> <br> With a va...

how to find combined xpath for given text as < PMP & 8999 POINTS >? I want to find the combined xpath for the text <PMP & 8999 points> as follows: <PMP & 8999 points> PMP (Duration: 29 hrs | Difficulty: Intermediate | Language: english | Mode Of Training: online)  8999 Points </div> </div> 1 Answer 1 You can try with following x-path, //div[@class='lng-info']/*[name()='h5' or name()='div'] no,its pointing only to text= PMP – hack Jun 29 at 8:44 click on next button. – Murthi ...

How can I find the randomly Xpath using Selenium? I am a newer in Selenium and use python to build it. Recently, I found a question which want to ask someone who can help me to figure it. The question is the Xpath I want to get is randomly, for example: '//*[@id="wiki-edit-wikiEdit26"]/div/div/div/div[2]/a[1]' '//*[@id="wiki-edit-wikiEdit27"]/div/div/div/div[2]/a[1]' '//*[@id="wiki-edit-wikiEdit28"]/div/div/div/div[2]/a[1]' These three xpath are used on the same button, but the number after wikiEdit will be changed every time. Therefore, are there any way which can help me to run my script more smoothly? Thank you very much! xpath Here is my python code: broswer.find.element_by_xpath('//*[@id="wiki-edit-wikiEdit26"]/div/div/div/div[2]/a[1]') .click() do you want to change no. in wikiEdit ? – Raghav Patnecha 33 mins ago ...