Web scraper - IDE

  • Coding environment - IDE Page

    A : See more examples - Examples of template code that our collector engineers built. B : Add another step (stage) - It is useful to add stages when you want to collect data from multiple pages. For example, in case you want to collect all products fro...

  • Coding environment - IDE Interaction code

    These are all of the codes that you can do with the IDE input - Global object available to the interaction code. Provided by trigger input or next_stage() calls navigate(input.url); navigate - Navigate the browser session to a URLurl: A URL to navigat...

  • Coding environment - IDE Parser code

    These are all of the codes that you can do with the IDE input - Global variable available to the parser code let url = input.url; $ - An instance of cheerioFind more information on the cheerio website. $('#example').text() $('$example').attr('href') ...

  • Finding element selectors

    In order to target an element (to click it or pull text out), you need to specify the element with a CSS selector. A CSS selector can match one or more items on the page. Most commands in the interaction code will require a selector to target the correc...

  • Building element selectors

    Selectors are built out of 4 basic components: p : the element type selector. This example will match any <p></p> element on the page [href] : square brackets is an attribute selector. This example will match any element with the href attribute set (e....

  • jQuery expressions

    You can use jQuery-like expressions in interaction code. For instance: wait($('selector')); // wait for this element to appear click($('selector')); // click on this element after it appears // wait for an element that matches the selector and contains ...