{"id":12204,"date":"2025-07-24T10:11:40","date_gmt":"2025-07-24T14:11:40","guid":{"rendered":"https:\/\/thirdwunder.com\/glossary\/crawler\/"},"modified":"2025-07-24T10:11:40","modified_gmt":"2025-07-24T14:11:40","slug":"crawler","status":"publish","type":"tw_glossary","link":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/","title":{"rendered":"Crawler"},"content":{"rendered":"<p>A crawler is an automated software program that systematically browses websites to collect and index information. Crawlers are used by search engines such as Google and Bing to discover web pages and evaluate their content for inclusion in search results. Sometimes called spiders or bots, crawlers help search engines understand what each page is about and how it connects to other content on the web.<\/p>\n<p>When a crawler visits a website, it starts with a known page and follows hyperlinks to discover new content. It scans the HTML code, extracts metadata, reads headings and text, and logs the relationships between pages. This data is stored in a search engine\u2019s index and used to determine how and where pages should appear in search rankings. Site owners can use tools like robots.txt and meta directives to control what a crawler is allowed to access and index.<\/p>\n<p>Optimising a website for crawler accessibility is a key part of technical SEO. A well-structured site with logical navigation, internal linking, fast load times and a clear sitemap makes it easier for crawlers to index content accurately. Pages that cannot be crawled may not appear in search results, which affects visibility and performance. Errors such as broken links, infinite loops or blocked resources can prevent successful crawling. Monitoring crawler activity through tools like Google Search Console or Screaming Frog helps teams identify problems early and maintain strong search engine visibility.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.<\/p>\n","protected":false},"featured_media":0,"parent":0,"menu_order":0,"template":"","glossary-letter":[1285],"glossary-category":[1274],"glossary-topic":[1254],"glossary-tag":[],"class_list":["post-12204","tw_glossary","type-tw_glossary","status-publish","hentry","tw_glossary_letter-c","tw_glossary_category-marketing","tw_glossary_topic-seo"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Crawler - Third Wunder<\/title>\n<meta name=\"description\" content=\"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Crawler - Third Wunder\" \/>\n<meta property=\"og:description\" content=\"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/\" \/>\n<meta property=\"og:site_name\" content=\"Third Wunder\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/crawler\\\/\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/crawler\\\/\",\"name\":\"Crawler - Third Wunder\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/#website\"},\"datePublished\":\"2025-07-24T14:11:40+00:00\",\"description\":\"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/crawler\\\/#breadcrumb\"},\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/crawler\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/crawler\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Terms\",\"item\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Crawler\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/#website\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\",\"name\":\"Third Wunder\",\"description\":\"Digital Marketing for Scaling Businesses\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-FR\",\"publisher\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\\\/#organization\"}},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\\\/#organization\",\"name\":\"Third Wunder\",\"description\":\"Digital Marketing for Scaling Businesses\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/wp-content\\\/themes\\\/wundertheme2025\\\/assets\\\/img\\\/logo.png\"},\"location\":[{\"@type\":\"ProfessionalService\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/#location_0\",\"name\":\"Third Wunder\",\"parentOrganization\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/#organization\"},\"address\":{\"@type\":\"PostalAddress\",\"streetAddress\":\"4471 rue de Lanaudiere\",\"addressLocality\":\"Montreal\",\"addressRegion\":\"QC\",\"postalCode\":\"H2J3P3\",\"addressCountry\":\"CA\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/wp-content\\\/themes\\\/wundertheme2024\\\/assets\\\/img\\\/logo.png\"},\"openingHoursSpecification\":[{\"@type\":\"OpeningHoursSpecification\",\"dayOfWeek\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\"],\"opens\":\"09:00\",\"closes\":\"17:00\"},{\"@type\":\"OpeningHoursSpecification\",\"dayOfWeek\":[\"Friday\"],\"opens\":\"10:00\",\"closes\":\"16:00\"}]}],\"contactPoint\":{\"@type\":\"ContactPoint\",\"contactType\":\"Sales\",\"email\":\"info@thirdwunder.com\",\"telephone\":\"+15144371652\"},\"sameAs\":[\"https:\\\/\\\/facebook.com\\\/ThirdWunder\",\"https:\\\/\\\/instagram.com\\\/thirdwunder\",\"https:\\\/\\\/twitter.com\\\/thirdwunder\",\"https:\\\/\\\/linkedin.com\\\/company\\\/ThirdWunder\",\"https:\\\/\\\/www.youtube.com\\\/@thirdwunder\",\"https:\\\/\\\/github.com\\\/thirdwunder\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Crawler - Third Wunder","description":"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/","og_locale":"fr_FR","og_type":"article","og_title":"Crawler - Third Wunder","og_description":"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.","og_url":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/","og_site_name":"Third Wunder","twitter_card":"summary_large_image","twitter_misc":{"Dur\u00e9e de lecture estim\u00e9e":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/","url":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/","name":"Crawler - Third Wunder","isPartOf":{"@id":"https:\/\/thirdwunder.com\/fr\/#website"},"datePublished":"2025-07-24T14:11:40+00:00","description":"A crawler systematically browses the web by following hyperlinks to collect information. Search engines use this data to build indexes and determine page rankings in search results.","breadcrumb":{"@id":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/#breadcrumb"},"inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/thirdwunder.com\/fr\/glossary\/crawler\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/thirdwunder.com\/fr\/"},{"@type":"ListItem","position":2,"name":"Terms","item":"https:\/\/thirdwunder.com\/fr\/glossary\/"},{"@type":"ListItem","position":3,"name":"Crawler"}]},{"@type":"WebSite","@id":"https:\/\/thirdwunder.com\/fr\/#website","url":"https:\/\/thirdwunder.com\/fr\/","name":"Third Wunder","description":"Digital Marketing for Scaling Businesses","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/thirdwunder.com\/fr\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-FR","publisher":{"@id":"https:\/\/thirdwunder.com\/fr\/\/#organization"}},{"@type":"Organization","@id":"https:\/\/thirdwunder.com\/fr\/\/#organization","name":"Third Wunder","description":"Digital Marketing for Scaling Businesses","url":"https:\/\/thirdwunder.com\/fr\/","logo":{"@type":"ImageObject","url":"https:\/\/thirdwunder.com\/wp-content\/themes\/wundertheme2025\/assets\/img\/logo.png"},"location":[{"@type":"ProfessionalService","@id":"https:\/\/thirdwunder.com\/#location_0","name":"Third Wunder","parentOrganization":{"@id":"https:\/\/thirdwunder.com\/#organization"},"address":{"@type":"PostalAddress","streetAddress":"4471 rue de Lanaudiere","addressLocality":"Montreal","addressRegion":"QC","postalCode":"H2J3P3","addressCountry":"CA"},"image":{"@type":"ImageObject","url":"https:\/\/thirdwunder.com\/wp-content\/themes\/wundertheme2024\/assets\/img\/logo.png"},"openingHoursSpecification":[{"@type":"OpeningHoursSpecification","dayOfWeek":["Monday","Tuesday","Wednesday","Thursday"],"opens":"09:00","closes":"17:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":["Friday"],"opens":"10:00","closes":"16:00"}]}],"contactPoint":{"@type":"ContactPoint","contactType":"Sales","email":"info@thirdwunder.com","telephone":"+15144371652"},"sameAs":["https:\/\/facebook.com\/ThirdWunder","https:\/\/instagram.com\/thirdwunder","https:\/\/twitter.com\/thirdwunder","https:\/\/linkedin.com\/company\/ThirdWunder","https:\/\/www.youtube.com\/@thirdwunder","https:\/\/github.com\/thirdwunder"]}]}},"_links":{"self":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary\/12204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/types\/tw_glossary"}],"wp:attachment":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/media?parent=12204"}],"wp:term":[{"taxonomy":"tw_glossary_letter","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-letter?post=12204"},{"taxonomy":"tw_glossary_category","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-category?post=12204"},{"taxonomy":"tw_glossary_topic","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-topic?post=12204"},{"taxonomy":"tw_glossary_tag","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-tag?post=12204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}