{"id":12343,"date":"2025-07-24T10:14:33","date_gmt":"2025-07-24T14:14:33","guid":{"rendered":"https:\/\/thirdwunder.com\/glossary\/robots-txt\/"},"modified":"2025-07-24T10:14:33","modified_gmt":"2025-07-24T14:14:33","slug":"robots-txt","status":"publish","type":"tw_glossary","link":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/","title":{"rendered":"Robots.txt"},"content":{"rendered":"

Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which pages or directories they are allowed or not allowed to access. It serves as a set of instructions for web robots, helping website owners control how their content is crawled and indexed. While not mandatory for all crawlers to follow, most major search engines, including Google and Bing, respect these directives.<\/p>\n

The file can be used to block access to sensitive or duplicate content, conserve crawl budget or guide bots to specific sitemaps. For example, a robots.txt file might prevent crawlers from indexing admin pages, internal search results or development sections of a site. A typical directive might look like this:<\/p>\n

User-agent: *
\nDisallow: \/private-folder\/
\nThis tells all bots not to crawl the specified folder. Additional rules can be added to allow or disallow specific bots or directories, and to link to an XML sitemap for easier discovery of content.<\/p>\n

For B2B and SaaS companies, robots.txt helps manage indexing of support pages, customer portals or gated content. For nonprofits, it ensures that confidential pages or staff-only resources are not exposed in search results. A misconfigured file, however, can unintentionally block important pages, leading to SEO issues. It is important to regularly audit robots.txt settings and test them using tools like Google Search Console to ensure they align with your site\u2019s indexing goals.<\/p>\n","protected":false},"excerpt":{"rendered":"

Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.<\/p>\n","protected":false},"featured_media":0,"parent":0,"menu_order":0,"template":"","glossary-letter":[1303],"glossary-category":[1274],"glossary-topic":[1254],"glossary-tag":[],"class_list":["post-12343","tw_glossary","type-tw_glossary","status-publish","hentry","tw_glossary_letter-r","tw_glossary_category-marketing","tw_glossary_topic-seo"],"acf":[],"yoast_head":"\nRobots.txt - Third Wunder<\/title>\n<meta name=\"description\" content=\"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/\" \/>\n<meta property=\"og:locale\" content=\"fr_FR\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Robots.txt - Third Wunder\" \/>\n<meta property=\"og:description\" content=\"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/\" \/>\n<meta property=\"og:site_name\" content=\"Third Wunder\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Dur\u00e9e de lecture estim\u00e9e\" \/>\n\t<meta name=\"twitter:data1\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/robots-txt\\\/\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/robots-txt\\\/\",\"name\":\"Robots.txt - Third Wunder\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/#website\"},\"datePublished\":\"2025-07-24T14:14:33+00:00\",\"description\":\"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/robots-txt\\\/#breadcrumb\"},\"inLanguage\":\"fr-FR\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/robots-txt\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/robots-txt\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Terms\",\"item\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/glossary\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Robots.txt\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/#website\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\",\"name\":\"Third Wunder\",\"description\":\"Digital Marketing for Scaling Businesses\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"fr-FR\",\"publisher\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\\\/#organization\"}},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\\\/#organization\",\"name\":\"Third Wunder\",\"description\":\"Digital Marketing for Scaling Businesses\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/fr\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/wp-content\\\/themes\\\/wundertheme2025\\\/assets\\\/img\\\/logo.png\"},\"location\":[{\"@type\":\"ProfessionalService\",\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/#location_0\",\"name\":\"Third Wunder\",\"parentOrganization\":{\"@id\":\"https:\\\/\\\/thirdwunder.com\\\/#organization\"},\"address\":{\"@type\":\"PostalAddress\",\"streetAddress\":\"4471 rue de Lanaudiere\",\"addressLocality\":\"Montreal\",\"addressRegion\":\"QC\",\"postalCode\":\"H2J3P3\",\"addressCountry\":\"CA\"},\"image\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/thirdwunder.com\\\/wp-content\\\/themes\\\/wundertheme2024\\\/assets\\\/img\\\/logo.png\"},\"openingHoursSpecification\":[{\"@type\":\"OpeningHoursSpecification\",\"dayOfWeek\":[\"Monday\",\"Tuesday\",\"Wednesday\",\"Thursday\"],\"opens\":\"09:00\",\"closes\":\"17:00\"},{\"@type\":\"OpeningHoursSpecification\",\"dayOfWeek\":[\"Friday\"],\"opens\":\"10:00\",\"closes\":\"16:00\"}]}],\"contactPoint\":{\"@type\":\"ContactPoint\",\"contactType\":\"Sales\",\"email\":\"info@thirdwunder.com\",\"telephone\":\"+15144371652\"},\"sameAs\":[\"https:\\\/\\\/facebook.com\\\/ThirdWunder\",\"https:\\\/\\\/instagram.com\\\/thirdwunder\",\"https:\\\/\\\/twitter.com\\\/thirdwunder\",\"https:\\\/\\\/linkedin.com\\\/company\\\/ThirdWunder\",\"https:\\\/\\\/www.youtube.com\\\/@thirdwunder\",\"https:\\\/\\\/github.com\\\/thirdwunder\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Robots.txt - Third Wunder","description":"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/","og_locale":"fr_FR","og_type":"article","og_title":"Robots.txt - Third Wunder","og_description":"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.","og_url":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/","og_site_name":"Third Wunder","twitter_card":"summary_large_image","twitter_misc":{"Dur\u00e9e de lecture estim\u00e9e":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/","url":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/","name":"Robots.txt - Third Wunder","isPartOf":{"@id":"https:\/\/thirdwunder.com\/fr\/#website"},"datePublished":"2025-07-24T14:14:33+00:00","description":"Robots.txt is a file instructing search engine crawlers on allowed website access. It helps manage indexing of pages or sections using directives for different bots.","breadcrumb":{"@id":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/#breadcrumb"},"inLanguage":"fr-FR","potentialAction":[{"@type":"ReadAction","target":["https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/thirdwunder.com\/fr\/glossary\/robots-txt\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/thirdwunder.com\/fr\/"},{"@type":"ListItem","position":2,"name":"Terms","item":"https:\/\/thirdwunder.com\/fr\/glossary\/"},{"@type":"ListItem","position":3,"name":"Robots.txt"}]},{"@type":"WebSite","@id":"https:\/\/thirdwunder.com\/fr\/#website","url":"https:\/\/thirdwunder.com\/fr\/","name":"Third Wunder","description":"Digital Marketing for Scaling Businesses","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/thirdwunder.com\/fr\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"fr-FR","publisher":{"@id":"https:\/\/thirdwunder.com\/fr\/\/#organization"}},{"@type":"Organization","@id":"https:\/\/thirdwunder.com\/fr\/\/#organization","name":"Third Wunder","description":"Digital Marketing for Scaling Businesses","url":"https:\/\/thirdwunder.com\/fr\/","logo":{"@type":"ImageObject","url":"https:\/\/thirdwunder.com\/wp-content\/themes\/wundertheme2025\/assets\/img\/logo.png"},"location":[{"@type":"ProfessionalService","@id":"https:\/\/thirdwunder.com\/#location_0","name":"Third Wunder","parentOrganization":{"@id":"https:\/\/thirdwunder.com\/#organization"},"address":{"@type":"PostalAddress","streetAddress":"4471 rue de Lanaudiere","addressLocality":"Montreal","addressRegion":"QC","postalCode":"H2J3P3","addressCountry":"CA"},"image":{"@type":"ImageObject","url":"https:\/\/thirdwunder.com\/wp-content\/themes\/wundertheme2024\/assets\/img\/logo.png"},"openingHoursSpecification":[{"@type":"OpeningHoursSpecification","dayOfWeek":["Monday","Tuesday","Wednesday","Thursday"],"opens":"09:00","closes":"17:00"},{"@type":"OpeningHoursSpecification","dayOfWeek":["Friday"],"opens":"10:00","closes":"16:00"}]}],"contactPoint":{"@type":"ContactPoint","contactType":"Sales","email":"info@thirdwunder.com","telephone":"+15144371652"},"sameAs":["https:\/\/facebook.com\/ThirdWunder","https:\/\/instagram.com\/thirdwunder","https:\/\/twitter.com\/thirdwunder","https:\/\/linkedin.com\/company\/ThirdWunder","https:\/\/www.youtube.com\/@thirdwunder","https:\/\/github.com\/thirdwunder"]}]}},"_links":{"self":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary\/12343","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary"}],"about":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/types\/tw_glossary"}],"wp:attachment":[{"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/media?parent=12343"}],"wp:term":[{"taxonomy":"tw_glossary_letter","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-letter?post=12343"},{"taxonomy":"tw_glossary_category","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-category?post=12343"},{"taxonomy":"tw_glossary_topic","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-topic?post=12343"},{"taxonomy":"tw_glossary_tag","embeddable":true,"href":"https:\/\/thirdwunder.com\/fr\/wp-json\/wp\/v2\/glossary-tag?post=12343"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}