{"id":1204,"date":"2024-04-28T15:33:55","date_gmt":"2024-04-28T15:33:55","guid":{"rendered":"https:\/\/buy-proxy-now.com\/?p=1204"},"modified":"2024-04-28T15:36:58","modified_gmt":"2024-04-28T15:36:58","slug":"9-tips-to-prevent-your-proxies-from-getting-blocked","status":"publish","type":"post","link":"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/","title":{"rendered":"9 Tips To Prevent Your Proxies from Getting Blocked"},"content":{"rendered":"<p>The results of your web scraping process depend on several factors \u2014 one of them is how well you manage your proxies. This article explores 9 tips to make this process easier.<\/p>\n<div class=\"article-content\">\n<ol>\n<li><a href=\"#1\">Use a user agents library<\/a><\/li>\n<li><a href=\"#2\">Use residential proxies from safe locations<\/a><\/li>\n<li><a href=\"#3\">Obey robots.txt and terms of use<\/a><\/li>\n<li><a href=\"#4\">Set a native referrer source<\/a><\/li>\n<li><a href=\"#5\">Set a limited number of requests<\/a><\/li>\n<li><a href=\"#6\">Change patterns<\/a><\/li>\n<li><a href=\"#7\">Beware of red flag search operators<\/a><\/li>\n<li><a href=\"#8\">Establish a decent rotation<\/a><\/li>\n<li><a href=\"#9\">Make sure the provider replaces proxies<\/a><\/li>\n<\/ol>\n<\/div>\n<p>Proxies are vital for successful scraping. However, if you don\u2019t manage them and your scraper properly, the IPs from your pool will constantly get blocked. Blocks slow down the data collection process and quickly drain the proxy pool your provider granted you access to.<\/p>\n<p>These tips will help you smooth out the data gathering and make sure you lose as few IPs as possible during scraping. The actions are very simple, but they will make a significant difference.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_82_2 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#1_Use_a_user_agents_library\" >1. Use a user agents library<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#2_Use_residential_proxies_from_safe_locations\" >2. Use residential proxies from safe locations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#3_Obey_robotstxt_and_terms_of_use\" >3. Obey robots.txt and terms of use<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#4_Set_a_native_referrer_source\" >4. Set a native referrer source<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#5_Set_a_limited_number_of_requests\" >5. Set a limited number of requests<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#6_Change_patterns\" >6. Change patterns<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#7_Beware_of_red_flag_search_operators\" >7. Beware of red flag search operators<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#8_Establish_a_decent_rotation\" >8. Establish a decent rotation<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/buy-proxy-now.com\/index.php\/9-tips-to-prevent-your-proxies-from-getting-blocked\/#9_Make_sure_the_provider_replaces_proxies\" >9. Make sure the provider replaces proxies<\/a><\/li><\/ul><\/nav><\/div>\n<h2 id=\"1\"><span class=\"ez-toc-section\" id=\"1_Use_a_user_agents_library\"><\/span>1. Use a user agents library<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Even if you\u2019re applying a new IP address for each request (e.g. when using proxies for sneaker bots), the same user agent will give you away and get your proxies blocked. The HTTP request header contains quite a lot of information about you and the device you\u2019re using. So it\u2019s a piece of cake for a destination server to tell that something is off if requests from different IPs come with the same user agent.<\/p>\n<p>Moreover, usually, scrapers send an empty header which is even worse. Then the destination server can detect that it\u2019s dealing with a bot right away because real users always have data in their user agents. That\u2019s why you have to configure your proxies and the scraper to send different headers with new requests. It is a widely-used practice, and you will even find user agent libraries on the Internet. You can feed it to your scraper so that it can use various headers.<\/p>\n<h2 id=\"2\"><span class=\"ez-toc-section\" id=\"2_Use_residential_proxies_from_safe_locations\"><\/span>2. Use residential proxies from safe locations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" class=\"kg-image\" src=\"file:\/\/\/media\/andrey\/Archives\/backups\/buy-proxy-now.com\/for-blog\/infatica\/9 Tips To Prevent Your Proxies from Getting Blocked_files\/residential-proxies-from-safe-locations.png\" alt=\"A map of various residential proxy servers\" \/><\/figure>\n<p>It\u2019s easy to tell which country the visitor is from by their IP address. To not raise suspicions, it\u2019s better to use proxies from the same place where the destination server is located. If, for example, a German e-commerce store that ships products only within Germany receives traffic from the USA, it is very odd. It can become a red flag for the server and, as a result, this IP address gets banned.<\/p>\n<p>Also, some locations are considered suspicious by default. For instance, many European and American websites won\u2019t allow users from Russia, China, and the Middle East because malefactors often come from these locations.<\/p>\n<p>So avoid using IP addresses from such countries if you\u2019re working with servers located in Europe, the States, Australia, or Canada. Obviously, if you\u2019re gathering data from Russian websites, it will be only logical to use Russian IPs. Infatica offers a very wide range of locations, so using our proxies you will never feel limited.<\/p>\n<h2 id=\"3\"><span class=\"ez-toc-section\" id=\"3_Obey_robotstxt_and_terms_of_use\"><\/span>3. Obey robots.txt and terms of use<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Every site has its rules that are registered in <code>robots.txt<\/code> and terms of use. Often, these rules outline which content can be used by visitors and how. Also, <code>robots.txt<\/code> controls crawlers and the pages they are allowed to access. Of course, you can bypass the restriction and get on restricted pages. But it will most likely get your IP address blocked. Moreover, it\u2019s not ethical to break the rules set by the website.<\/p>\n<p>It\u2019s also useful to go through terms of use and see if website owners have set any specific rules about the content. If you ignore those terms, an owner of the content you\u2019re gathering has the right to sue you for jeopardizing their intellectual property.<\/p>\n<h2 id=\"4\"><span class=\"ez-toc-section\" id=\"4_Set_a_native_referrer_source\"><\/span>4. Set a native referrer source<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" class=\"kg-image\" src=\"file:\/\/\/media\/andrey\/Archives\/backups\/buy-proxy-now.com\/for-blog\/infatica\/9 Tips To Prevent Your Proxies from Getting Blocked_files\/set-a-native-referrer-source.png\" alt=\"Users being referred to a link by twitter\" \/><\/figure>\n<p>The referrer is similar to the user agent \u2014 it provides the destination server with the information about the user. The difference is that referrers tell the site where the user comes from \u2014 the source that contained the link to the page. It can be some social media platform, another website, or a search engine.<\/p>\n<p>The traffic that had no referrer shows up as direct traffic. It can come from a normal user behavior \u2014 your request won\u2019t have a source if you type in the URL right into the web address field of your browser. But this kind of behavior is rare, especially if we\u2019re talking about pages with long addresses or ones that involve a random set of symbols.<\/p>\n<p>So empty referrers can become the reason why a destination server blocks your IP. It\u2019s even worse if you set a referrer to be some site that can\u2019t really send so many referred users. Then the bot activity is very easy to spot. That\u2019s why you need to use native referrer sources considering websites you\u2019re working with and your location. For instance, if you\u2019re scraping eBay, you want requests to pages to be referred by eBay.<\/p>\n<h2 id=\"5\"><span class=\"ez-toc-section\" id=\"5_Set_a_limited_number_of_requests\"><\/span>5. Set a limited number of requests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>If your scraper is sending requests at an insanely fast pace, the destination server will detect this activity and block it because most servers are protected from DDoS attacks. And a scraper that sends tons of requests looks like a malefactor who\u2019s trying to put the site down.<\/p>\n<p>Sure, raising the number of requests seems logical if you want to speed up data gathering. But in reality, this will only set you back as, for instance, your sneaker bot proxies will get blocked all the time. Implement a rate limit to make sure your scraper is not sending 10 requests within a second. Also, set breaks between requests \u2014 a 2-second delay can save you a lot of proxies.<\/p>\n<h2 id=\"6\"><span class=\"ez-toc-section\" id=\"6_Change_patterns\"><\/span>6. Change patterns<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" class=\"kg-image\" src=\"file:\/\/\/media\/andrey\/Archives\/backups\/buy-proxy-now.com\/for-blog\/infatica\/9 Tips To Prevent Your Proxies from Getting Blocked_files\/change-your-web-scraping-patterns.png\" alt=\"Various patterns of web scraping\" \/><\/figure>\n<p>It\u2019s easy to spot a pattern in the behavior of the bot. And servers with advanced anti-scraping measures and protection from attacks can detect repetitive patterns. In this case, you can avoid blocks by setting your scraper to be a bit random \u2014 for instance, show some cursor movement, random scrolls, and some clicks. This behavior will look human and natural.<\/p>\n<h2 id=\"7\"><span class=\"ez-toc-section\" id=\"7_Beware_of_red_flag_search_operators\"><\/span>7. Beware of red flag search operators<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Google has a whole list of search operators, but some of them can lead to CAPTCHA. One of the good examples of a red flag operator is <code>intitle<\/code> and <code>inurl<\/code> parameters as they\u2019re often used for stealing the content.<\/p>\n<p>So if you\u2019re performing bulk searches, it\u2019s better to not use search operators. But if you can\u2019t avoid using them, take all the tips we\u2019ve listed in the article and go extra with them. Set longer delays, implement more random actions, use more IPs, user agents, and so on. This will help you to minimize the risk of facing a CAPTCHA and getting blocked.<\/p>\n<h2 id=\"8\"><span class=\"ez-toc-section\" id=\"8_Establish_a_decent_rotation\"><\/span>8. Establish a decent rotation<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<figure class=\"kg-card kg-image-card\"><img decoding=\"async\" class=\"kg-image\" src=\"file:\/\/\/media\/andrey\/Archives\/backups\/buy-proxy-now.com\/for-blog\/infatica\/9 Tips To Prevent Your Proxies from Getting Blocked_files\/establish-a-decent-rotation.png\" alt=\"User accesses the website with different IP addresses\" \/><\/figure>\n<p>This is an obvious tip that, nonetheless, often remains forgotten. Each new request should have another IP address if you\u2019re sending requests to the same server. Otherwise, the site will quickly suspect something and block you. That\u2019s why you need quite a lot of <a href=\"https:\/\/buy-proxy-now.com\/index.php\/proxy-buy\/infatica\/\">IPs for scraping<\/a>. Using Infatica you will have access to a very large pool of proxies \u2014 we offer over one million <a href=\"https:\/\/buy-proxy-now.com\/index.php\/buy-proxy\/infatica\/\">residential IP addresses<\/a>.<\/p>\n<h2 id=\"9\"><span class=\"ez-toc-section\" id=\"9_Make_sure_the_provider_replaces_proxies\"><\/span>9. Make sure the provider replaces proxies<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Any rotation pattern will be useless if you have a small pool of IPs, and you get each of them blocked. In this case, your vendor should be able to provide you with more proxies. Infatica offers plans with pools of different sizes so that you can choose the one that fits your needs. Therefore, it\u2019s unlikely that you manage to get the whole pool blocked. However, if you face this issue, we always have new IPs for you. Infatica sources new proxies all the time.<\/p>\n<p>Following these tips, you will keep your data gathering fast and free from blocks. Therefore, you will get to use the same pool of proxies for much longer. These actions are quite effortless, and they might seem insignificant. But if you implement all of them, you will notice the difference.<\/p>\n<div id=\"collapseOne\" class=\"collapse show\" aria-labelledby=\"headingOne\" data-parent=\"#accordion\">\n<div class=\"card-body\"><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>The results of your web scraping process depend on several factors \u2014 one of them is how well you manage your proxies. This article explores 9 tips to make this process easier. Use a user agents library Use residential proxies from safe locations Obey robots.txt and terms of use Set a native referrer source Set [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_glsr_average":0,"_glsr_ranking":0,"_glsr_reviews":0,"footnotes":""},"categories":[78],"tags":[],"class_list":["post-1204","post","type-post","status-publish","format-standard","hentry","category-proxy-server"],"_links":{"self":[{"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/posts\/1204","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/comments?post=1204"}],"version-history":[{"count":4,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/posts\/1204\/revisions"}],"predecessor-version":[{"id":1208,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/posts\/1204\/revisions\/1208"}],"wp:attachment":[{"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/media?parent=1204"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/categories?post=1204"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/buy-proxy-now.com\/index.php\/wp-json\/wp\/v2\/tags?post=1204"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}