Protect Web Endpoints

Important: Bot Defense Self-Service Policy Management is an Early Access feature.

Bot Defense uses JavaScript to collect telemetry from web applications. This telemetry is then attached in the form of HTTP headers or included in the POST body to the protected requests.

Bot Defense then examines the telemetry collected from requests before they are permitted to reach your application. Based on the Bot endpoint policy that you configure, Bot Defense can apply different mitigation actions that prevent automated traffic from reaching your application by blocking or redirecting the traffic.

Identify Web-Based Endpoints to Protect

When you configure Bot Defense, you must identify all the endpoints that you want to protect. F5 recommends that you protect all entry pages. An entry page is a page that users typically reach when they first visit your website. For example:

• A login page where a username and password are required to access protected content.
• A home page or landing page that users reach before visiting other pages on your site.

For information about how to identify the endpoints in your application that you must protect, see Perform a Site Analysis.

Web Endpoint Best Practices

F5 recommends that you follow these best practices when you configure protection for web endpoints:

• If your site is served from an apex domain, configure a redirect from the apex domain to the FQDN domain (for example, example.com redirects to www.example.com). This is required because your domain must point to a Distributed Cloud-provided CNAME. Pointing a CNAME from an apex domain is not recommended as that prevents you from having SOA and all other DNS records at apex.
• Do not use IP-based session persistence to load-balance internally. F5 Distributed Cloud has a range of egress IP addresses and if you have IP-based load balancing at the origin, this may break your session persistence. Use cookie-persistence, if possible.
• Restrict your infrastructure to Distributed Cloud egress IP addresses only. This prevents attackers from bypassing Distributed Cloud altogether and hitting your origin directly.
• Make sure your internal tools (such as logging or fraud tools) see the traffic accurately when going through Bot Defense.

Web Scraping

Configure web scraping protection for documents and endpoints on pages where JavaScript cannot be injected, or that are accessible by GET requests without visiting the main page, such as through a link on social media or from an email or saved bookmark.

For example, when you search for an airline flight through an online booking site and click a link that initiates a query on the airline site, you often bypass the entry page for the airline site.

This direct access to resources prevents Bot Defense from running the protection JavaScript. When you configure an endpoint for web scraping, Bot Defense displays an interstitial page that is transparent to the user but that allows Bot Defense to collect telemetry data about the requests.

HTTP Methods

For each endpoint that you protect, you must select which HTTP methods you want Bot Defense to protect. You can select multiple methods.

• GET(XHR/Fetch): Use when the protected application makes an XHTTPRequest or Fetch API call to get the content of the page. GET requests are protected only if they are sent by XHTTPRequest from a page with Bot Defense JavaScript injected, not from direct navigation using the address bar or link.
• POST: The most commonly attacked method. F5 recommends that you protect all POST requests. Bot Defense analyzes POST requests to determine if they are part of a bot attack such as credential stuffing or form submission abuse.
• PUT: Use to protect PUT requests and prevent data uploads or modifications from bots.
• PATCH: Use to protect PATCH requests and prevent bots from altering resource data.
• DELETE: Use to protect DELETE requests and prevent bots from deleting content or resources.
• GET(Document): Use to protect endpoints on a web page that can be accessible by GET requests without visiting the main page. When you configure an endpoint using GET(Document), Bot Defense displays an interstitial page that is transparent to the user but that allows it to collect telemetry data about the requests. Note that you cannot use GET(Document) with mobile endpoints.
• HEAD: Use to protect HEAD requests and to prevent bots from repeatedly pinging a server to check metadata.
• OPTIONS: Use to protect OPTION requests and to prevent bots from checking to find out what CORS options are supported by a server.
• TRACE: Use to protect TRACE requests to and prevent bots from repeatedly pinging a server with diagnostic requests.

Note: If you select both GET(XHR/Fetch) and GET(Document), only GET(XHR/Fetch) requests are protected. To protect both methods for the same endpoint, add the endpoint separately for each method.

Mitigation Actions

For each endpoint that you configure, you can specify which action you want Bot Defense to take when a bot is detected.

• Continue: Allows requests to continue to the origin. A log record is created. You can optionally add headers to the request.
• Block: The endpoint returns a status code and message or HTML code that you configure.
• Redirect: The endpoint forwards the browser to the URL that you specify. You can only select Redirect for web endpoints.
• Transform: Allows the request to continue to the origin. Headers are added to the request for inference and automation type. Use transform when you want to allow the request to continue and use a system other than Bot Defense to mitigate.

F5 recommends that you configure mitigation actions to mimic the normal behavior of the application so that an attacker does not know that the request was detected and intercepted. For example:

• For login pages, if you choose to block automated traffic, you can copy the HTML code for your "failed login" page into the mitigation configuration fields in Bot Defense. Bot Defense then presents the fake failed login page to bots.
• For a multi-factor authentication (MFA) page, you can block the request and configure Bot Defense to present a status code and message to make it appear that the MFA was not successfully completed.
• For a checkout page, you can configure Bot Defense to redirect automated traffic to a fake checkout page.
• When a web scraper tries to collect data from your pages, you can Configure Bot Defense to block the scraper and return information similar to the following:
  • Return a "202 Accepted" status code and a message that indicates processing has not completed yet, making a scraper think that processing is still in progress.
  • Present content that looks authentic, but change the body of the content to provide false information to the scraper, such as incorrect prices, quantity, and so on.

Bot Defense JavaScript Insertion

To detect automation, Bot Defense adds JavaScript tags to your HTML pages. The JavaScript runs in a user’s browser to collect environmental, behavioral and other data to help Bot Defense determine if a request is from a human source or from an automated source (a bot).

Before you configure how JavaScript tags are inserted, you must perform a site analysis to identify the endpoints in your environment that you want to protect. After you identify the endpoints that you want to protect, you must determine which is the entry point. The entry point is the URL of the page from which the request to the endpoint is sent. This is usually a page that a user or an attacker encounters first when they access your site. For example:

• A login page where a username and password are required to access protected content.
• A home page or landing page that users reach before visiting other pages on your site.

You must insert Bot Defense JavaScript tags in all entry points. In many cases, the entry point is a page with an HTML form. Keep in mind that users reach your website in different ways, such as by following bookmarks or through Internet search results, so you may have a large number of entry points to protect.

Decide which pages to include and exclude from protection

You can configure Bot Defense to inject JavaScript tags in all pages so that all pages are protected or select only specific pages in which to inject JavaScript tags. You can also explicitly exclude specific pages from JavaScript tag insertion so that they are not protected by Bot Defense.

JavaScript insertions count against your daily Bot Defense transactions. To avoid unnecessary transaction costs, F5 recommends that you configure Bot Defense to inject JavaScript tags only in pages that call protected endpoints.

However, inserting JavaScript tags in too few pages can cause an excessive number of false positive results. This happens when Bot Defense determines that a request is from an automated source because the request is missing the required telemetry.

Configure protection for specific pages

F5 recommends that you insert JavaScript tags in all entry points and any HTML pages where users are required to insert data. For example:

• Login pages, such as `/login/*.
• Your application Home page, such as /index.html.
• Pages that require data insertion, such as HTML form pages. For example, /catalog.

To specify the pages you want to protect, you add paths for the pages to a JavaScript inclusion list when you configure JavaScript insertion settings. You can also specify to protect a page in a specific domain or in all domains.

When you configure the entry points to protect, keep the following best practices in mind:

• To avoid protecting more endpoints than necessary, use wildcards cautiously when you enter endpoint URLs. If you choose to protect all HTML pages and endpoints (/*), F5 recommends that you select specific HTTP methods to protect. Protecting all HTTP methods for all pages can adversely impact performance.
• F5 recommends that you do not protect every HTML page in your application, especially pages that can only be accessed through an entry page, such as a login page, that you already protect.
• When you configure endpoints, use wildcards to terminate the match. For example: /login*. This prevents attackers from adding a slash to the end of the request, for example, /login/, when both variants reach the same application endpoint.
• If you want to protect a login POST request, the endpoint is the actual POST call that is triggered when the user clicks "Log in" after they enter their credentials. The entry point is the URL of the HTML page where the user enters credentials and clicks "Log in".

Exclude application pages from protection

You can configure Bot Defense to exclude specific HTML pages from protection. Bot Defense does not inject the Bot Defense JavaScript tags in pages that you exclude. You might exclude pages where you do not insert data, such as an image page.

This option is useful when you want to protect all paths under, for example, /main/central/*, but exclude /main/central/404.html or /main/central/502.html from Bot Defense protection.

You might also exclude protection, for example, from /login/images or /catalog/soldout/*.

You can also choose to exclude a specific page only in one specific domain or for all domains. You configure exclusion lists when you configure JavaScript insertion settings for your HTML pages.

Choose a Bot Defense JavaScript Path

Bot Defense allows you to specify a path or URL to display to the browser and in the page source code in place of the actual download path for the Bot Defense JavaScript. This prevents malicious actors from determining what system you are using to protect your endpoints.

F5 recommends that you choose a URL or path that is similar to your existing JavaScript files, but that does not include “F5,” “Bot Defense” or other indications that it is used for security purposes. For example, enter /examplepath.js or https://example.com/customer1.js.

Choose a JavaScript Tag Location

You must decide where in your HTML pages you want to inject the JavaScript tags. You can choose one of the following options:

• After <head> tag
• After </title> tag
• Before <script> tag

F5 recommends that you select After <head> so that the Bot Defense JavaScript is executed early. This allows time for the Bot Defense JavaScript to be fetched and executed while the rest of page is rendered.

Important: Make sure your HTML pages have the tag required for the option you choose (<head>, <title> or <script>).

---