Crawler Options

Top  Previous  Next

This panel allows you to configure all aspects of resource parsing, including parser restrictions and javascript engine's behavior.

 

Parser Options

 

crawler-options-1

 

Parse Robots file to obtain additional URLs

When enabled, N-Stalker will attempt to obtain additional URLs from robots.txt file (if available).

Parse Shockwave/Flash objects to obtain additional URLs

When enabled, N-Stalker will attempt to obtain additional URLs from SWF objects.

Parse CSS to obtain additional URLs

When enabled, N-Stalker will attempt to obtain additional URLs from Cascade Stylesheet files.

Parse Error Pages

When enabled, N-Stalker will also parse error pages to obtain additional URLs (status code 404, 403, 401, etc).

 

Javascript Engine

 

crawler-options-2

 

Interpret Javascript content using N-Stalker JS Engine

When enabled, N-Stalker's scan engine will execute javascript within its javascript engine sandbox.

Allow off-site javascript to be processed

When enabled, N-Stalker will obtain external javascript files and will execute them within the application's context (e.g: "google analytics").

Process Javascript events

When enabled, N-Stalker will process out-of-context javascript events such as "onClick", "onLoad", etc.

Parse Javascript to obtain additional URLs

If this option is enabled instead, N-Stalker will not execute javascript but will attempt to extract URLs from reading the code itself.

Ignore Javascript content

If this option is enabled instead, N-Stalker will completely ignore javascript code (neither it will interpret nor parse).

 

Excluded Javascript Events

 

crawler-option-3

 

Excluded Javascript Events

Use "Plus/Minus" button in the upper right corner to add/remove expressions to match javascript events that will not be processed (e.g: onclick). Note: JS Event processing must be toggled on.

 

Web Form Parsing Options

 

crawler-options-4

 

Ignore URL parameters when comparing FORM actions

When enabled, N-Stalker will treat form actions equally (ignoring query parameters) in order to parse and order resources. In this case, different web forms can be better classified as variations and enhance overall scan performance.