CrawlLoom documentation

Start an audit

CrawlLoom is a browser-rendered SEO crawler and a focused alternative to Screaming Frog for inspecting dynamic website content. It can extract JavaScript-rendered and Kentico-delivered content, but it does not implement every Screaming Frog feature or identify a CMS with certainty.

1. Open the dashboard and sign in. An administrator leaves the Username field blank and uses the owner password; an Auditor uses the username and password assigned by the administrator. Only administrators can open administration.
2. Enter a target such as example.com or https://www.example.com/section/. A bare hostname is expanded to HTTPS.
3. Select the crawl scope. The initial Single URL setting audits only one page. Choose Exact hostname to follow links across that host.
4. Set the page limit and, if needed, open Advanced Directives & Exclusions. Switching to a multi-page scope normally starts with 50 pages and depth 3.
5. Execute the crawl. The main button changes to Pause and then Resume. Abort stops the run; Clear / Reset clears the current working results, not saved database history.

Only crawl sites you own or have permission to audit. A successful HTTP response does not guarantee that every dynamic component has finished loading.

When an audit finishes

A crawl you started that completes with at least 50 audited pages triggers an in-app notification. Desktop notifications depend on browser support and permission. Aborted runs and loading saved history do not count as completed-crawl notifications.

Scope, limits & advanced settings

Page limit and depth are different

The page limit controls how many pages may be audited. The crawler reserves a page slot before a worker starts it, so the selected limit remains exact even with multiple workers. Depth controls how many links away from the starting page the crawler follows: the starting page is depth 0, its linked pages are depth 1, and their links are depth 2. A depth of 3 does not mean three pages.

No page limit removes the user-selected count, but the depth, scope, exclusions and server safety ceiling still apply. The default server ceiling is 50,000 pages. A crawl can finish below its limit when no eligible URLs remain.

Custom proxy endpoints are disabled. Request settings do not make a crawl anonymous or bypass a website’s access controls.

What target sites see

Browser, direct-page and link-check requests currently use this browser-style User-Agent, rather than a CrawlLoom-branded crawler name:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

The robots.txt fetch uses a shorter Mozilla/macOS User-Agent. Target logs can still record the hosting server’s real public IP and request patterns. The advertised browser version is a header value, not necessarily the installed Chromium version.

Reading the dashboard

• Pages: search, filter, sort and paginate audited pages. The default page size is 50 rows. All pages shows the complete result set; Page title, Meta description, Meta keywords, H1, H2 and Content show only pages with the corresponding data. To find missing H1s, use All pages or SEO issues—not the H1 subtab.
• Discovered links & anchors: inspect destinations, anchor text, source pages, internal/external classification, content-area membership, status and redirects.
• Resources & assets: search, sort and filter scripts, styles, images, fonts, media and other discovered resources. Source-page information shows where an asset was encountered.
• SEO issues: select an issue category in the compact list to filter affected records. Search, severity indicators, pagination and Inspect help review each finding.
• Content inspection: use Inspect on a page, then open Content area to review detected content, headings and extracted text. Use Copy all content there to place the extracted text on your clipboard.
• History: browse saved MySQL runs and load a selected run into the same dashboard tabs.

Inspect opens a tabbed page dialog with Overview & SEO directives, Content area, Discovered links and source-versus-rendered code tools. Inspect from the Content subtab opens the content view. Long content and tables may require scrolling; exports provide another way to examine large audits.

Summary counts measure different things: audited pages, queued URLs, discovered link records, detected content areas and errors. A link appearing on multiple source pages can contribute multiple records. Issue totals count findings, not unique affected pages.

Pages and Discovered links tables wrap long URLs and text to keep every column inside the panel, including source pages, redirect destinations and Inspect. On narrow screens, records switch to labelled fields without hiding values. Sorting remains available above the records.

Rendering & content detection

Browser rendering versus direct HTML

The engine first attempts Chromium rendering through Playwright. On managed Linux hosting it prefers the bundled Sparticuz Chromium. It waits for document loading and bounded hydration checks, then optionally scrolls and extracts the DOM. It cannot guarantee readiness of every SPA, delayed API request, login wall or interactive widget.

If browser startup or a page render fails, direct HTML extraction can still read the server response. Direct DOM / HTTP mode does not execute JavaScript. This explains why a server-rendered homepage may contain rich content while other client-rendered routes appear empty. Check the engine indicator and runtime error before interpreting missing content as an SEO problem.

“Content area not detected” does not mean “no page content”

The app separately extracts full-page text and identifies a useful main content block. With no custom selector, it tries known content selectors and then scores semantic or structured containers, favouring headings and paragraphs over navigation, menus, cookie notices and link-heavy blocks. The heuristic requires at least 120 words and sufficient structure and score.

A short valid page, an unusual template or an unmatched custom selector can have full-page text without a detected content area. Inspect the page’s text and DOM, clear a stale custom selector to enable automatic detection, or supply a selector matching the intended section. Detection is not proof that the content came from Kentico.

Source HTML versus rendered DOM

Comparison metrics describe the server HTML and the extracted rendered DOM: bytes, words, scripts, elements and whether the DOM changed. The inspector can also fetch the actual source and rendered code while the crawler is idle.

Code previews are captured on demand and capped at approximately 2 MB per side. They are not saved historical HTML snapshots: opening code for an old audit captures the website as it exists now. If rendering fails, the comparison explains the limitation. No screenshot or complete offline website archive is stored.

Redirects, links & resources

The crawl queue normalizes URLs and tracks visited destinations to avoid repeatedly auditing the same page after a root / www redirect. Link records retain the original target and, when checked, the initial status, redirect chain, final URL and final status. A redirect can therefore remain visible even when the destination is not audited twice.

Link checking uses bounded parallel HTTP requests, typically HEAD with GET fallback. A link check is not a browser-rendered page audit. Timeouts, access blocks, unsupported schemes and unverified destinations must be interpreted separately from a confirmed HTTP 404.

Resource discovery combines references in the DOM with observed browser requests. To reduce memory and bandwidth, the browser intentionally blocks some images, media, fonts and analytics requests. A blocked resource is not automatically broken. An unknown status or size is not evidence that the resource is empty or missing.

Only metadata about assets is retained; CSS, JavaScript, images, fonts and media binaries are not downloaded into a permanent asset library. Resources injected after the bounded extraction window or requiring interactions may not be discovered.

Image SEO auditing

Open Resources & assets → Images. Each row represents an img occurrence on a source page—not a unique image URL. Reusing an image with different alt text produces separate rows. The other resource filters still provide the broader asset inventory.

Filter missing alt attributes, explicitly empty alt, whitespace-only alt, useful text presence, missing declared dimensions, known responses over 200 KB, unknown sizes or HTTP errors. Search includes image URLs, alt text, source pages and declarations. Columns can be sorted, results are paginated at 50 rows, and Inspect shows the stored attributes without loading a thumbnail.

Missing alt and empty alt are different: an absent attribute creates a review warning, but empty alt is not automatically an error. Decorative images may intentionally use empty alt, as explained in W3C’s decorative image guidance. The app cannot decide an image’s purpose or judge the quality of its description.

Dimensions distinguish declared HTML width/height, natural loaded dimensions and the rendered box. Unknown values appear as “?” rather than zero. A page using CSS sizing may validly omit HTML dimensions. Sizes come only from successful observed image response Content-Length headers; they are not guaranteed decoded sizes or total transfer costs. No extra downloads are made to obtain sizes. Crawler-blocked images, lazy content and unobserved responses often have unknown sizes/natural dimensions.

The 200 KB threshold is a configurable-in-code review rule, not a universal SEO requirement. Missing-alt and large-image findings also appear in SEO issues and issue exports. Unknown size is never treated as an oversized or broken image.

Browser mode records the selected currentSrc when available. Direct HTML mode retains src/srcset and common lazy-source declarations but does not guess responsive candidates. This audit covers img elements, not CSS backgrounds or standalone SVG elements. Inline data-URL payloads are omitted from image records; no permanent image archive is created.

Images CSV and the workbook’s Image SEO sheet retain occurrence-level details. New audits persist them in the additive, nullable crawl_pages.images_json column. Existing saved runs remain intact but need a fresh crawl to collect missing image detail. The column is added automatically when configured database storage initializes.

SEO rules & duplicate content

The dashboard and CSV / Excel exports use the same rules from src/shared/seoIssues.js. Most on-page checks apply to HTTP 200 results. These are review prompts, not search-engine ranking guarantees.

Exact duplicate detection normalizes Unicode, letter case and whitespace, then compares text from successful pages with at least 100 words. Detected content-area text is preferred, otherwise full-page text is used; matching requires the same text source. It does not calculate near-duplicate similarity and cannot find pages outside the audit.

A noindex directive may be intentional. Likewise, multiple H1s, canonical targets and short content need human context before changes are made. Thin-content checks prefer the content-area word count and otherwise use the full-page count.

Exporting results

The Export menu offers a multi-sheet Excel workbook and separate Pages, All Links, Resources, Images, SEO Issues and Content Area CSV files. The workbook contains overview, links, issues, resources, Image SEO and per-page detail sheets. Exports use the currently loaded audit, including a saved audit restored from History.

Exports represent the available audit data, not just the visible table page or selected UI filter. Wait for completion if you need a final report. Export endpoints require a signed-in session and the server-issued dashboard session ID; copied download URLs are not public sharing links.

Keep exported reports in an appropriate location: targets, extracted text and link details can be sensitive even though this documentation is public.

Saved history & database storage

MySQL persistence is optional. Without configured database credentials, the app can crawl in memory, but it cannot provide durable saved history. With MySQL enabled, runs and page results are written as the crawl progresses.

Opening a saved crawl

Choose History, then View for a run. CrawlLoom restores a small first window of saved pages rather than downloading every stored text field and link at once. Page, discovered-link and resource tables retrieve further windows from MySQL as you search, sort or paginate; this keeps even a very large CSS, JavaScript, image, font and media inventory available without loading it all into the browser. Opening a page retrieves its complete saved content, links and assets only for that inspection. Exports still retrieve the complete saved audit on demand. Stop the current crawl first; a paused run still counts as active. The elapsed timer freezes while paused, so it reports active crawl time rather than time left idle. Restoring history is a read-only review of saved results, not resuming its unfinished queue.

Resuming an unfinished crawl

Pausing, stopping or signing out checkpoints the remaining queue and completed URLs in MySQL. After signing in again, choose Resume beside an unfinished run in History, or open it and choose Resume saved crawl. CrawlLoom resumes only the saved pending URLs; it does not expand the queue from old external links. A completed crawl cannot be resumed. If a process restart happened before a queue checkpoint was saved, start a new crawl instead.

Comparing saved crawls

In History, select an earlier saved audit as Previous and a later audit as Current, then choose Compare crawls. CrawlLoom reports URLs that are new, missing or changed, including status codes, titles, descriptions, canonicals, robots directives, H1s, word counts, internal-link counts and rendered-content fingerprints. Comparison reads saved metadata only; it does not reopen either audit in the working dashboard.

Older runs can only show fields that were saved at the time. New storage support cannot recover data never retained. Database write failures are logged and may leave partial history even while an in-memory crawl continues successfully.

Clearing and retaining data

Clear / Reset affects the working dashboard. In Administration, Saved crawls lets an administrator permanently delete one audit and all of its pages, links, resources and extracted content. The separate clear-history action deletes every saved audit after the required confirmation. Both actions preserve the schema and security-event records; a currently running audit cannot be deleted. There is no in-app undo.

History is shared within this private admin workspace, not a separate archive for each person. There is no automatic age-based history or security-event cleanup. The History list initially requests the latest 25 runs; a listing limit does not mean older database records were deleted.

The database layer creates missing tables and adds supported missing columns at startup without intentionally dropping existing history. Operator-managed backups and restore procedures remain separate from the app. Test a backup restore independently; a successful crawl does not verify backup recoverability.

Live updates & simultaneous crawls

Each dashboard tab obtains a server-issued session bound to its signed-in account session, whether Administrator or Auditor. Queues, results, controls and exports belong to that dashboard session. Refreshing normally reconnects to it while server memory remains available. This is not a durable job queue across deployments.

The default capacity is three simultaneous crawls with one asynchronous page worker each. On the managed Linux configuration, active crawls share a Chromium process but use separate page contexts. Contexts are closed after extraction; cookies are not a persistent login session across the entire audit. More workers and larger audits increase memory, CPU and outbound request load.

The dashboard receives live events over Server-Sent Events (SSE). It takes an initial snapshot, applies revision-ordered updates and resynchronizes after commands, reconnects and completion. Server heartbeats arrive about every 10 seconds. If the stream stalls for about 25 seconds, a two-second snapshot fallback keeps it updated without overlapping requests. Healthy streaming does not continually poll full results.

Pause stops scheduling new work; work already in progress may settle. Abort cancels the active run and closes its work as cleanup permits. Do not interpret a short cleanup delay as a new crawl. Starting another run, clearing results or restoring history resynchronizes the display to avoid mixing old updates with new state.

A process restart or deployment loses in-memory sessions and unfinished queues. Saved database records remain, but an interrupted run can retain a “running” status if it could not write its final update.

Private access & administration

The homepage, this documentation, About, Privacy, Terms and Acceptable Use are public. The dashboard at /app, administration, crawler API and exports require authentication. Protected routes also send noindex directives. Robots exclusions are advisory; authentication is the access control.

There are two access levels. The administrator is the owner account configured with ADMIN_PASSWORD. Administrators create and disable separate Auditor accounts in MySQL; Auditor passwords are salted and hashed, never stored as readable passwords. Every newly created crawl is assigned to the signed-in account. Dashboard history, comparison, inspection, restore, resume and export are private to that owner: Auditors cannot access another Auditor’s crawls, and Administrator dashboard history contains only Administrator-owned crawls. The administrator can still view and manage all stored runs from Administration. Older unassigned runs are retained for administration, but are not shown in an account dashboard history.

• Signed authentication cookies use HttpOnly and SameSite=Strict, with Secure handling on secure/production requests. Administrator and Auditor sessions expire after eight hours. The dashboard Sign out button ends the account session and stops/revokes its active dashboard tabs.
• Login failures are limited to five attempts per IP in a 15-minute window. There is currently no crawl-start frequency limit; concurrency and workload ceilings still apply.
• Origin checks guard state-changing requests when an Origin header is supplied. A strict Content Security Policy and other response headers apply across the dashboard and public pages.
• Input validation bounds crawl settings, patterns and request sizes. Outbound network policy rejects unsafe targets and checks addresses and redirects to reduce server-side request forgery risk.

These controls reduce risk; they are not a claim of complete security or an independent penetration test. Keep dependencies and deployment configuration maintained.

Admin tools

Auditor accounts let the administrator create separate credentials and disable access; disabling an Auditor immediately revokes their active dashboard sessions. The database overview reports stored counts and size information. Sessions shows recent/active dashboard and sign-in activity, including full IP addresses and browser information. Revoke unfamiliar sessions to remove access; revoking a dashboard session stops its crawl. An IP address is a network clue, not reliable proof of a unique person or physical device.

Session activity is held in memory, capped at 100 records and retained for up to seven days; “active” means recently seen, roughly within two minutes. Security events can persist separately in MySQL. The admin event view shows 10 events per page with Previous and Next controls; it does not automatically delete older records.

Local setup & development

Requirements: Node.js 22.17.0 or newer, npm and a supported Chromium runtime. MySQL is optional for local development. Run commands from the repository root.

npm ci
npx playwright-core install chromium
npm run client:build

Provide ADMIN_PASSWORD and ADMIN_SESSION_SECRET through your environment. For a local file, create .tmp/local.env in your editor; .tmp/ is ignored by Git. Replace the placeholders with private values. Do not put credentials under src/public/ or in client-side configuration.

ADMIN_PASSWORD=replace-with-your-local-password
ADMIN_SESSION_SECRET=replace-with-a-long-random-secret
PUBLIC_APP_URL=http://localhost:3000

One way to generate a random secret locally is:

node -e "console.log(require('node:crypto').randomBytes(32).toString('hex'))"

Start Express with explicit environment-file loading:

npm run start:local

Use npm run dev:local instead to watch for server changes. Both commands explicitly load the ignored local file; stop any existing server on port 3000 first. Include NODE_ENV=development in that file when using local HTTP, and keep these credentials separate from production.

Open http://localhost:3000/ for the public homepage, /app for the dashboard, /admin for administration and /docs for this guide. The standard npm start command uses existing process environment variables; it does not automatically read a .env file.

For integrated UI/API testing, rebuild the client and use Express at /app. Backend auto-reload does not refresh a stale compiled client bundle. The application rejects private-network crawl targets by design; do not remove that protection merely to audit localhost fixtures.

Deployment & environment reference

The repository’s current Hostinger workflow serves Express and committed React build output. GitHub pushes to the connected deployment branch trigger deployment; local commits alone do not. Build and test before an agreed batch push. Documentation becomes publicly available on the hosted site only after that version is deployed.

Use Node 22.17.0 or newer, entry file server.js and start command npm start. The committed client bundle means production does not need to run Vite on every deployment. Dependencies must still be installed from the lockfile. If the deployment is changed to build the client remotely, it also needs development dependencies.

Keep concurrency conservative and observe memory, CPU, latency and browser failures under simultaneous load. A free crawl slot means capacity is available according to the configured limit, not that the host has unlimited resources. Sparticuz avoids many missing desktop-library problems but startup still depends on host compatibility.

The authenticated /api/debug/browser endpoint helps diagnose browser launch failures. It starts/checks the browser and should not be treated as a cheap public health probe. Database backups, hosting retention and restoration are operator-managed outside this app.

Code map & API reference

The React dashboard is the only dashboard interface at /app. /next/ is its protected asset path, not a separate public application.

Authenticated API families

This is the app’s internal same-origin API, not a public API-key service. Obtain a sign-in cookie through login, then create a dashboard session with POST /api/crawler/session. Crawler and export requests carry the returned sessionId; it is bound to the authenticated session, not a standalone access token. See src/client/api/crawler-client.ts and src/client/types/crawl.ts for request/response contracts.

Schema tables are crawl_runs, crawl_pages, crawl_links, security_events and app_users. Page and link persistence uses transactions with ordered pending writes. The Crawl Coordinator owns active in-process crawl state; saved checkpoints allow an unfinished crawl to be resumed after the user signs in again, but this is not yet a multi-server database-backed job scheduler.

Troubleshooting & known boundaries

Only one page is audited

Check Single URL scope, page limit 1 and depth 0 first. Then inspect discovered links, redirects, exclusions, robots rules and whether the engine fell back to non-rendered HTML.

Content exists, but the content-area badge says missing

Content extraction and identifying a main block are separate. Review full-page text, rendering status and the selector. Short pages may not pass the automatic heuristic.

The dashboard appears stuck at “Starting engine”

Check authentication expiry, the snapshot and stream requests in browser developer tools, and runtime logs. The live controller resynchronizes after commands and reconnects; repeated stalls need the failing request/error, not repeated starts.

Localhost shows an old interface

The public homepage is now at /. Use /app for React, rebuild with npm run client:build, and restart Express if server routes changed. Vite development and Express are separate processes.

History is unavailable or incomplete

Check database configuration and runtime persistence errors. In-memory results are not proof of a successful database write. Older records may lack fields introduced later.

Internal link redirects seem different from page status

The link record describes the original request and redirect chain. The audited page may describe the final successful destination. Compare the fields rather than only the displayed URL.

Resources say blocked or have no HTTP status

Some resource types are intentionally blocked to save bandwidth. Declared but unrequested resources can have unknown status. Neither is automatically a 404.

A site returns 403 or region-specific content

The target can restrict hosting-provider traffic. Browser region presets do not change the public egress IP. Do not treat these presets as a VPN or bypass.

A large audit slows down

Lower concurrency and scope, inspect hosting memory, and keep bounded page/depth limits. UI pagination does not eliminate server/client memory use for the complete dataset.

Not currently provided: durable pause/resume across restarts, individual user roles, permanent raw HTML/asset archives, near-duplicate similarity analysis, guaranteed full SPA readiness, or a full replacement for every desktop crawler capability.

This guide describes the repository revision reviewed on 8 September 2026. Keep it updated whenever configuration defaults, authentication, storage fields, UI behaviour or API contracts change. Use your browser’s Print command to save a PDF copy.