Connect an Amazon CloudFront site to AI Crawlers

This guide connects a site served by Amazon CloudFront to AI Crawlers, so you can see which AI bots read it. CloudFront already records every request. A small function in your own AWS account reads those logs, keeps the bot requests and sends them to us. We never get access to your AWS account. The result is a view of which AI bots read which pages of your site. It is written for the person who administers AWS for the site. Read it all, including the cost, before you start.

Before you start

• Pick CloudFront on your AI Crawlers page. Choose Create key and start setup to create your site key (shown once; store it like a password) and download index.js. Need a hand with your distribution or log bucket? Email team@aipeekaboo.com with your site address and the AWS Region of your log bucket.
• If the key may have leaked, replace it immediately, before anything else. Choose New key on your AI Crawlers page (or tell us) and update AIPEEKABOO_KEY on the Lambda function.
• Use one connection per site. If this site is already connected another way (the Cloudflare Worker, the WordPress plugin, a Vercel log drain or another CloudFront feed), do not also set up this one. Two feeds count the same crawler visit twice and the daily totals double.
• Allow about an hour of hands-on time for the steps, plus the wait for the first logs (see What to expect).
• If someone else manages your AWS account, send them this page. Send the key to them over a private channel such as a password manager, never email or chat.

What it costs

You pay AWS for the pieces that run in your account. The AWS services that bill are:

• Amazon CloudFront standard logging and its delivery to S3: check both CloudFront pricing and CloudWatch pricing for log delivery charges.
• Amazon S3 storage and requests for the log files: see S3 pricing.
• AWS Lambda requests and run time for the function: see Lambda pricing.
• Amazon SQS requests for the failures queue: see SQS pricing.
• Amazon CloudWatch for the function's own logs and any alarms: see CloudWatch pricing.

CloudFront logs every request, visitors included, so the amount grows with your traffic. Add a lifecycle rule to the bucket to delete old log files you do not need. After a day, check your AWS Billing page.

What is sent

CloudFront logs every request. The function sends us bot requests only:

• The page path without its query string, the method, the status and the time.
• The website's host name, the bot's name (its user agent) and the request ID.
• The website a visitor came from, reduced to its address only (no path).

It never sends requests from ordinary visitors, IP addresses, cookies or query strings, and the log fields in step 6 do not include them.

What you need from us

You get these three items on your AI Crawlers setup screen (or from us if we set the site up with you). Do not start before you have them.

What you need in AWS

• Permission to change the CloudFront distribution's logging, an S3 bucket, IAM, Lambda and SQS.
• The AWS CLI v2, signed in to the account that owns the distribution. The commands below also work from CloudShell.
• A Region for the bucket and the function. The function must be in the same Region as the bucket. The CloudFront log delivery itself is always configured in us-east-1 (N. Virginia), even when the bucket is in another Region.
• Keep the log bucket private: S3 Block Public Access on, as in step 1.
• A bucket folder used only for this site. These steps use the folder cloudfront/. Use a new bucket or a folder nothing else writes to, and do not use the bucket that holds the website itself, or the function could trigger on its own files.
• One distribution per site. If one distribution serves several unrelated domains, talk to us first: requests for domains that are not the connected site are dropped, and a file that holds only such requests is reported as failed (see Troubleshooting).

Replace these placeholders in every command and policy below. Keep your own values in a note as you go.

1. Prepare the bucket

The bucket must allow CloudFront's log delivery service to write. If you later enable logging in the CloudFront console, AWS can add this statement for you. When you use the CLI, add it yourself. This is the statement:

{
  "Sid": "AWSLogsDeliveryWrite",
  "Effect": "Allow",
  "Principal": { "Service": "delivery.logs.amazonaws.com" },
  "Action": "s3:PutObject",
  "Resource": "arn:aws:s3:::LOG_BUCKET/cloudfront/*",
  "Condition": {
    "StringEquals": {
      "s3:x-amz-acl": "bucket-owner-full-control",
      "aws:SourceAccount": "ACCOUNT_ID"
    },
    "ArnLike": {
      "aws:SourceArn": "arn:aws:logs:us-east-1:ACCOUNT_ID:delivery-source:*"
    }
  }
}

Put it in a bucket policy document. First see whether the bucket already has a policy:

aws s3api get-bucket-policy --bucket LOG_BUCKET --query Policy --output text

If it answers NoSuchBucketPolicy, wrap the statement above like this, save it as policy.json, and apply it. If a policy exists, that command prints it as JSON: save it to a file, add the statement to its Statement list instead, because put-bucket-policy replaces the whole policy:

{ "Version": "2012-10-17", "Statement": [ <the statement above> ] }

aws s3api put-bucket-policy --bucket LOG_BUCKET --policy file://policy.json

Keep the bucket private too. The logs hold page paths and user agents. Check whether S3 Block Public Access is on for the bucket:

aws s3api get-public-access-block --bucket LOG_BUCKET

If it answers NoSuchPublicAccessBlockConfiguration, or any of its four settings is false, turn all four on:

aws s3api put-public-access-block --bucket LOG_BUCKET --public-access-block-configuration BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

This does not stop log delivery: the policy above grants write access to an AWS service for your own account only, which S3 does not count as public.

If the bucket uses encryption with your own KMS key, the key policy must also allow delivery.logs.amazonaws.com to use it (AWS publishes the statement in the logging guide linked at the end). The function's role in step 2 then needs permission to decrypt with the same key.

2. Create the function's role

Create an IAM role that Lambda can assume, then give it these permissions: read the log folder, and write its own logs (a third, for the failures queue, comes in step 4).

Trust policy, saved as trust.json:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" }
  ]
}

Permission policy, saved as read-logs.json. It reads the log folder and nothing else. If the bucket has versioning turned on, keep the second action as well:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:GetObjectVersion"],
      "Resource": "arn:aws:s3:::LOG_BUCKET/cloudfront/*"
    }
  ]
}

aws iam create-role --role-name aipeekaboo-cloudfront-forwarder \
  --assume-role-policy-document file://trust.json
aws iam put-role-policy --role-name aipeekaboo-cloudfront-forwarder \
  --policy-name read-logs --policy-document file://read-logs.json
aws iam attach-role-policy --role-name aipeekaboo-cloudfront-forwarder \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

AWSLambdaBasicExecutionRole lets the function write its own CloudWatch logs. Do not give the role any wider S3 access.

3. Create the function

Put index.js at the root of a ZIP file. Nothing else goes in it. The function uses the AWS SDK that Lambda's Node.js runtime already includes.

Create env.json with the four settings below. The key is a secret, so work in a local folder that is not synced (not iCloud or Dropbox) and is not a git repository, such as the one the first command below makes, and copy index.js into it. Create the file privately first, then open it with an in-place editor such as nano or vi and paste the JSON in. Do not use echo or a heredoc: they put the key in your shell history. Before you run aws, check that ls -l env.json shows -rw-------, and delete any editor backup copy (env.json~, .env.json.swp). On Windows use CloudShell, or type the four values in the Lambda console under Configuration > Environment variables instead. WSL works only in a Linux folder such as one under /tmp, never under /mnt/c.

cd "$(mktemp -d)"
cp PATH_TO/index.js .
rm -f env.json
(umask 077; : > env.json)

{
  "Variables": {
    "AIPEEKABOO_KEY": "ck_<site>_<64 hex characters>",
    "AIPEEKABOO_ENDPOINT": "ENDPOINT",
    "CLOUDFRONT_LOG_BUCKET": "LOG_BUCKET",
    "CLOUDFRONT_LOG_PREFIX": "cloudfront/"
  }
}

A new role can take a few seconds to become usable. If create-function says the role cannot be assumed, wait ten seconds and run it again. Create the function in the same Region as the bucket:

zip function.zip index.js
aws lambda create-function --region BUCKET_REGION \
  --function-name aipeekaboo-cloudfront-forwarder \
  --runtime nodejs22.x --handler index.handler \
  --role arn:aws:iam::ACCOUNT_ID:role/aipeekaboo-cloudfront-forwarder \
  --zip-file fileb://function.zip \
  --timeout 300 --memory-size 256 \
  --environment file://env.json
rm -f env.json

The last line removes the file even when create-function fails. If the command is interrupted instead, run rm -f env.json yourself.

Notes:

• Lambda encrypts environment variables at rest with an AWS managed key by default. With the default key, anyone who can view the function's configuration can read the key, so limit that access. You may choose your own KMS key for the variables. Never copy the key into a ticket, a chat message, shell history or a log.
• The 300-second timeout and 256 MB of memory suit large log files. The function reads one file, then sends it in sequential requests, each with its own 15-second limit. Lambda's default timeout of 3 seconds is too short and makes every run fail. Raise the timeout (up to 900 seconds) if the Errors metric shows timeouts.
• To change a setting later, recreate env.json the same private way (in a fresh mktemp -d folder, as in step 3), run aws lambda update-function-configuration --function-name aipeekaboo-cloudfront-forwarder --environment file://env.json with all four variables in the file, and delete the file afterwards (rm -f env.json), even if the command fails or is interrupted. The command replaces the whole set.
• AIPEEKABOO_ENDPOINT must be an origin only: https, no path, query or credentials. CLOUDFRONT_LOG_PREFIX must end with / and match the folder used in step 6.

4. Retries and a place for failures

S3 calls the function asynchronously. Configure two retries and a queue that keeps any event that still fails, so nothing is lost silently.

aws sqs create-queue --region BUCKET_REGION --queue-name aipeekaboo-cloudfront-failures \
  --attributes MessageRetentionPeriod=1209600,SqsManagedSseEnabled=true

Allow the function's role to write to that queue. Save this as failures-queue.json:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": "sqs:SendMessage", "Resource": "arn:aws:sqs:BUCKET_REGION:ACCOUNT_ID:aipeekaboo-cloudfront-failures" }
  ]
}

aws iam put-role-policy --role-name aipeekaboo-cloudfront-forwarder \
  --policy-name failures-queue --policy-document file://failures-queue.json
aws lambda put-function-event-invoke-config --region BUCKET_REGION \
  --function-name aipeekaboo-cloudfront-forwarder \
  --maximum-retry-attempts 2 --maximum-event-age-in-seconds 21600 \
  --destination-config '{"OnFailure":{"Destination":"arn:aws:sqs:BUCKET_REGION:ACCOUNT_ID:aipeekaboo-cloudfront-failures"}}'

A new policy can take a few seconds to take effect. If put-function-event-invoke-config says the execution role cannot call SendMessage, wait 30 seconds and run it again.

Also create a CloudWatch alarm on the function's Errors metric and one on the queue's ApproximateNumberOfMessagesVisible. Restrict who can read the queue: each message holds the name of a log file.

5. Let S3 start the function

First allow S3 to invoke the function, limited to your bucket and your account:

aws lambda add-permission --region BUCKET_REGION \
  --function-name aipeekaboo-cloudfront-forwarder \
  --statement-id s3-invoke --action lambda:InvokeFunction \
  --principal s3.amazonaws.com \
  --source-arn arn:aws:s3:::LOG_BUCKET --source-account ACCOUNT_ID

Then add the notification. `put-bucket-notification-configuration` replaces the bucket's whole notification setup, so look at what exists first and keep it:

aws s3api get-bucket-notification-configuration --bucket LOG_BUCKET

If that prints nothing, use the file below as it is. Otherwise add the LambdaFunctionConfigurations entry to the existing document. Save as notification.json:

{
  "LambdaFunctionConfigurations": [
    {
      "Id": "aipeekaboo-cloudfront",
      "LambdaFunctionArn": "arn:aws:lambda:BUCKET_REGION:ACCOUNT_ID:function:aipeekaboo-cloudfront-forwarder",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": { "Key": { "FilterRules": [ { "Name": "prefix", "Value": "cloudfront/" } ] } }
    }
  ]
}

aws s3api put-bucket-notification-configuration --bucket LOG_BUCKET \
  --notification-configuration file://notification.json

Do not set a file-name suffix filter. The function recognizes compressed files by their content.

6. Turn on CloudFront logging

Run these commands in `us-east-1`, whatever Region the bucket is in. They use CloudFront standard logging (v2) with JSON output. The output format is fixed when the destination is created, so choose it now.

aws logs put-delivery-source --region us-east-1 --name aipeekaboo-source \
  --resource-arn arn:aws:cloudfront::ACCOUNT_ID:distribution/DISTRIBUTION_ID \
  --log-type ACCESS_LOGS

aws logs put-delivery-destination --region us-east-1 --name aipeekaboo-destination \
  --output-format json \
  --delivery-destination-configuration destinationResourceArn=arn:aws:s3:::LOG_BUCKET/cloudfront

aws logs create-delivery --region us-east-1 \
  --delivery-source-name aipeekaboo-source \
  --delivery-destination-arn arn:aws:logs:us-east-1:ACCOUNT_ID:delivery-destination:aipeekaboo-destination \
  --record-fields date time x-edge-request-id x-host-header cs-method cs-uri-stem \
                  sc-status "cs(User-Agent)" "cs(Referer)"

A distribution can have only one standard-logging delivery source. If it already logs somewhere else, add this destination to that delivery rather than replacing it, so your existing consumers keep working.

In the console the equivalent is the distribution's Logging tab: add an Amazon S3 destination, choose JSON, set the folder to cloudfront/, and select exactly the fields in the next table.

Fields

Select these fields and no others. Do not log c-ip, x-forwarded-for, cs(Cookie) or cs-uri-query.

Page paths can contain personal data that we cannot always recognize, so check your URLs before enabling this, and tell us first if they do.

7. Check that it works

Logging changes can take up to 12 hours to apply, and CloudFront starts delivering reliably about four hours after logging is turned on. Most logs then arrive within an hour of the request, and some can be up to 24 hours late.

1. Look for the first file under s3://LOG_BUCKET/cloudfront/.
2. When a file appears, open the function's CloudWatch log group /aws/lambda/aipeekaboo-cloudfront-forwarder. Each run prints one line of counts, for example {"event":"cloudfront_forward","outcome":"ok","objects":1,"rows":42,"forwarded":3,...}. On a run that ends "outcome":"ok", forwarded is the number of bot requests sent. The line also reports how many rows were skipped, by reason, and batchesNotStored. It never prints a path, user agent or key. If every bot request in a file lacks a required field, the run fails with invalid_row. If only some do, skipped.invalid_row counts them: recheck step 6 (a user agent that cannot be read counts under skipped.invalid_user_agent).
3. Make a test request that identifies as a crawler, for example curl -A "GPTBot/1.2" https://www.example.com/, and wait for the next log file.
4. In AI Peekaboo, open the site's AI Crawlers page. Rows appear about 2 to 4 minutes after the function ran, and the page rolls them up hourly, so allow up to an hour.

What to expect

• The first logs can take hours (see step 7). This is CloudFront's schedule, not a fault.
• Only requests that identify as bots or crawlers are sent. Ordinary visitors are not.
• Requests answered from the CloudFront cache are in the log too. The log does not say whether a request was a cache hit or a miss, so AI Peekaboo does not report it. A 403 is shown as an HTTP status only: this connection does not know why a request was denied.
• Requests older than about 24 hours cannot be stored, so totals can differ slightly from other tools when logs arrive very late.
• There is no backfill. Only files created after setup are read.

Troubleshooting

• The run prints skipped counts. Some rows cannot be used and are skipped without failing the file: damaged lines, malformed values, and requests older than about 23 hours 55 minutes or more than five minutes in the future. A file in which not one row, or not one bot request, can be read is reported as failed, which usually means the wrong format or fields (for example W3C output instead of JSON, or a missing field).
• `cloudfront: intake_not_stored` in the function's errors. AI Peekaboo received the data but kept none of it. Usual causes: every request in the file was for a host other than the connected site, or the site is not collecting yet. Write to us and we will check the site. When part of a file is stored, the requests for other hosts are simply left out.
• `intake_rejected`. The key is wrong, was replaced or was revoked, the endpoint is wrong, or AI Peekaboo answered with a server error. The file is retried twice and then waits in the failures queue, so replay it once the cause is gone. After a new key is entered, allow up to a minute for it to take effect.
• `invalid_row`. Not one row of the file could be read, or the file has bot requests and none of them could be read: the wrong output format (W3C, Parquet) or missing fields, for example no x-host-header. The file is retried and then waits in the failures queue. Recheck step 6. A destination's format cannot be changed, so create a new JSON destination.
• `invalid_notification`. The S3 event is not an ObjectCreated event for the configured bucket and folder. Recheck CLOUDFRONT_LOG_BUCKET, CLOUDFRONT_LOG_PREFIX and step 5.
• `timestamp_out_of_window`. Rows aged past the 24-hour limit while the file was being sent. Replaying is safe and skips those rows; they cannot be stored.
• `not_configured`. One of the four variables is missing or malformed.
• `object_too_large`, `line_too_large` or `too_many_rows`. A log file above 8 MiB (16 MiB uncompressed), a line above 64 KiB, or more than 20,000 bot requests in one file is refused. Contact us.

Replaying failed files

Failed events wait in the aipeekaboo-cloudfront-failures queue for 14 days. After fixing the cause, replay them one by one. The queue is a Lambda failure destination, not a dead-letter queue, so the SQS redrive feature does not apply. Each message body holds the original S3 event in requestPayload. Read one message first and confirm it contains an S3 event with a Records list (the message stays hidden for 30 seconds, so save the output to a file rather than running the command twice):

aws sqs receive-message --region BUCKET_REGION --queue-url QUEUE_URL --max-number-of-messages 1 > message.json
jq '.Messages[0].Body | fromjson | .requestPayload.Records | length' message.json

RECEIPT_HANDLE below is the message's ReceiptHandle (jq -r '.Messages[0].ReceiptHandle' message.json). Save the requestPayload value as event.json (CloudShell has jq), then run the function again with it and delete the message:

jq -r '.Messages[0].Body | fromjson | .requestPayload' message.json > event.json
aws lambda invoke --region BUCKET_REGION --function-name aipeekaboo-cloudfront-forwarder \
  --invocation-type Event --cli-binary-format raw-in-base64-out \
  --payload fileb://event.json response.json
aws sqs delete-message --region BUCKET_REGION --queue-url QUEUE_URL --receipt-handle RECEIPT_HANDLE

Replaying is safe: AI Peekaboo recognizes requests it has already stored. Rows older than 24 hours cannot be stored, so replay soon.

Changing the key

Choose New key on your AI Crawlers page (or ask us for a new key if we manage the connection), and when you have the new key, update AIPEEKABOO_KEY (in the console, or with update-function-configuration and all four variables, using a private env.json as in step 3, and running rm -f env.json afterwards even if the command fails), then check that the next run prints "outcome":"ok". The old key can keep working for up to a minute after it is replaced (our servers keep a key for 60 seconds). Files delivered between the change and your update fail with intake_rejected and wait in the failures queue, so replay them afterwards.

Uninstalling

1. Remove the logging delivery. It lives in us-east-1. Find its ID (an empty answer means it is already gone), delete the delivery first, then its destination and source:

aws logs describe-deliveries --region us-east-1 --query "deliveries[?deliverySourceName=='aipeekaboo-source'].id" --output text
aws logs delete-delivery --region us-east-1 --id DELIVERY_ID
aws logs delete-delivery-destination --region us-east-1 --name aipeekaboo-destination
aws logs delete-delivery-source --region us-east-1 --name aipeekaboo-source

2. Remove the bucket notification. Put the notification configuration back without the aipeekaboo-cloudfront entry. If it held nothing else, save {} as empty-notification.json and run:

aws s3api put-bucket-notification-configuration --bucket LOG_BUCKET --notification-configuration file://empty-notification.json

3. Delete the function, the role and the queue. Remove the role's two inline policies and detach the managed one first, or delete-role fails with DeleteConflict:

aws lambda delete-function --region BUCKET_REGION --function-name aipeekaboo-cloudfront-forwarder
aws iam delete-role-policy --role-name aipeekaboo-cloudfront-forwarder --policy-name read-logs
aws iam delete-role-policy --role-name aipeekaboo-cloudfront-forwarder --policy-name failures-queue
aws iam detach-role-policy --role-name aipeekaboo-cloudfront-forwarder \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
aws iam delete-role --role-name aipeekaboo-cloudfront-forwarder
aws sqs delete-queue --region BUCKET_REGION --queue-url QUEUE_URL
aws logs delete-log-group --region BUCKET_REGION --log-group-name /aws/lambda/aipeekaboo-cloudfront-forwarder

4. Clean up the rest. Delete the two CloudWatch alarms you created in setup step 4. Remove the AWSLogsDeliveryWrite statement from the bucket policy: save the policy with aws s3api get-bucket-policy --bucket LOG_BUCKET --query Policy --output text > policy.json, edit it, and apply it with aws s3api put-bucket-policy, or run aws s3api delete-bucket-policy --bucket LOG_BUCKET if it held nothing else. To stop paying for their storage, delete the log files under cloudfront/.

5. Disconnect the site last, once steps 1 to 4 are done, by choosing Disconnect on your AI Crawlers page, or the function will fail against a revoked key and the alarms will fire. Disconnecting does not stop the pipeline or your AWS charges, and it does not delete data already received.

AWS charges for the storage, requests and Lambda time this uses. You own those costs and the bucket's retention.

Need help?

Not sure what a screen means, or something looks wrong? Stop there and write to team@aipeekaboo.com or see Contact support. Tell us which step you were on and never send the key.

AWS references

• CloudFront standard logging
• Standard log fields
• Lambda with S3 events
• Lambda asynchronous invocation and failure destinations
• S3 Block Public Access and put-public-access-block
• S3 event message structure