Business support

CDN Log Collection and Delivery

By analyzing access logs, you can view utilization and performance of your websites, applications, content, and hardware/software resources. They can also help you analyze or debug new or errant functionality. Response times, resource usage, traffic, trending, geographic patterns, errors and a wealth of available information can be derived from the access logs generated by the CDN edge servers. The volume of log data generated by high request-rate properties can be extremely large, so you'll want to plan carefully and consider the resources required to retrieve and analyze the data.

Log delivery formats

CenturyLink provides two methods for delivery of log information. Typically, logs are delivered from the edge to you within five minutes.
  • Log Streaming: the log streaming processor parses the logs, maps to specified customer endpoint(s), and sends the appropriate messages to the designated customer endpoint(s).  Supported customer designated endpoints include 1) a generic, customer-specified host or 2) a 3rd party cloud log analytic solution. The Log Streaming application sends CDN access logs through either an HTTP or HTTPS protocol (configurable as part of activation).

  • Log File Delivery: the log file delivery processor parses the logs, maps to specified customers, and sends the appropriate files to the designated customer log file folder on the CenturyLink Origin Storage Platform (OSP).  Log files follow W3C extended log file format. Once the service is enabled for you, the log data is written to a your Origin Storage Platform (OSP) account using your primary alias (e.g. centurylink.example.com) (by default, but can be configured) and will be retained until you delete the log files.

Message formats

  • JSON: messages are either JSON messages separated by newline characters or a JSON formatted array messages. A simple JSON example is as follows (New lines and beginning of line spacing added for clarity, not in final message):
      {
        "cached":1,
        "cs(Cookie)":"-",
        "cs(Referer)":"http://www.test.com/some/referrer/path",
        "cs(User-Agent)":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)
         AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
        "cs-ip":"127.75.23.245",
        "cs-method":"get",
        "cs-uri":"/test/test.gif?auth=ouAoARQAFoKCAAQABgAIAAoAGAAaR0b24xLnNq",
        "cs-version":"HTTP/1.1",
        "date":"2017-07-12",
        "sc(Content-Type)":"-",
        "sc-bytes":383,
        "status":200,
        "time":"22:27:10",
        "time-taken":0.01,
        "x-disid":2124407,
        "x-headersize":340,
        "cs(Range)":"bytes=0-123456”,
        "x-tcpinfo_rcv_space": 28960,
        "x-tcpinfo_rtt": 14000,
        "x-tcpinfo_rttvar": 7000,
        "x-tcpinfo_snd_cwnd": 32,
        "x-tcwait":245,
        "x-tdwait":0.003
      }

    There is no set number of messages in a batch sent for each POST.  Messages will be sent when either the Max Bytes/Message (in bytes) or when the Max Post Interval (in seconds) has been reached since the last HTTP POST.  If there are no messages to be sent in a batch, then the system will skip sending on that interval and wait until there is at least one message in the batch. This is done for each process, currently one per machine. Both configuration parameters have defaults defined by CenturyLink for optimal performance.

    The log streaming JSON encoding escapes common HTML characters by default.  The following characters are mapped to provided escaped Unicode:

    • ‘&’: ‘\u0026’
    • ‘<’: ‘\u003c’
    • ‘>’: ‘\u003e’
    • U+2028 (Unicode line separator): ‘\u2028’
    • U+2029 (Unicode paragraph separator): ‘\u2029’

These can commonly appear inside URL query strings or HTML parts sent through the log streaming system:

        --   “/example?key=value&hello=world” becomes “/example?key=value\u0026hello=world”

        --   “<b>hello world!<\b>” becomes “\u003cb\u003ehello world! \u003c\b\u003e”
 

  • W3C File: the log file format follows the W3C Extended Log File Format defined by the World Wide Web Consortium: http://www.w3.org/TR/WD-logfile.html. By default, messages will be available once the MaxBytesPerFile threshold is met (default set to 1 Gig) or when the MaxWaitSecForFinalWrite is met (default to 15-minute duration) (whichever occurs first).

    #Version: 1.0
    #Software: 1.0
    #Fields: date time cs-ip cs-method cs-uri status sc-bytes time-taken cs(Referer) cs(User-Agent) cs(Cookie) x-disid x-headersize cached cs(Range) x-tcwait x-tcpinfo_rtt x-tcpinfo_rttvar x-tcpinfo_snd_cwnd x-tcpinfo_rcv_space x-tdwait sc(Content-Type) cs-version
    2020-03-04      21:21:31.336    80.12.34.56     GET     /xyz.com/files/74c064e   206     66595   0.55    -       XYZ/10.0    -       584601  1059    1       bytes=38600704-38666239                 551     47240   20750   9       29200   0.000   -       HTTP/1.1
    2020-03-04      21:21:09.942    77.12.34.56   GET     /xyz.com/files/ff5bd902-c750      206     132136  0.00    -       XYZ/10.0    -       580701  1064    1       bytes=78774272-78905343                 0       71411   9533    100     29200   0.000   -       HTTP/1.1

Default log fields

The following fields are included in the default configuration for log delivery:

Field Type Example format Description
date <date> 2017-07-09 Date transaction completed
time <time> 23:27:35:142 Time transaction completed, HH is the hour in 24-hour format
cs-ip <address> IPv4: xxx.xxx.xxx.xxx
IPv6: xxxx:xxxx:xxxx:xxxx: xxxx:xxxx:xxxx:xxxx
IPv4 or IPv6 address of the requesting client, in dot notation
cs-method <string> get Method (e.g., get, head) in client request
cs-uri <string> /test Path fragment of URI
status <integer> 200 HTTP status code returned to the client, expressed as a single integer
Note: legacy name of this field was 'sc status'.
sc-bytes <integer> 1953 Number of bytes returned in response to request (including headers), expressed as a single integer
time-taken </float> 0.106 Time taken for transaction to complete (in seconds)
cs(Referer) <string> - Contents of referrer request header
cs(User-Agent) <string> urlgrabber/3.1.0 yum/3.2.22 Contents of user agent request header
cs(Cookie) <string> - Content of all cookie headers presented
x-disid <integer> 2037223 ID of distributor delivering request
x-headersize <integer> 301 Size of response header in bytes
cached <integer> 0|1
Indicates whether the object was served from cache or filled. The status will be either 0 (miss) or 1 (hit)
Note: legacy name of this field was 'cache'
cs(Range) <string> "bytes=0-123456" Contents of the range header
Note: legacy name of the field was 'x-range'
x-tcwait <integer> 567 Time (ms) spent waiting for the client to be ready—i.e. data was available to be sent, but the socket to the client was full
x-tcpinfo_rtt <integer> 678 Smoothed RTT in usecs
x-tcpinfo_rttvar <integer> 789 RTT variance in usecs
x-tcpinfo_snd_cwnd <integer> 890 Send congestion window
x-tcpinfo_rcv_space <integer> 901 Advertised recv window
x-tdwait <integer> 0.222 Time (ms) spent waiting for the upstream to send data
sc(Content Type) <string> "what did I just get" Contents of the content-type header
cs-version <string> "HTTP/1.0" The HTTP version (e.g., HTTP/1.1, h2)

Optional log fields

The following fields are optional. They're not included in the default configuration, but you can request that CenturyLink include them in your configuration:

Field Type Example format Description
cs-scheme <string>
"http" The scheme that the client request was made for, commonly http or https.
cs-host <string> "hello.world.com" The host of the client's request.
c-asn <integer> 432 The client ASN as determined from a lookup of the client IP in an online GeoIP database.  If unable to determine the value, the system will return zero.
c-city <string> "Denver" The client city as determined from a lookup of the client IP in an online GeoIP database.  The city name is provided in all lower case.  If unable to determine the value, the system will return "unknown".
c-state <string> "CO" The client state as determined from a lookup of the client IP in an online GeoIP database.  The state is provided in an all lower-case full state name and is used for other regional identifies outside of the United States for example, provinces and territories.  If unable to determine the value, the system will return "unknown".
c-cc <string> "US" The client country code as determined from a lookup of the client IP in an online GeoIP database.  This field follows the ISO 3166-1 alpha-2 standard for two letter country codes.  If unable to determine the value, the system will return an empty string.
x-extras <JSON key value> "freeform-key": A freeform set of key value pairs of extra information that can be custom defined.  All additional keys must be unique.
Note: This field can ONLY be enabled for JSON format.

Deprecated log fields

The following fields have been deprecated. CenturyLink will only provide these to you if you were previously configured to receive these fields:

Field Type Example format Description
x-Custom
<string>
-
All logs will contain the default value of -.
x-LogGroup
<string>
- All logs will contain the default value of -.
x-extstatus
<integer>
0 All logs will contain the default value of 0.

Activating CDN Log Collection and Delivery

CDN Log Streaming is a billable service and can be activated by the CenturyLink Service Activations team once we receive your signed order. Log collection and delivery can be enabled for one or more (or all) primary aliases under a given account as determined by you upon initial configuration.

Limitations

If your peak volume is >50,000 requests/second, you need special approval before activating Log Streaming so we can scale the solution to meet your needs. Contact your CenturyLink sales representative and they will work with our internal teams to review your needs.

Log Streaming delivery format is directly tied to the JSON message format.  File Delivery format is directly tied to the W3C message format.

View all topics for: