Business support

CDN Log Streaming

By analyzing access logs, you can view utilization and performance of your websites, applications, content, and hardware/software resources. They can also help you analyze or debug new or errant functionality. Response times, resource usage, traffic, trending, geographic patterns, errors and a wealth of available information can be derived from the access logs generated by the CDN edge servers.

Log files are sent from the edge machines to the Log Streaming processor every 60 seconds. The Log Streaming processor parses the logs, maps to specified customer endpoint(s), and sends the appropriate messages to the designated customer endpoint(s). Supported customer designated endpoints include
 

  • a generic, customer-specified host
  • a third-party cloud log analytic solution

The Log Streaming application sends CDN access logs through either an HTTP or HTTPS protocol (configurable as part of activation). All messages are sent using a HTTP POST method and are either JSON messages separated by newline characters or a JSON formatted array of messages.

Notes:

  • Please check with your CDN sales specialist to find out if Log Streaming is available for your properties.
  • Only one customer-defined endpoint can be specified per customer.

Customer interface

The Customer Log Streaming Application sends CDN access logs to your designated endpoints. To do this, it has a defined default interface that is provided to your HTTP server. This can be done through either an HTTP or HTTPS protocol, configurable as part of activation. All messages are sent using a HTTP POST method with a JSON blob inside the body.

Message format

Messages are either JSON messages separated by newline characters or a JSON formatted array messages. A simple JSON example is as follows. (New lines and beginning of line spacing added for clarity and are not in final message.)

  {
    "cached":1,
    "cs(Cookie)":"-",
    "cs(Referer)":"http://www.test.com/some/referrer/path",
    "cs(User-Agent)":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)
     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
    "cs-host":"www.testsite.com",
    "cs-ip":"127.75.23.245",
    "cs-method":"GET",
    "cs-scheme":"http",
    "cs-uri":"/test/test.gif?auth=ouAoARQAFoKCAAQABgAIAAoAGAAahNidXR0b24xLnNq",
    "cs-version":"HTTP/1.1",
    "date":"2017-07-12",
    "sc(Content-Type)":"-",
    "sc-bytes":383,
    "sc-status":200,
    "time":"22:27:10",
    "time-taken":0.01,
    "x-disid":2124407,
    "x-extras": {
                  "freeform-key": "value"
                },
    "x-headersize":340,
    "x-range":"-",
    "x-tcpinfo_rcv_space": 28960,
    "x-tcpinfo_rtt": 14000,
    "x-tcpinfo_rttvar": 7000,
    "x-tcpinfo_snd_cwnd": 32,
    "x-tcwait":245,
    "x-tdwait":0.003
  }

Each log is made up of several fields, all fields are based on the W3C Extended Log Format notation:
 

  • cached[boolean] - is a boolean value that represents if the response was pulled from server's local cache. 1 is true and 0 is false in line with JSON boolean notation

  • cs(Cookie)[string, nullable] - is the cookie provided by the client header. Returns "-" if not existent

  • cs(Referer)[string, nullable] - is the referring URL as given in the client header. Returns "-" if not existent

  • cs(User-Agent)[string, nullable] - information about the client application (e.g. web browser type) sending the logging message. Returns "-" if not existent

  • cs-host[string] - is the host of the client's request

  • cs-ip[string] - is the client IP used in the request for content from our server. This is usually a user of one of our customer's products

  • cs-method[string, nullable] - is the method used in the request for content from our servers. This method is almost always "GET". Only returns a "-" in rare cases where the request is un-parsable

  • cs-scheme[string] - is scheme that the client request was made for, commonly http or https

  • cs-uri[string, nullable] - is the URI that the client tried to request. Only returns a "-" in rare cases where the request is un-parsable

  • cs-version[string, nullable] - is protocol version that the client request was made for, commonly HTTP/1.1. Only returns a "-" in rare cases where the request is un-parsable

  • date[string] - is the UTC date that the request was fulfilled, with the format of “YYYY-MM-DD”

  • sc(Content-Type)[string, nullable] - is the content type header from the request. Returns "-" if not existent

  • sc-bytes[int] - is the number of bytes sent for the full response. This is header and body together

  • sc-status[int] - is the HTTP status code returned by our server

  • time[string] - is the UTC time that the request was fulfilled, with the format of "HH:MM:SS"

  • time-taken[float] - is the time taken since first byte read to the last byte sent. This is measured in seconds and is a floating point value, with two significant digits

  • x-disid[int] - is the distributor id

  • x-extras[JSON key-value] - is a freeform set of key value pairs of extra information that can be custom defined. All additional keys must be unique

  • x-headersize[int] - is the byte size of the HTTP request header

  • x-range[string, nullable] - is the byte range of the request, formatted as "bytes=1111-2222", or "-" if it does not exist

  • x-tcpinfo_rcv_space[int] - is the buffer space allocated to process incoming requests measured in bytes

  • x-tcpinfo_rtt[int] - is the round-trip time in microseconds

  • x-tcpinfo_rttvar[int] - is the variance round trip time in microseconds

  • x-tcpinfo_snd_cwnd[int] - is the sending congestion window, a TCP variable that limits the amount of data sent before requiring a acknowledgment from the receiver

  • x-tcwait[int] - is the milliseconds that the system is waiting on the client to by ready

  • x-tdwait[float64, nullable] - is the time it takes in seconds with millisecond resolution to gather the data from upstream sources. This is pulled from the upstream response time metric. If null, then the field is omitted from the JSON

The following fields can be added upon request to CenturyLink CDN support with a custom field configuration:
 

  • c-asn[int] - is the client ASN as determined from a lookup of the client IP in an online GeoIP database. If unable to determine the value, the system will return zero

  • c-city[string] - is the client city as determined from a lookup of the client IP in an online GeoIP database. The city name is provided in all lower case. If unable to determine the value, the system will return "unknown"

  • c-state[string] - is the client state as determined from a lookup of the client IP in an online GeoIP database. The state is provided in an all lower-case full state name and is used for other regional identifies outside of the United States for example, provinces and territories. If unable to determine the value, the system will return "unknown"

  • c-cc[string] - is the client country code as determined from a lookup of the client IP in an online GeoIP database. This field follows the ISO 3166-1 alpha-2 standard for two letter country codes. If unable to determine the value, the system will return an empty string

There is no set number of messages in a batch sent for each POST. Messages will be sent when either the max message size (in bytes) has been reached as defined by the provisioning parameter, max_bytes_per_message, or when the max send interval has been reached since the last HTTP POST. This send interval is defined as part of the provisioning parameters as max_post_interval (in seconds). If there are no messages to be sent in a batch, then the system will skip sending on that interval and wait until there is at least one message in the batch. This is done for each process, currently one per machine. Both of these configuration parameters have defaults defined by CenturyLink for optimal performance.

Implementation

Log Streaming is a billable service and can be activated by the CenturyLink Service Activations team upon receipt of your signed order. Log Streaming can be enabled for one or more (or all) primary aliases under a given account as determined by you upon initial configuration.