streaming-logs.html.md.erb

---
title: App logging in Cloud Foundry
owner: PCF Metrics
---

<%# Reset page title based on platform type %>
<% if vars.platform_code != 'CF' %>

<% set_title("App Logging in", vars.app_runtime_abbr) %>

<% end %>

Loggregator, the <%= vars.app_runtime_full %> (<%= vars.app_runtime_abbr %>) component responsible for logging, provides a stream of log output from your app and from <%= vars.app_runtime_abbr %> system components that interact with your app during updates and execution.


## <a id='overview'></a> Overview

By default, Loggregator streams logs to your terminal. To persist more than the limited amount of logging information that Loggregator can buffer, you can drain logs to a third-party log management service. For more information, see [Third-party log management services](../services/log-management.html).

<%= vars.app_runtime_abbr %> gathers and stores logs in a best-effort manner. If a client cannot consume log entries quickly enough, the Loggregator buffer might need to overwrite some lines before the client has consumed them. A syslog drain or a command-line interface (CLI) tail can usually keep up with the flow of app logs.

## <a id='types'></a> Contents of a log entry

Every log entry contains four fields:

* Timestamp

* Log type, or origin code

* Channel: either `OUT`, for logs emitted on `stdout`, or `ERR`, for logs emitted on `stderr`

* Message

Loggregator assigns the timestamp when it receives log data. The log data is opaque to Loggregator, which only puts it
in the message field of the log entry. Apps or system components sending log data to Loggregator might include their own
timestamps, which then appear in the message field.

<%= vars.human_readable_timestamp %>

Origin codes distinguish the different log types. Origin codes from system components have three letters. The app origin code is `APP`, followed by slash and
a digit that indicates the app instance.

Many frameworks write to an app log that is separate from `stdout` and `stderr`. This is not supported by Loggregator. Your app must write to `stdout` or
`stderr` for its logs to be in the Loggregator stream. Ensure that the buildpack your app uses
writes logs to `stdout` and `stderr` only. Some buildpacks do this, and some do not.


## <a id='format'></a> Log types and their messages

Different types of logs have different message formats, as shown in the following examples. The digit appended to the code
indicates the instance index: 0 is the first instance, 1 is the second, and so on.

### <a id='api'></a> API

Users make API calls to request changes in app state. The Cloud Controller, the <%= vars.app_runtime_abbr %> component responsible for the API, logs the
actions that the Cloud Controller takes in response.

For example:

<pre class="terminal">
2016-06-14T14:10:05.36-0700 [API/0]      OUT Updated app with guid cdabc600-0b73-48e1-b7d2-26af2c63f933 ({"name"=>"spring-music", "instances"=>1, "memory"=>512, "environment_json"=>"PRIVATE DATA HIDDEN"})
</pre>

### <a id='stg'></a> STG

The Diego Cell or the Droplet Execution Agent emits `STG` logs when staging or restaging an app. These actions implement the state requested by the user. After the droplet is uploaded, `STG` messages end and `CELL` messages begin. For `STG`, the instance index is almost always `0`.

For example:

<pre class="terminal">
2016-06-14T14:10:27.91-0700 [STG/0]      OUT Staging...
</pre>

### <a id='rtr'></a> RTR

The Gorouter emits `RTR` logs when it routes HTTP requests to the app. Gorouter messages include the app name followed by a Gorouter timestamp and selections from the HTTP request.

For example:

<pre class="terminal">
2016-06-14T10:51:32.51-0700 [RTR/1]      OUT www.example.com - [14/06/2016:17:51:32.459 +0000] "GET /user/ HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.123 Safari/537.30" 192.0.2.132:46359 x_forwarded_for:"198.51.100.120" x_forwarded_proto:"http" vcap_request_id:9365d216-623a-45cb-6ef6-eba495c19fa8 response_time:0.059468637 app_id:79cc58aa-3737-43ae-ac71-39a2843b5178
</pre>

#### <a id='traceid'></a> Zipkin trace logging

If Zipkin trace logging is allowed in <%= vars.app_runtime_abbr %>, Gorouter access log entries contain Zipkin HTTP headers.

Here is an example access log entry containing Zipkin headers:

<pre class="terminal">
2016-11-23T16:04:01.49-0800 [RTR/0]      OUT www.example.com - [24/11/2016:00:04:01.227 +0000] "GET / HTTP/1.1" 200 0 109 "-" "curl/7.43.0" 10.0.2.150:4070 10.0.48.66:60815 x_forwarded_for:"198.51.100.120" x_forwarded_proto:"http" vcap_request_id:87f9d899-c7a4-46cd-7b76-4ec35ce9921b response_time:0.263000966 app_id:8e5d6451-b369-4423-bce8-3a7a9e479dbb app_index:0 x_b3_traceid:"2d5610bf5e0f7241" x_b3_spanid:"2d5610bf5e0f7241" x_b3_parentspanid:"-"
</pre>

For more information about Zipkin tracing, see [Zipkin tracking in HTTP headers](../../concepts/http-routing.html#zipkin-headers) in _HTTP Routing_.

### <a id='lgr'></a> LGR

Loggregator emits `LGR` logs to indicate problems with logging. Examples include `can't reach syslog drain url` and `dropped log messages due to high rate`.

### <a id='sys'></a> SYS

The Syslog Agent emits `SYS` logs to indicate problems with syslog drain delivery. This log type is emitted alongside `LGR` logs when messages are dropped due to slow drain consumption or connectivity issues. The `SYS` source type includes the source index of the Syslog Agent instance.

### <a id='app'></a> APP

Every app emits `APP` logs to Loggregator.

For example:

<pre class="terminal">
2016-06-14T14:10:15.18-0700 [APP/0]      OUT Exit status 0
</pre>

The logs allow you to determine the particular revision and process of the application instance logging.

For example, below shows the log line from two different revisions and web instances.
<pre class="terminal">
2022-09-15T01:59:33.99+0000 [APP/REV/2/PROC/WEB/0] OUT hello world from new
2022-09-15T01:59:34.99+0000 [APP/REV/1/PROC/WEB/1] OUT hello world from old
</pre>

<p class="note">
Logs with revision id are only available if the application uses revisions and it is enabled in the platform</p>


Each app might have a configured log rate limit. If the app logs exceed the configured log rate limit, you see a log entry indicating that the limit was exceeded.

For example:

<pre class="terminal">
2022-09-15T01:59:33.99+0000 [APP/PROC/WEB/0] OUT app instance exceeded log rate limit (16384 bytes/sec)
</pre>

### <a id='ssh'></a> SSH

The Diego Cell emits `SSH` logs when a user accesses an application container through SSH by using the Cloud Foundry Command Line Interface (cf CLI) `cf ssh` command.

For example:

<pre class="terminal">
2016-06-14T14:16:11.49-0700 [SSH/0]      OUT Successful remote access by 192.0.2.33:7856
</pre>

For more information about the `cf ssh` command, enter `cf ssh --help`.

### <a id='cell'></a> CELL

The Diego Cell emits `CELL` logs when it starts or stops the app. These actions implement the state requested by the user. The Diego Cell also emits messages when an app fails.

For example:

<pre class="terminal">
2016-06-14T13:44:38.14-0700 [CELL/0]     OUT Successfully created container
</pre>

### <a id='health'></a> HEALTH

The Diego Cell emits `HEALTH` logs when running health check actions for an app. These logs are generated by the health check binary that monitors app instances. By default, health check log output is suppressed unless an error occurs.

### <a id='proxy'></a> PROXY

Every app container includes an Envoy sidecar proxy that handles the mTLS handshake with the Gorouter and proxies traffic to the app process. The Envoy sidecar emits `PROXY` logs. By default, Envoy runs with a critical log level, so `PROXY` logs mainly appear when Envoy encounters critical errors, fails to start, or is killed due to the container running out of memory.

For example:

<pre class="terminal">
2026-01-26T13:49:18.72+0000 [PROXY/0]     OUT Exit status 137 (out of memory)
</pre>


## <a id='writing'></a> Writing to the log from your app

Your app must write logs to `stderr` or `stdout`. Both are typically buffered, and you can flush the buffer before
delivering the message to Loggregator.

Alternatively, you can write log entries to `stderr` or `stdout` synchronously. This approach is mainly used for
debugging because it might affect app performance.

## <a id='view'></a> Viewing Logs

To view logs, run the `cf logs` command. You can tail, dump, or filter log output. For more information about the  `cf logs` command, enter `cf logs --help`.

### <a id='tail-log'></a> Tailing logs

To stream Loggregator output to the terminal:

1. Run:

    ```
    cf logs APP-NAME
    ```
    Where `APP-NAME` is the name of your app. The command returns output similar to this example:
    <pre class="terminal">
    Connected, tailing logs for app spring-music in org example / space development as admin@example.com...

    2016-06-14T15:16:12.70-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:16:12.582 +0000] "GET / HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:27743 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:bd3e6ed1-5dd0-43ab-70ed-5d232b577b09 response_time:0.12050583 app_id:79bb58ab-3737-43be-ac70-39a2843b5177
    2016-06-14T15:16:20.06-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:16:20.034 +0000] "GET /test/ HTTP/1.1" 200 0 6879 "http://www.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:a31f0b1d-3827-4b8f-57e3-6f42d189f025 response_time:0.033311281 app_id:79bb58aa-3747-43be-ac70-39a3843b5178
    2016-06-14T15:16:22.44-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:17:22.415 +0000] "GET /test5/ HTTP/1.1" 200 0 5461 "http://www.example.com/test5" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.322" x_forwarded_proto:"http" vcap_request_id:5d6855a2-4a79-4432-7927-de8215f5a2c7 response_time:0.029211609 app_id:79bb58aa-3737-43bb-ac70-39a2943b5178
        ...
    </pre>

Press **Ctrl+C** or **Command+C** on your keyboard to exit the real-time stream.

### <a id='dump-log'></a> Dump logs

To display all of the lines in the Loggregator buffer:

1. Run:

    ```
    cf logs APP-NAME --recent
    ```
    Where `APP-NAME` is the name of your app.

### <a id='filter'></a> Filter logs

To view some subset of log output, run `cf logs APP-NAME` in conjunction with filtering commands of your choice, where `APP-NAME` is the name of your app. In the following example, run `cf logs spring-music --recent | grep -v RTR` to exclude all Gorouter logs:

<pre class="terminal">
$ cf logs spring-music
Connected, tailing logs for app spring-music in org example / space development as admin@example.com...

2016-06-14T15:16:12.70-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:16:12.582 +0000] "GET / HTTP/1.1" 200 0 103455 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:27743 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:bd3e6ed1-5dd0-43ab-70ed-5d232b577b09 response_time:0.12050583 app_id:79bb58ab-3737-43be-ac70-39a2843b5177
2016-06-14T15:16:20.06-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:16:20.034 +0000] "GET /test/ HTTP/1.1" 200 0 6879 "http://www.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.222" x_forwarded_proto:"http" vcap_request_id:a31f0b1d-3827-4b8f-57e3-6f42d189f025 response_time:0.033311281 app_id:79bb58aa-3747-43be-ac70-39a3843b5178
2016-06-14T15:16:22.44-0700 [RTR/4]      OUT www.example.com - [14/06/2016:22:17:22.415 +0000] "GET /test5/ HTTP/1.1" 200 0 5461 "http://www.example.com/test5" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36" 192.0.2.206:2228 x_forwarded_for:"203.0.113.322" x_forwarded_proto:"http" vcap_request_id:5d6855a2-4a79-4432-7927-de8215f5a2c7 response_time:0.029211609 app_id:79bb58aa-3737-43bb-ac70-39a2943b5178
    ...
</pre>

Use **Ctrl-C** (^C) to exit the real-time stream.

### <a id="dump-log"></a> Dumping logs

To display all of the lines in the Loggregator buffer, run `cf logs APP-NAME --recent`, where `APP-NAME` is the name of
your app.

### <a id="filter"></a> Filtering logs

To view some subset of log output, run `cf logs APP-NAME` in conjunction with filtering commands of your choice.
Replace `APP-NAME` with the name of your app. In the following example, `grep -v RTR` excludes all Gorouter logs:

<pre class="terminal">
$ cf logs spring-music --recent | grep -v RTR
2016-06-14T14:10:05.36-0700 [API/0]      OUT Updated app with guid cdabc604-0b73-47e1-a7d5-24af2c63f723 ({"name"=>"spring-music", "instances"=>1, "memory"=>512, "environment_json"=>"PRIVATE DATA HIDDEN"})
2016-06-14T14:10:14.52-0700 [APP/0]      OUT - Gracefully stopping, waiting for requests to finish
2016-06-14T14:10:14.52-0700 [CELL/0]     OUT Exit status 0
2016-06-14T14:10:14.54-0700 [APP/0]      OUT === puma shutdown: 2016-06-14 21:10:14 +0000 ===
2016-06-14T14:10:14.54-0700 [APP/0]      OUT - Goodbye!
2016-06-14T14:10:14.56-0700 [CELL/0]     OUT Creating container
    ...
</pre>

### <a id='log-ordering'></a> Log ordering

Both operators and developers must ensure log ordering in drains.

* Diego uses a nanosecond-based timestamp that can be ingested properly by Splunk. For more information, see the [Splunk
documentation](https://answers.splunk.com/answers/1946/time-format-and-subseconds.html).

* The Elastic Stack can ingest the nanosecond timestamps but only supports millisecond precision, so timestamps are truncated. For more information, see the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) and [Date type has not enough precision for the logging use case](https://github.com/elastic/elasticsearch/issues/10005) on GitHub.

If you are developing a client that displays a stream of <%= vars.app_runtime_abbr %> logs to users, you can order the logs to improve the debugging experience for your user. Here are some general tips for ordering logs:

* For CLIs, batch the logs and display the logs you have in that timeframe, sorted by timestamp.

* For web clients, use dynamic HTML to insert older logs into the sorting as they appear. This creates complete, ordered logs.

* Java app developers might want to convert stack traces into a single log entity. To simplify log ordering for Java
  apps, use the multi-line Java message workaround to convert your multi-line stack traces into a single log entity.
  For more information, see <a href="https://github.com/cloudfoundry/java-buildpack/issues/544">Multi-line Java message workaround</a>.
  <br><br>
  By changing the Java log output, you can force your app to reformat stack trace messages, replacing newline
  characters with a token. Set your log parsing code to replace that token with newline characters again to display the
  logs properly in Kibana.