Jobs in Logpush now have a new key, **output\_options**, which replaces **logpull\_options** and allows for more flexible formatting. You can modify **output\_options** via the API.

## Replace logpull\_options

Previously, Logpush jobs could be customized by specifying the list of fields, sampling rate, and timestamp format in **logpull\_options** as [URL-encoded parameters](https://developers.cloudflare.com/logs/get-started/api-configuration/#options). For example:

```json
{
  "id": <JOB_ID>,
  "dataset": "http_requests",
  "enabled": false,
  "name": "<DOMAIN_NAME>",
  "logpull_options": "fields=ClientIP,EdgeStartTimestamp,RayID&sample=0.1&timestamps=rfc3339",
  "destination_conf": "s3://<BUCKET_PATH>?region=us-west-2"
}
```

We have replaced this with **output\_options** as it is used for both Logpull and Logpush.

```json
{
  "id": <JOB_ID>,
  "dataset": "http_requests",
  "enabled": false,
  "name": "<DOMAIN_NAME>",
  "output_options": {
    "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
    "sample_rate": 0.1,
    "timestamp_format": "rfc3339"
  },
  "destination_conf": "s3://<BUCKET_PATH>?region=us-west-2"
}
```

## Output types

By default Logpush outputs each record as a single line of JSON (also known as `ndjson`).

With **output\_options** you can switch to CSV or single JSON object, further customize prefixes, suffixes, delimiters, or provide your own record template (in a stripped-down version of Go [text/template](https://pkg.go.dev/text/template) syntax).

The **output\_options** object has the following settings:

* **field\_names**: array of strings. For the moment, there is no option to add all fields at once, you need to specify the fields names.

* **output\_type**: string to specify output type, such as `ndjson` or `csv` (default `ndjson`). This sets default values for the rest of the settings depending on the chosen output type. Some formatting rules (like string quoting) are different between output types.

* **batch\_prefix**: string to be prepended before each batch.

* **batch\_suffix**: string to be appended after each batch.

* **record\_prefix**: string to be prepended before each record.

* **record\_suffix**: string to be appended after each record.

* **record\_template**: string to use as template for each record instead of the default comma-separated list. All fields used in the template must be present in **field\_names** as well, otherwise they will end up as `null`. Format as a Go text/template without any standard functions (like conditionals, loops, sub-templates, etc.). The template can only consist of these three types of tokens:

  * Action: this is either a `{{ .Field }}` or a `{{ "constant text" }}`.
  * Text: this is just constant text in-between the `{{ actions }}`.
  * Comment: the `{{/* comments */}}` are silently dropped.

* **record\_delimiter**: string to be inserted in-between the records as separator.

* **field\_delimiter**: string to join fields. Will be ignored when **record\_template** is set.

* **timestamp\_format**: string to specify format for timestamps, such as `unixnano`, `unix`, or `rfc3339`. Default `unixnano`.

* **sample\_rate**: floating number to specify sampling rate (default 1.0: no sampling). Sampling is applied on top of filtering, and regardless of the current sample\_interval of the data.

* **CVE-2021-44228**: bool, default false. If set to true, will cause all occurrences of `${` in the generated files to be replaced with `x{`.

## Examples

Specifying **field\_names** and **output\_type** will result in the remaining options being configured as below for the specified **output\_type**:

### ndjson

* `ndjson` with different field names:

### csv

### csv/json variants

Based on above, other formats similar to csv or json are also supported:

* csv with header:

- tsv with header:

* json with nested object:

## How to migrate

In order to migrate your jobs from using **logpull\_options** to the new **output\_options**, take these steps:

1. Change the `&fields=ClientIP,EdgeStartTimestamp,RayID` parameter to an array in `output_options.field_names`.
2. Change the `&sample=0.1` parameter to `output_options.sample_rate`.
3. Change the `&timestamps=rfc3339` parameter to `output_options.timestamp_format`.
4. Change the `&CVE-2021-44228=true` parameter to `output_options.CVE-2021-44228`.

For example, if logpull\_options are `fields=ClientIP,EdgeStartTimestamp,RayID&sample=0.1&timestamps=rfc3339&CVE-2021-44228=true`, the output\_options would be:

```json
"output_options": {
  "field_names": ["ClientIP", "EdgeStartTimestamp", "RayID"],
  "sample_rate": 0.1,
  "timestamp_format": "rfc3339",
  "CVE-2021-44228": true
}
```
