Skip to content

Post request params + support for functions #102

Description

@pocin

this config

{
  "parameters": {
    "api": {
      "baseUrl": "https://httpbin.org",
      "pagination": {
        "method": "offset",
        "limit": 100,
        "limitParam": "size",
        "offsetParam": "from"}
    },
    "config": {
      "debug": true,
      "outputBucket": "my-bucket",
      "jobs": [
        {
          "endpoint": "/post",
          "method": "POST",
          "dataField": "data",
          "dataType": "raw_mentions",
          "params": {
              "filters": {
                  "period": {
                      "end": {
                          "function": "strtotime",
                          "args": [
                    "today"
                          ]
                      },
                      "shortcut": "last-1w"
                  }
              }
          }
        }
      ]
    }
  }
}

adds the pagination to the post parameters body

[2018-07-23 09:29:07] logger.DEBUG: Using no authentication. [] [] 
[2018-07-23 09:29:07] logger.DEBUG: POST /post HTTP/1.1 Host: httpbin.org User-Agent: Guzzle/5.3.1 curl/7.38.0 PHP/7.1.14 Content-Type: application/json Content-Length: 113 {"filters":{"period":{"end":{"function":"strtotime","args":["today"]},"shortcut":"last-1w"}},"size":100,"from":0} [] [] [2018-07-23 09:29:08] logger.DEBUG: Processing results for my-bucket. [] [] 
[2018-07-23 09:29:08] logger.INFO: Extractor finished successfully. [] [] 

Is this a bug or a feature? I would expect it to do a request like this

POST /post?size=100&from=0
Host: httpbin.org
---json payload here---
{"filters":{"period":{"end":{"function":"strtotime","args":["today"]},"shortcut":"last-1w"}

2nd thing - the function is not evaluated in the jobs[0].params + an attribute can't be referenced "params": {"attr": "my_attr"}.

The "params" is simply taken as is - which I understand why, but it's a big bummer. Is there a way around this?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions