this config
{
"parameters": {
"api": {
"baseUrl": "https://httpbin.org",
"pagination": {
"method": "offset",
"limit": 100,
"limitParam": "size",
"offsetParam": "from"}
},
"config": {
"debug": true,
"outputBucket": "my-bucket",
"jobs": [
{
"endpoint": "/post",
"method": "POST",
"dataField": "data",
"dataType": "raw_mentions",
"params": {
"filters": {
"period": {
"end": {
"function": "strtotime",
"args": [
"today"
]
},
"shortcut": "last-1w"
}
}
}
}
]
}
}
}
adds the pagination to the post parameters body
[2018-07-23 09:29:07] logger.DEBUG: Using no authentication. [] []
[2018-07-23 09:29:07] logger.DEBUG: POST /post HTTP/1.1 Host: httpbin.org User-Agent: Guzzle/5.3.1 curl/7.38.0 PHP/7.1.14 Content-Type: application/json Content-Length: 113 {"filters":{"period":{"end":{"function":"strtotime","args":["today"]},"shortcut":"last-1w"}},"size":100,"from":0} [] [] [2018-07-23 09:29:08] logger.DEBUG: Processing results for my-bucket. [] []
[2018-07-23 09:29:08] logger.INFO: Extractor finished successfully. [] []
Is this a bug or a feature? I would expect it to do a request like this
POST /post?size=100&from=0
Host: httpbin.org
---json payload here---
{"filters":{"period":{"end":{"function":"strtotime","args":["today"]},"shortcut":"last-1w"}
2nd thing - the function is not evaluated in the jobs[0].params + an attribute can't be referenced "params": {"attr": "my_attr"}.
The "params" is simply taken as is - which I understand why, but it's a big bummer. Is there a way around this?
this config
adds the pagination to the post parameters body
Is this a bug or a feature? I would expect it to do a request like this
2nd thing - the function is not evaluated in the
jobs[0].params+ an attribute can't be referenced"params": {"attr": "my_attr"}.The "params" is simply taken as is - which I understand why, but it's a big bummer. Is there a way around this?