KETTLE-53: kettle.dataSource.URL.writable appears to truncate payloads that contain special characters...

Description

In recent work migrating older work to using kettle.dataSource.URL.writable in place of request, I started noticing odd problems when I would PUT or POST records. Around 400 out of 2700+ records I was uploading were failing to save.

After long investigation, I determined that JSON payloads containing special characters are not transmitted correctly, resulting in ECONNRESET errors.

To demonstrate this in isolation, I prepared the following simple code:

"use strict";
var kettle = require("kettle");
var dataSource = kettle.dataSource.URL({
    gradeNames: ["kettle.dataSource.URL.writable"],
    writeMethod: "PUT",
    url: "http://requestb.in/1jbw0ek1"
});

dataSource.set({}, { payload: "some éüø stuff"});

If I submit this, the payload is truncated before the closing quote and brackets, as in:

{
    "payload": "some éüø stuff

I used requestb.in for the demo, but when working with other services like CouchDB, I get ECONNRESET errors, which makes sense given the output above.

If I replace the payload with non-special characters, the entire payload is transmitted as expected. Looking at the docs, it seems like the defaults (JSON payloads, utf8 encoding) should be enough to work with special characters, but apparently it's not.

cc: @@Antranig Basman

Comments

• Antranig Basman commented 2017-01-05T07:48:27.313-0500

  This is most likely a duplicate of https://issues.fluidproject.org/browse/KETTLE-51 - would you like to test @@Cindy Li 's fix for it?

  https://github.com/fluid-project/kettle/pull/29

• Tony Atkins [RtF] commented 2017-01-05T08:41:09.202-0500

  Thanks, this very much seems like the same issue. I will test her branch/commit with the proof-of-concept code and report back shortly.

• Tony Atkins [RtF] commented 2017-01-05T08:48:21.170-0500

  Confirmed, Cindy's fix addresses the problem. I will close this as a duplicate.