Skip to content

Json parser performance boost#491

Open
jteagles wants to merge 1 commit into
twitter:masterfrom
jteagles:json-parser-performance
Open

Json parser performance boost#491
jteagles wants to merge 1 commit into
twitter:masterfrom
jteagles:json-parser-performance

Conversation

@jteagles

Copy link
Copy Markdown

I have created a test that shows that the new faster xml jackson library parser API can be taken advantage of in this case (and perhaps other elephant bird cases). My tests results shows 2x speed boost for typical deeper json hierarchies. The drawback here is the upgrade of the jackson library from 1.x line to 2.x line

@CLAassistant

CLAassistant commented Oct 16, 2018

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@jteagles jteagles force-pushed the json-parser-performance branch from ded82ca to 8544582 Compare October 16, 2018 21:34
@jteagles

jteagles commented Dec 5, 2018

Copy link
Copy Markdown
Author

@gerashegalov, do you know who the best person to contact for review?

@gerashegalov

gerashegalov commented Dec 5, 2018

Copy link
Copy Markdown
Contributor

@isnotinvain can you take a look, please ?

jsonGenerator.writeObject(jsonParser.getEmbeddedObject());
break;
default:
throw new IOException("Cno");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what doesn't mean?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Should have a better exception message

Map<String, String> result = udf_.exec(input);
assertTrue("It should return a Map", result instanceof Map<?, ?>);
assertEquals("value", result.get("name"));
assertEquals("\"value\"", result.get("name"));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why we got here additional quoting?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

extra quoting seems like exactly what we'd want to make sure isn't happening, so it'd important to sort that out. We would need this upgrade to be transparent to users, we can't have it behave differently.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jasonsimple is parsing and removing the quotes and leaving a bare string as the value. this is something to figure out.

return null;
protected Map<String, String> parseStringToMap(String line) throws IOException {
JsonParser jsonParser = jsonFactory.createParser(line);
jsonParser.nextToken();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need this call?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A nextToken is needed to advance to the first element. perhaps this could be done is a less hard coded way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants