Karate vs. Rest-assured
REST-assured | Karate | References / Comments | ||
---|---|---|---|---|
1 |
BDD Syntax |
Yes |
Yes |
|
2 |
True DSL |
No. Fluent Interface. |
Yes |
DSL vs Fluent Interface. Also see (24) and (25) |
3 |
Runs on the JVM |
Yes |
Yes |
|
4 |
Implementation |
Java and Groovy |
Java |
|
5 |
Code-base Size |
Large. 40,000 lines of code |
Medium. 30,000 lines of code |
Karate has API-mocks, perf-testing and UI automation as well. |
6 |
Mature |
Yes. Inception 2010. Lots of blog posts, tutorials and StackOverflow posts. In the ThoughtWorks Tech Radar at 5 years, appeared only once. |
Yes. Inception February 2017. Clear signs of wide and rapid adoption. In the ThoughtWorks Tech Radar within just 2 years and rating upgraded within 1 year. |
3000+ GitHub “stars” in 3 years Active community on Stack Overflow. |
7 |
JsonPath Implementation |
Groovy GPath |
JayWay JsonPath |
GPath has some limitations |
8 |
XPath implementation |
Groovy GPath and “XMLSlurper”. Standard XPath is also supported, but paths that return XML nodes cannot be used in assertions. Updating an XML document is not possible. |
W3C standard XPath using the Java built-in XML lib. You can even update XML documents using XPath. |
|
9 |
HTTP Client |
Apache 4.X, but the code depends on deprecated APIs. There are some concerns with this design. More details in this issue. |
Pluggable (future-proof). From v0.3 onwards, both the Apache and Jersey HTTP clients are supported. This means that you won’t be blocked if your project already has a conflicting version of one of these. |
Karate even has an option to mock a servlet container because of this abstraction. Karate also has minimal maven dependencies. |
10 |
Quick Start / Project Scaffolding |
No |
Yes (Maven Archetype). There’s also a standalone executable. |
Dev onboarding experience much better with Karate. Archetype Includes working example. |
11 |
Test-Scripting Language |
Java |
Karate-Script (Cucumber / Gherkin) |
No Java knowledge needed for Karate |
12 |
Test Scripts have to be compiled |
Yes |
No |
Tests are plain-text. No IDE required for Karate |
13 |
IDE Support |
Yes. Intelli-Sense, Auto-Complete and Refactoring work for Java and POJO-s |
Partial. Eclipse and IntelliJ have Cucumber plugins that work well and have pretty good syntax coloring. Not needing POJO-s means that the lines of code required for a test is dramatically reduced, see (39). |
Since you can re-use JSON payloads across tests, the “re-factorability” aspect is covered as well. |
14 |
Step Through / Debug-ability |
Yes. Java + IDE Support. |
Even better ! Debug in Visual Studio Code with step-back and even hot-reload (since v0.9.5). also see the Dev-mode HTML report: steps, error diagnostics and HTTP logs in-line - since v0.6.2 |
Also see (42) |
15 |
Test Runner |
Any, bring your own.TestNG or JUnit will work. |
JUnit supported and can co-exist with TestNG in the same project if needed. |
And Karate’s parallel execution capability is in “core”, independent of even Maven or any unit-testing framework. |
16 |
Tags / Groups Built In |
No (have to use TestNG or equivalent) |
Yes |
|
17 |
Extend with custom routines via... |
Java code |
JavaScript |
No need to compile, and easier for non-programmers. |
18 |
Re-use Java code |
Yes |
Yes (via JavaScript interop) |
|
19 |
Validate All Payload values in one step |
Not built-in. You need to use external libraries such as Hamcrest.
|
Karate natively supports a “deep equals” and “contains in any order” assertion for JSON, JSON arrays and XML, and lets you ignore chosen fields at the same time. This is super-important for GraphQL. |
IMO the biggest limitation of REST-Assured: |
20 |
Built-in data-type, conditional-logic and RegEx validations |
No |
Yes, includes RegEx and Macros |
|
21 |
Validate schema of all elements in a JSON array in one step |
No |
||
22 |
Built-in JSON Schema and XML Schema validation support |
Yes |
RegEx and Macros support is sufficient (and far simpler) for most use cases. That said, users can easily add a Java lib via Karate’s Java interop - if needed. |
For details on how Karate’s approach is simpler and more intuitive than JSON (or XML) Schema see this link. |
23 |
Native support for expressing JSON or XML in test-scripts |
No "{ \"name\": \"Billie\" }" "<cat name=\"Billie\"></cat>" |
Yes { name: 'Billie' } <cat name="Billie"></cat> |
No need to use double-quotes or “escape” characters. You can also read from files and re-use. |
24 |
Example – JSON assertions |
@Test public void lotto_resource_returns_200_with_e when(). |
Scenario: lotto resource returns 200 with expected id and winners Given path ‘lotto’, 5 |
Matching built-in, and more readable syntax. Note the simpler way to specify path parameters without placeholders. For REST-assured, IDE formatting is a known challenge. |
25 |
Example - GET with params |
given(). |
Given param key1 = ‘value1’ |
Karate is a true DSL. No syntax “noise”, no unnecessary symbols or punctuation. No need to worry about indenting a giant “one liner” of Java code. |
26 |
Extracting multiple data-elements for reuse in subsequent HTTP calls |
Convoluted. The Fluent Interface which is supposed to be the main highlight of REST-Assured actually gets in the way here. More examples |
Easy. You can even use JsonPath to extract JSON chunks or arrays and save them to variables for use in later steps. For XML, XPath does the same. |
Some of the quirks of the REST-assured JsonPath implementation get in the way as well. |
27 |
Can update a given JSON or XML using a path expression |
Especially for data-driven tests, updating nested JSON is near impossible. |
Yes. There are actually multiple ways to update payloads: a) by path b) using embedded expressions and c) via a built-in string replacement keyword. |
You can even modify a response and re-use it ‘as-is’ as the next request. |
28 |
Data Driven Testing |
No (have to use TestNG or equivalent) |
Yes. Can even use dynamic JSON or even CSV as a data-source. |
|
29 |
SOAP support |
Yes |
Plus, Karate’s XML support is far more flexible and easier to use. |
|
30 |
HTTPS / SSL without certificates |
Although there is “relaxed” HTTPS, a certificate is needed in some cases |
Yes |
|
31 |
Built-in support for switching environment config |
No Also config is somewhat convoluted in REST-Assured |
Yes Adding a new variable to a test is just one step: edit karate-config.js |
In REST-assured, you have to use something like a dependency-injection framework (or roll your own) to read properties files. |
32 |
File Upload / Multipart Support |
Partial / Buggy Libraries | Content-Type | Dependencies | ‘multipart/related’ not supported | questions on ‘multipart/mixed’ |
Yes |
|
33 |
URL encoded HTML Form data |
Yes |
Yes |
|
34 |
Cookies |
Yes |
Can be configured one-time for all subsequent requests, just like headers. |
|
35 |
Auth Schemes out of the box |
Yes |
No (but easily pluggable via re-usable scripts or JavaScript without needing to write Java code) |
|
36 |
Custom Auth |
Java code (needs compilation). Existing mechanism is not extensible. |
Unified plug-in system via JavaScript (no compilation needed) |
|
37 |
Parallel Execution of Tests |
No. While some teams seem to have had success running REST-assured in parallel, there are some cases in which multi-threading is not supported. Also see this thread. The creator has also confirmed that “REST Assured is not 100% safe to use in multi-threaded scenarios”. |
Even if you run tests in parallel, reports and logs are collected per test and HTML reporting works as you would expect. |
This is a critical requirement for HTTP integration tests which typically take a much longer time than unit tests. |
38 |
Floating-point precision |
All numbers are converted to float and you shouldn’t forget to use floats (not the default double) in assertions. get("/odd") |
Numeric assertions work just as you expect and even auto-conversion to BigDecimal happens if needed. Given path ‘odd’ |
Even this works: And $.odd.ck == 12.2000000000000 |
39 |
Lines of Code Needed to express a test |
More. By nature, Java is verbose and especially if you depend on POJO representations of payloads - you need more Java code in place. |
Less This particular comparison shows a dramatic difference, 431 lines of code reduced to 67 |
Another example of how Java “gets in the way” - the contortions you need to do to handle JSON arrays in REST-assured. |
40 |
Test Reports Built-in |
No, you have to use JUnit, TestNG or equivalent for test reporting. |
Karate has text and HTML reports out of the box and you get the option of choosing from the Cucumber ecosystem of 3rd party reports. |
Here is an example of the very nice-looking reports you can get by using the cucumber-reporting library. |
41 |
Test any Java servlet or HTTP resource without a container |
REST-assured has support for “out-of-container” testing of specifically Spring-MVC but your tests will be “hard-coded” in this mode. There is no support for things like JAX-RS or custom servlets or controllers - and for these you have to deploy to an app-server. |
Karate v0.5.0 onwards has support for testing any servlet by providing extension points for teams to write an adapter. The huge advantage of Karate’s approach is that the same test-script can be re-used for http-integration tests without changes. |
This is possible because of Karate’s pluggable abstraction of the HTTP Client. Refer to the documentation for more details. You will be able to quickly implement a custom adapter for any Java server-side stack in a similar way. |
42 |
Report includes HTTP request and response logs in-line |
No |
Karate 0.6.0 onwards includes HTTP request and response logs in the JSON report output. If you use the print keyword, the console output appears in the report as well, which is great for troubleshooting. All this works even when tests are run in parallel. |
|
43 |
Construct JSON or XML from scratch using just path expressions |
No |
Karate’s set keyword was enhanced in v0.6.0 to support a ‘builder’ approach using cucumber tables. This is best explained via some examples. |
|
44 |
Test Doubles or Mocks built-in |
No. You have to use 3rd party frameworks such as Wiremock |
Karate 0.7.0 onwards has support for creating API mocks that can fully simulate stateful CRUD. Having both a client and server in the same unified framework keeps things simple and you can move fast. |
Karate is a superior alternative to Wiremock, here’s a comparison. Also see this perf benchmark. |
45 |
Performance Testing |
No. Also see [37] |
Karate 0.8.0 onwards has support for re-using Karate tests as Gatling performance tests. You can compose multiple Karate feature files or “work-flows” into a single performance-test and use Gatling to define the load-model (ramp-up, concurrent users, etc) |
This alone is reason to choose Karate and no other open-source test-automation framework has this option. In 0.9.0 onwards you can test any Java code, not just HTTP. |
46 |
Websockets support |
Karate 0.9.0 onwards has support for websockets and even generic async / await support (see below) |
||
47 |
Async Support |
No, you have to use Java code or a library like Awaitility |
Karate 0.9.0 has built-in support via a very elegant API. |
|
48 |
Retry Support |
No, you have to write custom Java logic. |
Karate 0.9.0 onwards has this built-in - where you only specify the condition to be polled for. |
|
49 |
CSV file support |
No, you have to use Java code or a library. |
In Karate 0.9.0 you can read a CSV file and it will be auto-converted to JSON. |
Karate also has YAML file support out of the box. |
50 |
Web-UI Automation |
No. |
In Karate 0.9.5 onwards Web-UI automation is possible. |
|
51 |
Distributed Testing |
No. |
In Karate 0.9.5 onwards, there is built-in support to split a test-suite and orchestrate the execution across multiple cloud or hardware / cluster nodes |