Wim Leers: State of JSON API (July 2018)

Quite a few people in the Drupal community are looking forward to see the JSONAPI module ship with Drupal 8 core.

Because:

• they want to use it on their projects
• the Admin UI&JS Modernization Initiative needs it
• they want to see Drupal 8 ship with a more capable RESTful HTTPAPI
• then Drupal will have a non-NIH (Not Invented Here) API but one that follows a widely used spec
• it enables them to build progressively decoupled components
• …

So where are things at?

Timeline

Let’s start with a high-level timeline:

1. The plan (intent) to move the JSONAPI module into Drupal core was approved by Drupal’s product managers and a framework manager 4 months ago, on March 19, 2018!
2. A core patch was posted on March 29 (issue #2843147). My colleague Gabe and I had already been working full time for a few months at that point to make the JSONAPI modules more stable: several security releases, much test coverage and so on.
3. Some reviews followed, but mostly the issue (#2843147) just sat there. Anybody was free to provide feedback. We encouraged people to review, test and criticize the JSONAPI contrib module. People did: another 1000 sites started using JSONAPI! Rather than commenting on the core issue, they filed issues against the JSONAPI contrib module!
4. Since December 2017, Gabe and I were still working on it full time, and e0ipso whenever his day job/free time allowed. Thanks to the test coverage Gabe and I had been adding, bugs were being fixed much faster than new ones were reported — and more often than not we found (long-existing) bugs before they were reported.
5. Then 1.5 week ago, on June 28, we released JSONAPI 1.22, the final JSONAPI 1.x release. That same day, we branched the 2.x version. More about that below.
6. The next day, on June 29, an updated core patch was posted. All feedback had been addressed!

June 29

I wrote in my comment:

Time to get this going again. Since #55, here’s what happened:

1. Latest release at #55: JSONAPI 1.14
2. Latest release today: JSONAPI 1.22
3. 69 commits: ($ git log --oneline --since "March 30 2018 14:21 CET" | wc -l)
4. Comprehensive test coverage completed (#2953318: Comprehensive JSONAPI integration test coverage phase 4: collections, filtering and sorting + #2953321: Comprehensive JSONAPI integration test coverage phase 5: nested includes and sparse field sets + #2972808: Comprehensive JSONAPI integration test coverage phase 6: POST/PATCH/DELETE of relationships)
5. Getting the test coverage to that point revealed some security vulnerabilities (1.16), and many before it (1.14, 1.10…)
6. Ported many of the core REST improvements in the past 1.5 years to JSONAPI (1.15)
7. Many, many, many bugfixes, and much, much clean-up for future maintainability (1.16, 1.17, 1.18, 1.19, 1.20, 1.21, 1.22)

That’s a lot, isn’t it? :)

But there’s more! All of the above happened on the 8.x-1.x branch. As described in #2952293: Branch next major: version 2, requiring Drupal core >=8.5 (and mentioned in #61), we have many reasons to start a 8.x-2.x branch. (That branch was created months ago, but we kept them identical for months.)
Why wait so long? Because we wanted all >6000 JSONAPI users to be able to gently migrate from JSONAPI 1.x (on Drupal ⇐8.5) to JSONAPI 2.x (on Drupal >=8.5). And what better way to do that than to write comprehensive test coverage, and fixing all known problems that that surfaced? That’s what we’ve been doing the past few months! This massively reduces the risk of adding JSONAPI to Drupal core. We outlined a plan of must-have issues before going into Drupal core: #2931785: The path for JSONAPI to core— and they’re all DONE as of today! Dozens of bugs have been flushed out and fixed before they ever entered core. Important: in the past 6–8 weeks we’ve noticed a steep drop in the number of bug reports and support requests that have been filed against the JSONAPI module!

After having been tasked with maturing core’s RESTAPI, and finding the less-than-great state that was in when Drupal 8 shipped, and having experienced how hard it is to improve it or even just fix bugs, this was a hard requirement for me. I hope it gives core committers the same feeling of relief as it gives me, to see that JSONAPI will on day one be in much better shape.

The other reason why it’s in much better shape, is that the JSONAPI module now has no API surface other than the HTTPAPI! No PHPAPI (its sole API was dropped in the 2.x branch: #2982210: Move EntityToJsonApi service to JSONAPI Extras) at all, only the HTTPAPI as specified by http://jsonapi.org/format/.

TL;DR: JSONAPI in contrib today is more stable, more reliable, more feature-rich than core’s RESTAPI. And it does so while strongly complying with the JSONAPI spec: it’s far less of a Drupalism than core’s RESTAPI.

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

EDIT: P.S.: 668K bytes of the 1.0M of bytes that this patch contains are for test coverage. That’s 2/3rds!

To which e0ipso replied:

So, with pride, and with lots of sweat (no blood and no tears fortunately), @gabesullice, @e0ipso and I present you this massively improved core patch!

So much pride! This was a long journey, that I walked (almost) alone for a couple of years. Then @Wim Leers and @gabesullice joined and carried this to the finish line. Such a beautiful collaboration!

:)

July 9

Then, about 12 hours ago, core release manager xjm and core framework manager effulgentsia posted a comment:

(@effulgentsia and @xjm co-authored this comment.) It’s really awesome to see the progress here on JSONAPI! @xjm and @effulgentsia discussed this with other core committers (@webchick, @Dries, @larowlan, @catch) and with the JSONAPI module maintainers. Based on what we learned in these discussions, we’ve decided to target this issue for an early feature in 8.7 rather than 8.6. Therefore, we will will set it 8.7 in a few days when we branch 8.7. Reviews and comments are still welcome in the meantime, whether in this issue, or as individual issues in the jsonapi issue queue. Feel free to stop reading this comment here, or continue reading if you want to know why it’s being bumped to 8.7. First, we want to give a huge applause for everything that everyone working on the jsonapi contrib module has done. In the last 3-4 months alone (since 8.5.0 was released and #44 was written):

• Over 100 issues in the contrib project have been closed.
• There are currently only 36 open issues, only 7 of which are bug reports.
• Per #62, the remaining bug fixes require breaking backwards compatibility for users of the 1.x module, so a final 1.x release has been released, and new features and BC-breaking bug fixes are now happening in the 2.x branch.
• Also per #62, an amazing amount of test coverage has been written and correspondingly there’s been a drop in new bug reports and support requests getting filed.
• The module is now extremely well-documented, both in the API documentation and in the Drupal.org handbook.

Given all of the above, why not commit #70 to core now, prior to 8.6 alpha? Well,

1. We generally prefer to commit significant new core features early in the release cycle for the minor, rather than toward the end. This means that this month and the next couple are the best time to commit 8.7.x features.
2. To minimize the disruption to contrib, API consumers, and sites of moving a stable module from core to contrib, we’d like to have it as a stable module in 8.7.0, rather than an experimental module in 8.6.0.
3. Per above, we’re not yet done breaking BC. The mentioned spec compliance issues still need more work.
4. While we’re still potentially evolving the API, it’s helpful to continue having the module in contrib for faster iteration and feedback.
5. Since the 2.x branch of JSONAPI was just branched, there are virtually no sites using it yet (only 23 as compared with the 6000 using 1.x). An alpha release of JSONAPI 2.x once we’re ready will give us some quick real-world testing of the final API that we’re targeting for core.
6. As @lauriii pointed out, an additional advantge of allowing a bit more time for API changes is that it allows more time for the Javascript Modernization Initiative, which depends on JSONAPI, to help validate that JSONAPI includes everything we need to have a fully decoupled admin frontend within Drupal core itself. (We wouldn’t block the module addition on the other initiative, but it’s an added bonus given the other reasons to target 8.7.)
7. While the module has reached maturity in contrib, we still need the final reviews and signoffs for the core patch. Given the quality of the contrib module this should go well, but it is a 1 MB patch (with 668K of tests, but that still means 300K+ of code to review.) :) We want to give our review of this code the attention it deserves.

None of the above aside from the last point are hard blockers to adding an experimental module to core. Users who prefer the stability of the 1.x module could continue to use it from contrib, thereby overriding the one in core. However, in the case of jsonapi, I think there’s something odd about telling site builders to experiment with the one in core, but if they want to use it in production, to downgrade to the one in contrib. I think that people who are actually interested in using jsonapi on their sites would be better off going to the contrib project page and making an explicit 1.x or 2.x decision from there. Meanwhile, we see what issues, if any, people run into when upgrading from 1.x to 2.x. When we’re ready to commit it to core, we’ll consider it at least beta stability (rather than alpha). Once again, really fantastic work here.

Next

So there you have it. JSONAPI will not be shipping in Drupal 8.6 this fall.
The primary reason being that it’s preferred for significant new core features to land early in the release cycle, especially ones shipping as stable from the start. This also gives the Admin UI&JS Modernization Initiative more time to actually exercise many parts of JSONAPI’s capabilities, and in doing so validate that it’s sufficiently capable to power it.

For us as JSONAPI module maintainers, it keeps things easier for a little while longer: once it’s in core, it’ll be harder to iterate: more process, slower test runs, commits can only happen by core committers and not by JSONAPI maintainers. Ideally, we’d commit JSONAPI to Drupal core with zero remaining bugs and tasks, with only feature requests being left. Good news: we’re almost there already: most open issues are feature requests!

For you as JSONAPI users, not much changes. Just keep using https://www.drupal.org/project/jsonapi. The 2.x branch introduced some breaking changes to better comply with the JSONAPI spec, and also received a few new small features. But we worked hard to make sure that disruption is minimal (example 123).¹
Use it, try to break it, report bugs. I’m confident you’ll have to try hard to find bugs … and yes, that’s a challenge to y’all!