JSON+REST API for Topics & Webs via browser Javascript

Error: no such plugin chili

Background

Requirements

By POSTing, PUTting or GETting JSON with Javascript in a web browser to a particular /bin/script or /bin/rest/Handler, Foswiki needs a standard way to -

Required, but the first version can live without

Further reading

See also:

Existing APIs

Check out:

Definition of HTTP Methods

Foswiki considerations

Brainstorming:

-- PaulHarvey - 12 Oct 2010

Discussion

this is a duplicate of the already partially existent RestPlugin? - in which I made QUERY (despite being told it couldn't be done), and even provide mime type to allow requests for json, xml, html, tml....

and it even does GET & PUT....

yes, now it need to re-written to take advantage of foswiki - ie, lots of code to be removed..

-- SvenDowideit - 12 Oct 2010

Ok, re-named this topic to use RestPlugin as the basis for the work.

-- PaulHarvey - 12 Oct 2010

HijaxPlugin already enables much of what you want (if I understand correctly what you want smile using foswiki over REST with returned content clearly defined via JSON). It can also be used as a wrapper around RenderPlugin calls to handle error responses.

-- DavidPatterson - 12 Oct 2010

Just last week I was also wondering why we don't have a full REST plugin, so I started rolling my own methods.

-- ArthurClemens - 12 Oct 2010

David, I dont' think I've reviewed HijaxPlugin since you first checked it in, I'll have another look. Arthur, it would be great if you could share your thoughts on methods that you need.

Obviously we're all going to want to do special handlers - but hopefully Foswiki can ship with an api that can manipulate topics in a standard way over rest for browser javascript. EugenMayer pointed me to drupal entities api which apparently has a rest interface component.

-- PaulHarvey - 13 Oct 2010

It would be good to come to a foswiki remote api also useful to base a soap service on, not only rest. See for example Confluence's remote api specification. Quite feature complete.

-- MichaelDaum - 15 Oct 2010

Updated the brainstorm with what I personally think I need from Foswiki smile The main point is that we have too many ways of doing it (working with topics in AJAX, that is). There aren't many well-documented recipes for a js dev to go to, especially when it comes to saving data back to the wiki.

-- PaulHarvey - 29 Oct 2010

to get the ball rolling on arguing over how to lay out the URI's and responses, I thought I'd toss this at the wall:

Sven's first thoughts for adding REST API

The URI of any element in foswiki needs to be definable

I'm proposing to use a URI like:

http://server.com/foswiki/bin/query.pl/SOMESearchQuery/elementalias[.mediatype]

the [.mediatype] is optional, and can be set as the proper list of accepted types in the request header.

for example:
  1. http://x61/query/Development.FeatureRequest/topic.html - return the WysiwygPlugin rendered version of the topic (this is afterall primarily a programatic interface)
  2. http://x61/query/Development.FeatureRequest/web.json - returns the Development web's object (whatever that is)
  3. http://x61/query/Development.FeatureRequest/info.xml - the topic's META info - date, author, rev
  4. http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor? ~ '*SvenDowideit*'/topic.json?fields(web,topic,Summary,State) -

because each item that is returned needs to be uniquely addressable, I want to define the return as a 'hash of items, where the key is the uri of the item'

eg, the reply for http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor? ~ '*SvenDowideit*'/topic.json?fields(topic,Summary,State) 
{
   'http://x61/query/Tasks/Item1234' : {
                                       'topic': 'Item1234',
                                       'Summary': 'Someone changed the way that SEARCH does something',
                                       'State': 'broken by design'
                                    }
   'http://x61/query/Tasks/Item7901' : {
                                       'topic': 'Item7901',
                                       'Summary': 'Add more icons',
                                       'State': 'Pretty'
                                    }
   'http://x61/query/Tasks/Item8432' : {
                                       'topic': 'Item8432',
                                       'Summary': 'DebugLogPlugin',
                                       'State': 'Strange'
                                    }
}

allowing the client to PATCH to any one of the URI's, or to PATCH the lot in one go.

*******************************FILTERING/PAGING/WHATEVER. can we add webtopic to query, so that we can have paging based on the last web.topic name - and limit(20) ... ala distinct.

HTTP verbs

If you can GET a set of elements in a non-lossy form (tml, json, xml), or a convertable form (ie, html from WysiwygPlugin, rather than normal view) you should (depending on your ACL's) be able to use that addressing, and incomng stream of information to modify that set of elements.

Interestingly, the PUT / POST difference means that AUTOINC00 or 10X can work with a POST asis, but not for PUT as that requires it to map to a unique URI.

If we presume to be like google, we can use PATCH as they do...

GET

retrives a web/topic/attachment resource

PATCH

patches an existing resource, updating only the elements that are sent in the body

POST

creates a resource (by default), or if action=rename renames the resource

DELETE

makes the resource go away, puts it behind the couch

PUT

uri elements

prefix url so the API is versioned

we should really hedge our bets and prefix the query with a 'API version' eg. http://x61/query/v1/Development.FeatureRequest/topic.html

specifying what object to return (this is basically missing in SEARCH, and )

looking at google, they seem to end all requests in /full - feels like a good match for the way the QUERY macro already works (plus webs and topics)
  1. /webs - for a set of 'web objects'
  2. /topics - for a set of 'topic objects'
  3. /revisions - meta data about a requested set of revisions - including URI to request that revision
  4. /attachments - for a set of 'attachment objects' - our meta, including a URI to request the attachment
  5. /form ....
  6. /fields - maybe
  7. /preferences
  8. /info
  9. /parent
  10. /moved
when requesting sets, it becomes necessary for each item in the set to have an address - so we need to either have the set be a hash where the key is the address that can be used to request/update that item, or to add the address into the element. I prefer to use a hash - do we (yet) have any topic meta arrays where we have defined duplicates to be both legal and useable (rather than undefined on update?)

specify subset fields to return

?fields=element(listofitems)

ordering.... and limiting and ....

=?orderby=um, like...

output format

  1. mediatype in the HTTP header (ie, content negotiation)
  2. request parameter contenttype=application/json,text/csv,application/msword (CommandAndCGIScripts has it as a scalar, so mmmm - might drop this)
  3. as an 'filename like extension' eg http://x61/query/web.json could return a set of web objects.
We need to differentiate between rendered text and raw tml markup - I'm guessing the correct MIME type would be text/vnd.tml and text/vnd.rawtml (we need something different for raw=debug...

addressing

Query syntax isn't yet expressive enough, but we should use it and drive its further development

longhand == full query syntax

Yes, a pig wrt encoding and decoding, but if we're building queries with javascript..
  1. http://x61/query/d2n('20 Oct 2010') < info.date AND info.date < d2n('1 Nov 2010') AND SomeField? = 'save'/topic.json - set of topics who's last revision was made between 10/10/10 and 01/11/10 that have a field SomeField? =save
  2. http://x61/query/d2n('20 Oct 2010') < info.date AND info.date < d2n('1 Nov 2010') AND SomeField? = 'save'/topic.json?fields=topic(web, topic, info.rev, info.author, info.date) - subset of the topic - what changed?

specific single elements

attachments
the alias.contenttype suffix might allow us to use server side converters - one day?
  1. http://x61/query/Development/FeatureRequest/attachment.json - returns and allows modification of the topic's attachment meta data -
  2. http://x61/query/Development/FeatureRequest/nameof_png_attachment/attachment.png same as viewfile to the nameof_png_attachment.png - and allows GET&POST
  3. http://x61/query/Development/FeatureRequest/nameof_png_attachment/attachment.json would allow GET&POST, DELETE etc on the attachment's meta data (er, unless we clash with an attachement ending in json, ? TODO - perhaps the specific attachment accessors should contain the source file ext and then the content-type allows attempts at conversion?

short cuts for the longhand addressing

Clearly, it would be best if the current view addressing still worked, so there's less surprise for someone familiar with it.

so the following are identical
  1. http://x61/query/Development.FeatureRequest/topic.html
  2. http://x61/query/Development.FeatureRequest

BUT /topic.html probably should be the WysiwygPlugin version of the topic

more examples

*NOTE: need to change this / resolve the issue wrt nested webs (see below)* Where we know the address
  1. http://x61/query/System/webs.json - =System' web's object
  2. http://x61/query/System/topics.json - all the topics in the System web
    • ok, so this points out a need for paging / iterator / something
  3. http://x61/query/System/topics.json?fields=topic(name) - names of all the topics in the System web
  4. http://x61/query/System/topics.json?fields=topic(name,FIELD.SomeField) - names of all the topics in the System web, paired with a formfield (name, value, and whatever else belongs in the FIELD object..)
  5. http://x61/query/System.ProjectLogos/attachments.html - the attachment table for ProjectLogos

obscure corners

Nested web implied WebHome

Someone once allowed foswiki to have both a topic and a web with the same URL name (the only difference being a trailing slash)

this means that http://x61/Development/Stuff does not equal http://x61/Development/Stuff/

case sensitivty

http://x61/Development/Stuff does not equal http://x61/Development/stuff

though this may change one day

-- SvenDowideit - 01 Nov 2010

Cool, this does look really nifty. It seems like a lot of problems to solve... Okay, some observations:

-- PaulHarvey - 01 Nov 2010

-- SvenDowideit - 02 Nov 2010

Next to existing topic data it would be necessary to store other kinds of (json) data, for example .../topic/put/mylist or .../topic/put/myhash.

These might be stored in the topic meta data, as long as putting and reading is using a verb syntax.

-- ArthurClemens - 02 Nov 2010

This would require PATCH verb to work; PATCH to 'topic'/META:FIELD[name='myhash']

-- PaulHarvey - 03 Nov 2010

I am sure I must be missing something, here, so please bear with me smile

Between TMCE and WysiwygPlugin, there are already ways to read the WysiwygPlugin version of a topic (although we don't use it at present, so I don't know that it still works), to save Wysiwyggy content to a topic, and to convert between HTML and TML. That is four operations, of which three are in regular use (only the first is not). This proposal addresses the first but not the other three. If the first does not work, then I would rather fix it than provide a completely separate interface.

Also: what should these do?
  1. http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor? ~ '*SvenDowideit*'/topic.html
  2. http://x61/query/Development.FeatureRequest/web.html
  3. http://x61/query/Development.FeatureRequest/info.html

-- MichaelTempest - 03 Nov 2010

Relevant work and observations

-- CrawfordCurrie - 03 Nov 2010

first up, Michael, yes, the html view is a duplication of the existing WysiwygPlugin response - it is just one view into the data, and the point of the programatic REST API is to provide one consistent API to GET/SET/PATCH/DELETE using what is currently considered a good resource based style.

it is just going to be an API to the existing code.

  1. http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor? ~ '*SvenDowideit*'/topic.html
    • this one can do with some discussion - imo it would be some variation on the TemplatingSearchResultSummaries - as it sets us up for using the API to request rendered sections - useful for distributing load, or simply defered rendering..
  2. http://x61/query/Development.FeatureRequest/web.html
    • I'm not entirely sure what a 'web' object is, but essentially, a html rendering of one = perhaps giving information like favicon, colour, other WebPreferences.. but using the sectional html view
  3. http://x61/query/Development.FeatureRequest/info.html
    • like attachments.html, this would be a html rendered partial element that could be requested dynamically and pasted into the right place

ok, Crawfords q's

-- SvenDowideit - 03 Nov 2010

I can see the usefulness of http://x61/query/form='ItemTemplate' AND web='Tasks' AND WaitingFor? ~ '*SvenDowideit*'/topic.html as you have explained it, but that interpretation appears to be at odds with "/topic.html probably should be the WysiwygPlugin version of the topic". Either that, or ".html" is magically context-dependent, which might not be a good idea.

Also, CRUD is fine, but WysiwygPlugin has two commonly-used REST interfaces (HTML2TML? and TML2HTML? ) that are not CRUD. Are these excluded from this discussion precisely because they are not CRUD? Is it an oversight? Or have we simply not got there yet? smile

-- MichaelTempest - 04 Nov 2010

Hi Michael, WysiwygPlugin provides a useful translation service - and TinyMCEPlugin gives you a way to to work on a topic's main text. I would hope that one day, my EditorAPI hack which secretly translates html-in-formfields to tml-in-formfields, could actually think about using RestPlugin directly - especially for in-line edits from a view.

-- PaulHarvey - 04 Nov 2010

At its root, the 'real' REST API should provide a single consistent way to GET,SET,MODIFY addressable resources. That means duplicating existing ways into the data (given that the entire thing is a duplication of all the GET&POST param RPC's we currently have.

I'm not sure i agree that /topic.tml in a scalar context (ie, Main.SvenDowideit/topic.html) being a WysiwygPlugin view of the topic is inconsistent with making the list context use a summary list ala WebSearch, but thats the point - i'm not sure.

mind you, you are foccusing strongly on one specific mimetype, when it is largely there for consistency - its pretty odd providing an API that can do HTTP content negotiation, by then arbitarily doesn't provide text/html.

-- SvenDowideit - 05 Nov 2010

This still appears to me as a tangled ball of threads. I am sure that one day, we will get a jersey out of it. In the mean-time, I tugged on the one thread I have a vague understanding about smile .

I have no problem with providing an alternative API. However, the API should then be flexible enough to deal with existing use-cases, or else those use-cases should be excluded from the outset. I asked about the tml<->html conversions because I wasn't sure about the intended scope of the API. Those conversions are performed with reference to an addressable object (a topic) but they don't set, get or modify the addressable object.

There is a use-case for wanting a non-WysiwygPlugin view of a topic in scalar context - browser-side %INCLUDE (e.g. Mystuff.TakesALongTimeToRender/topic.html ). Perhaps the WysiwygPlugin view of a topic should be a different mime-type, or accessed as (say) topic.editable.html

Something else to consider: the WysiwygPlugin view might turn out to be many views. We are already at a place where WysiwygPlugin emits HTML that is intended for editing within TMCE. Before that, it was tailored for Kupu. We are now considering supporting various different editors which may have differing capabilities, which means the WysiwygPlugin conversions may have to be configurable, which means passing parameters. Does that affect this API?

-- MichaelTempest - 05 Nov 2010

We lack a coherent javascript API to do basic CRUD from the browser. So that's the basic goal of this plugin. The idea is that we should be able to also do basic CRUD on any 'TOM' (QuerySearch ?) addressable element of a topic. Given that tables aren't yet covered, your plugin example would not be able to use this.

As for WYSIWG, I share your concerns - but I think it's possibly too early to worry too much just yet. Wysiwyg isn't the primary target for this service just yet, but I hope we could consolidate everything after we get some experience with it. Maybe there will always be a case for WYSIWYG to do its own thing - but I do hope that RestPlugin could take over listing/saving attachments, and most other topic-related operations required on an edit screen.

Because really, getting/setting QuerySearch addressable elements of a topic should be easy to do.

-- PaulHarvey - 06 Nov 2010

Yes, I agree, a basic CRUD API should be the goal. When you are using Rails (or any other REST-based framework) you tend to build apps using CRUD as long as possible, but you often have to flip to providing your own services that don't fit the CRUD model, in the way we did in WysiwygPlugin. So, we need 2 things:
  1. A Javascript CRUD API
  2. A framework for the extension of the API for non-CRUD calls.
BTW my recent work on the EditRowPlugin is pushing the boandaries, as it is addressing elements that are not addressable using the query API (specifically, elements in TML tables). I'm torn between completing that work with a roll-my-own API, or trying to extend the query model to support addressing those elements. I don't have a problem with doing that for REST, but addressing table elements in SEARCH is hairy.....

More thoughts in HowToAddressNonMETAmetadata

-- CrawfordCurrie - 06 Nov 2010

Have a look at backbone.js, a lightweight MVC framework. It includes a RESTful JSON interface connector. I am using this (small, 2K) library for another unrelated project.

The documentation on REST is at http://documentcloud.github.com/backbone/#Sync

-- ArthurClemens - 06 Nov 2010

Discovered Development.AddMetaSetActionToEditAndSave, where it seems many of the same ideas are repeated here smile

-- PaulHarvey - 07 Nov 2010

BIG TODO: both Error handling and authentication (sessions, you name it) are undefined atm, and will need work.

I'm starting implementation adding Foswiki::UI::query (in the plugin) in the hope that this allows us to define the authentication better

-- SvenDowideit - 12 Nov 2010

Also, what about request validation (StrikeOne? )? It should be hidden behind the JS API, IMHO.

-- CrawfordCurrie - 02 Dec 2010

Again, HijaxPlugin with its hijax.js offers error handling, authentication requests and strikeone validation on most of Foswiki's actions via the foswiki.HijaxPlugin.serverAction({...}) interface (a practically 1:1 drop in for jQuery.ajax({...})). This is without having to duplicate anything. As a wrapper around RenderPlugin calls, you can perform any macro call you want with all this protection.

Yes, it's a hack, but until RestPlugin is ready, you can already start building applications on this sort of interface.

Still some bugs in my working version on IE and Chrome where I'm trying to execute the onSubmit js on forms before executing the action over ajax. And upload is missing.

-- DavidPatterson - 02 Dec 2010

given that one way to use the RestPlugin is to use curl: 

curl -X PATCH -H "Content-Type:text/json" -d '{"_text": "set the topic text to something"}' http://x61/f/bin/query/Main/SvenDowideit/topic.json

can be used to send a JSON payload that will change only the topic text of my user topic (ignoring authentication and request validation info)

I like the restriction that the HTTP payload is only used for resource information, and so am working towards having both auth and strikeone info site in the HTTP Header.

yes, there should be a Perl and javascript class to abstract the sending of requests - but curl is a wonderful touchstone for simplicity of resource oriented actions smile

-- SvenDowideit - 02 Dec 2010

Sorry about my previous post, really egocentric. What I meant to say was, Why are you putting effort into another plugin rather than fixing the core rest implementation? You're talking about duplication of the core functions (actions, error handling, authentication etc.), wouldn't it be better to upgrade the rest interface to map to the core functions from within the core? Hijax already maps to the actions and Render already maps to the macros. Perhaps I don't get what you're doing (quite likely), but it seems to me that you're just creating an alternative. Isn't the best solution to fix what's broken rather than implement another workaround, and all of you core devs seem to agree that the rest implementation is broken. Then we wouldn't need the likes of HP or RP.

-- DavidPatterson - 02 Dec 2010

ok, we need to be clearer in our writings. the existing RestHandler? work is not broken, its just mis-named. it should be called RPCHandler - as that is what it implements - a way to call methods that then may or may not affect the resources.

What I'm working on, is a new 'script' query (though I am willing to rename, i don't find myself caring about the name) that is trying to implement a CRUD style REST API - a different view into the data.

Mapping core functions is basically what I'm doing - in that this API gives access to what are essentially methods on Meta - using the HTTP verbs (ie, what REST means)

-- SvenDowideit - 03 Dec 2010
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. see CopyrightStatement. Creative Commons LicenseGet Foswiki at sourceforge.net. Fast, secure and Free Open Source software downloads