HTMX SSE Swap Proposal

sse.md

HTMX SSE Improvements Doc

Overview

The current SSE implementation is geared towards triggering events in the DOM, requiring an HTTP request to be sent up to get new content:

  <div hx-sse="connect /event_stream">
    <div hx-get="/chatroom" hx-trigger="sse:chatter">
      ...
    </div>
  </div>

In this case, an SSE event named chatter would trigger a GET to the /chatroom URL.

This is a very limited use of the Server Sent Events API. In particular, there are two aspects of SSE that are ignored:

1. SSE events can have data (not just names) associated with them
2. SSE events do not need to have a name (i.e. they can be pure data-only messages)

I propose the following extensions to the current SSE functionality:

Swapping on Named Events

If an element wants to directly swap the data of a message into its body, I propose the following syntax:

  <div hx-sse="connect /event_stream">
    <div hx-sse="swap on chatter">
      ...
    </div>
  </div>

The swap on indicates that this element wants to listen for chatter messages and swap the body when they occur.

The hx-swap attribute can then be used to determine exactly how the swap is done, as with normal swaps.

Data-only Messages

In the case of a data-only message, the message type is message according to the SSE spec. Thus, the form swap on message should work the same as the named event implementation.

  <div hx-sse="connect /event_stream">
    <div hx-sse="swap on message">
      ...
    </div>
  </div>

Declaring and Using An SSE Endpoint On The Same Element

We should support both connecting and listening for an event (or events) in the same hx-sse declaration, by allowing comma separated values:

  <div hx-sse="connect /event_stream, swap on message">
     ...
  </div>

Summary

These two changes would be relatively modest, but I think would address the major shortcomings of the current SSE behavior without requiring a large code refactor. With these changes SSE is on equal, or perhaps greater footing than WebSockets as a way to achieve dynamic web UI within the declarative model.

Another nice aspect of this proposal is that the features are additive, which means that SSE improvments, which I very much wnat to see implemented, would not be a blocker to an htmx 1.0 release.

Links

1. MDN SSE Article - https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events
2. SSE Spec - https://html.spec.whatwg.org/multipage/server-sent-events.html
3. Initial SSE improvement PR - bigskysoftware/htmx#155
4. Second SSE improvement PR - bigskysoftware/htmx#185

I've just committed an update to this branch. I think it meets the spec provided by @1cg. I'm trying to avoid cluttering the HTMX repository with PRs, so just let me know when I should post it into the HTMX repo.

All of the changes from the previous commit are in the processSwap function. It removes @tomberek 's doSwap function in favor of inlining code from the WebSockets implementation. These changes are consolidated into the commented section beginning on line 846. The comments are just placeholders for now, and should probably be removed once this branch lands on dev.

I'm still concerned that commas in URLs could break this in ways that are difficult to troubleshoot. Also, (though less important for now) is that this code doesn't implement some other features that are common in other parts of HTMX -- specifically hx-settle, which would be very useful for highlighting new content that is added into the DOM. Is there an easy way to just abstract hx-settle into something we could call from here?

awesome @benpate

Can you close out the other PRs and create a new one for this?

On the commas, the problem shouldn't be on the server side, right? If you use URI encoding it would be on the client only, the server should still see commas. It looks like commas are "reserved" and should probably be avoided:

https://stackoverflow.com/questions/198606/can-i-use-commas-in-a-url

You can leave the comments in and I'll try to extract the swap code out and get it in place for this change once I have it.

Done My apologies for the sloppy commit, I keep making updates on top of the wrong branch. I believe this PR should be correct

Regarding the comma, you should avoid using it as a delimiter before you encounter the space character (or the double quote, that can be safely percent-encoded)

From https://tools.ietf.org/html/rfc3986#section-2.2, "Percent-encoding a reserved character, or decoding a percent-encoded octet that corresponds to a reserved character, will change how the URI is interpreted by most applications. Thus, characters in the reserved set are protected from normalization and are therefore safe to be used by scheme-specific and producer-specific algorithms for delimiting data subcomponents within a URI".

You should not assume that a user of your library can simply percent-encode reserved characters, such as the comma. Based upon the URI specification, reserved characters are distinguished from their percent-encoded variant. If you percent-encode a reserved character, you could be changing the meaning of the URI. For example, a search API may choose to treat , in the value part of a key-value selector as an "OR", and if an actual comma is to be matched, it could be percent encoded. In this case, "?arg=foo,bar" is not the same thing as `?arg=foo%2cbar". In my presumed example, the former means "arg is either foo or bar", in the latter, it means "arg is foo,bar" -- quite different meaning. There are search APIs in the wild that have this exact treatment of the comma in a URL. Hence, neither server nor browser infrastructure should automatically percent-encode a reserved character. Equivalently, proxies should treat a percent-encoded character as being different for caching or other purposes. For our purposes, we should not assume that percent-encoding the comma is a transparent operation, it does create a different URI.

thanks @clarkevans

let me think about it more

we use commas in other places as well, so it'd be annoying to not use them here, and I don't want to have to implement a real lexer... but I'm sure we can figure something out.

we could go full hyperscript and split on and :)

connect https://blah.blah?whatever and swap on message

Sure. I think and is fine. Alternatively, one could use double quotes, but this requires you use single quotes for the attribute... <div hx-swap='connect "my-url-with-commas" swap Event1,Event2' /> -- it's ugly. "connect <https://blah.blah?whatever,comma>; swap on message" works, but, frankly, I like and ...

Hey all.. I agree on NOT making a full Lexer. There was yet another problem with PR #193, so I'm putting together a correct one with the existing syntax while we consider and discuss commas. It will be easy to switch it out for whatever final form is decided.

If we're voting, I still like hx-swap="connect my-url-with-commas swap:eventname" best, because it seems consistent with the way that other tags already work (like hx-swap)

I do want to thank you both for working though this. I don't expect to use htmx for a few more months, but it's really encouraging to see the thoughtful work going into it. So, I'm OK with "swap:eventname" as well.

Apparently, I may have figured out how to make a clean PR, because I just posted PR#194 and it hasn't blown up, yet. Please check my work to see if it looks good?

What about:

hx-sse="connect:https://my-url-with-commas swap:eventname"

so no whitespace between the connect and the URL. That would allow us to split on whitespace.

We could update the websockets to be similar.

Sure, verb:args works. Do your verbs, connect and swap always have a known number of arguments? In which case, "connect https://... swap eventname" work fine. You split on space. Then process the stack from the front. connect would eat the next item on the stack, https://my-url-with-commas and return; swap could then be triggered next, which could eat eventname, etc. There's also nothing wrong with and, this provides you the ability to have variable number of verb arguments (provided they arn't the and keyword).

@1cg - I like the syntax you’re suggesting. Although the double colon looks a little odd, in most cases a page would be calling back to a relative path on its own server, so it’s not an issue. We’d need to be a little careful with how we parse double colons (so we don’t drop content after the (https:”), but there are already examples of this in the library, so it should be simple enough to match.

I’ll let you think on it for a bit. Just say “go” and I’ll update this PR.

@clarkevans - yes. At this time, all of the keywords have only one argument. Think of them more as named parameters, or key/value pairs. We should be able to maintain this convention, if we just pick a new keyword for any additional arguments that we add in the future.

I have pushed up changes so that we now use whitespace as the declaration separator and the first colon as the argument delimiter:

<div hx-sse="connect:/foo swap:bar">...</div>

Same for both SSE and web sockets.

I also looked at combining the swap code for at least web sockets and SSE, but they acted differently enough that I didn't want to do it at this point. Web Sockets are a pure out of band situation, since there isn't a way to signal which messages you are interested in, so it just acts differently. And when I looked at pulling out the AJAX swap, there was a large number of parameters required.

I'll take another look at it when I have a bit more time to think.