Repository navigation
Add RayforceDB support - #212
protocolstardust wants to merge 1 commit into
Conversation
Connect to RayforceDB over its own IPC protocol (wire version 3), with no JDBC driver: handshake with optional credentials, framed messages, run-length/delta decompression, and decoding of every Rayforce type into the same Java shapes kdb results use, so the grid, charts, export and pivot all work unchanged. - New com.timestored.rayforce package: RayIpc client and RayConnection, plus a NativeConnection interface shared with KdbConnection. - Rayforce entry in JdbcTypes with icon, ;; comments and .rfl language. - Server tree built from one Rayfall expression over (env 0), filtering builtins; table previews and value previews over the native protocol. - Generated queries (top/bottom 1000, count, pivot) use select with a take: clause since the take builtin cycles on short tables; pivot is finished client-side with ordered grouping. - Console shows output the query printed server-side; errors carry the Rayforce error code. - Query maximum size limit enforced client-side for Rayforce replies. - AI query prompt for Rayfall. - RayIpcTest drives the client against a fake server speaking the wire format.
|
I only had a quick look and PR looks mostly good. My main question would be, can we use a standard JDBC driver?
The server connection logic is 95% shared between Pulse (Our closed commercial product) and QStudio. Another fairly standard JDBC driver I can add relatively easy to both. To add this custom logic I'll need to go try and do the same to Pulse , which itself has many modifications as it's trying to support a large server deploy rather than one person querying. Because the changes are intermingled with the kdb connection work which is heavily relied upon and has been tweaked over many years, together with issue 4. it massively increases the risk to me. I would need to work with the large companies that have Pulse deployed and notify them, their connection handling may change and to be vigilant. I'm not against a native connection but I think we should be able to justify our decision why we used that. |
| * shipping the values themselves. {@code env} lists builtins too; they and | ||
| * user lambdas both have type {@code ?}, and only formatting tells them apart. | ||
| */ | ||
| public static final String RAYFORCE_LISTING_QUERY = "((fn [e] (table [name type len cols] (list (key e)" |
There was a problem hiding this comment.
@singaraiona Is this listing query stable? i.e. Should it work forever? and/or is there a standard API call to fetch structured data on all objects?
There was a problem hiding this comment.
Agreed, this shouldn't rely on env plus formatting tricks. There's no structured introspection call in Rayforce today, so I'll add a builtin that returns exactly this table (name, type, length, table columns) as a stable API and we'll switch qStudio to it. The current query will keep working in the meantime.
|
|
||
| if(serverConfig.isRayforce()) { | ||
| // No JDBC driver to pool - test the way we will actually query. | ||
| try { |
There was a problem hiding this comment.
Without connection pooling this driver won't work in Pulse:
https://www.timestored.com/pulse/integrations/
We use 90% the same driver code and I've never added a database to one without adding it to the other.
I think adding as is here would break Pulse.
| if(k != null) { | ||
| rs = jdbc.getRS(k, queryTitle); | ||
| } | ||
| // kdb pivots server-side; Rayforce only groups, so pivot here like plain SQL databases. |
There was a problem hiding this comment.
Not a blocker to this PR but
@anton - Maybe something worth adding?
There was a problem hiding this comment.
Actually Rayforce already has a server-side pivot ((pivot table index pivot_col value_col agg_fn), multi-index supported), so this comment is stale. Not a Rayforce change, we just need to wire the Rayforce path in qStudio to use it instead of pivoting client-side. Happy to do that in a follow-up.
|
@singaraiona QStudio (not visible in this repo) uses Testcontainer to spin up databases e.g. for QuestDB: |
|
@ryanhamilton Yes, we'll publish an official image ourselves rather than have a generic one drift from releases. Plan: I'll link it here once it's published. |
Summary
Adds RayforceDB as a server type. Rayforce has no JDBC driver, so qStudio talks its native IPC protocol (wire version 3) the same way it talks to kdb, and decodes replies into the same Java shapes
kx.cproduces, so the result grid, charts, export and pivot work without changes.com.timestored.rayforcepackage:RayIpc(handshake with optional credentials, framing, run-length/delta decompression, decoding of every Rayforce type) andRayConnection(reconnect-once, likeKdbConnection). Both kdb and Rayforce connections now implement a smallNativeConnectioninterface.JdbcTypes.RAYFORCEwith icon,;;line comments, default port 5000 and the existing.rfllanguage.(env 0), filtering builtins. Tables preview as a grid, other globals show their value.selectwith atake:clause, because thetakebuiltin cycles on tables shorter than the count. Pivots are grouped and ordered server-side and finished client-side like other non-PIVOT databases.Verification
RayIpcTestdrives the client against a fake server that speaks the wire format: handshake, auth, version mismatch, compression, typed nulls, tables, errors, console capture.Known limitation
Only the editor query path and the server tree use the native protocol. SQL notebooks, BabelDB
query_dband dashboard charts still go through the JDBC pool, which cannot serve a Rayforce server.