DB Replication
DivConq Framework’s CTP
A file and message transfer protocol that transcends the network protocol. • Common Transfer Protocol provides a complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT) • It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket • It does not assume full duplex communications • It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues • It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities • It is open and free
Client
Server CTP Service
App CTP Session Connector
Network Layer
CTP Session
CTP Session provides the desired abstraction by living just outside of the network protocol. A CTP Session represents one authenticated user, it can accept duplex or half-duplex interactions and otherwise doesn’t care about the network layer. The CTP Service processes the commands sent to the session and provides support for the file systems the server exposes.
Client
Server CTP Service
App CTP Session Connector
Network Layer
CTP Session
The CTP Session Connector may be JavaScript in a web page, a Python or Java library, or otherwise. It keeps track of session context for all calls to the server. The session connector may be paired with a full-duplex or a half-duplex networking library. To the App developer the API calls will look pretty much the same either way (with a few exceptions for straight HTTP support).
The past has many examples of file transfer protocols. • FTP – commands and network completely intertwined • SFTP – the file transfer protocol *is* separate from the network protocol, a good model but • Effectively stuck on TCP/IP because of SSH • Assumes full duplex communications • Associates the communications client credentials with the file transfer, which can lead to scalability issues • Can be challenging to debug/trace, especially if custom extensions are present • HTTP POST/GET – commands and network not separated • Existing accelerated transfer protocols – commands and network not separated No existing candidates for a Common Transfer Protocol, but a lot to learn from all of these.
DivConq CTP
Client 1
CTP Session Connector
UDP
App
CTP Session
CTP Service
UDP
Client 2
HTTP
CTP Session Connector
HTTP
App
Server HTTPS Port 443
CTP Session
CTP Session
WSS
UDP Port 2020
Web Socket Port 2022 Client 3
Session Connectors paired with networking protocols. A single server can listen for different networking protocols on different ports – but again CTP Session and CTP Service can be mostly ignorant of the network protocol.
WebSocket App
CTP Session Connector
Client 2
CTP Service
TCP Port 2021
TCP
15 CTP Session Connectors
TCP
15 App Threads
Server
15 CTP Sessions
3 Connections
Full duplex communications allow for much more efficient use of a TCP connection. Traditionally (think FTP or HTTP) we follow a send request then wait for response (half-duplex) approach to issuing commands. If you just send a request and assume a response will come back when it is ready, then you can keep sending more commands and data blocks. Because of this efficiency, and because we do not tie a user to the network connection, we could be servicing 15 “users” (Sessions) on the client on as little as 3 TCP connections.
Server CTP Service
HTTPS 443 (HCTP)
WSS 443 (WCTP)
UDP 2020 (ACTP)
TCP 2021 (SCTP)
Though not limited to only these options, the design of CTP has support the following options within its design: • • • •
HTTP/HTTPS – henceforth called H(yperText)CTP WS/WSS – henceforth called W(ebSocket)CTP [using WS Binary mode] TCP - henceforth called S(tandard)CTP UDP - henceforth called A(ccelerated)CTP
DivConq CTP
Many of the following slides present details and examples of a reference implementation of CTP. As long as the results are the same over the wire, it doesn’t really matter how the CTP server or client is implemented. But understanding a reference implementation will help expose the design ideas.
CTP Connectors
Server UDP (ACTP)
TCP (SCTP) WS (WCTP) HTTP (HCTP)
Common Security
Web Security
Common Handler
Session Handler
HTTP Handler
There are four options for network connectivity, but there are really only three paths through the system. Each message is required to have a session identity by the time it reaches the Session Handler. Typically this means each message contains a session identity, except with HTTP which has the identity in a secure HTTP only cookie.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Common Security the host name provided on initial network connection becomes the default host name, but individual messages may override that name when authenticating a user. Although encryption of these connections resembles SSH, the client key given at connection is not a user key – it is just a way to confirm that the client app is allowed. User keys are provided during user authentication. A single network connection may support multiple users/sessions.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Web Security the host name provided on connection may not be overridden by a message. Encryption of these connections uses SSL, the client key (if any) *is* considered to be the user’s key. There can only be one user and one session per connection. An Authenticate message must still be sent, however, before a Session is ready to use.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Common Handler communications are expected to be full duplex. The server or client may push messages at any time. There is no need to wait for responses. Messages entering Session Handler can be either “send and forget” or “send”, there is no “send and wait”.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With HTTP Handler communications are expected to be half duplex, the server may NOT push messages at all. Caller must wait on responses. Messages entering Session Handler can be “send and forget” or “send and wait”. HTTP Handler sets a SessionId cookie to track the session identity.
DivConq CTP
To the client and server message structure is JSONlike (exact details vary depending on network layer). Actually messages are YAML-like in that data types may be associated with fields, but for practical purposes messages are JSON-like. Messages contain fields. Fields may hold a scalar value (string, number, true, false or null) or an object or an array. Just like JSON. Structures may be nested. Web page scripts may deal with messages “natively” using JSON. But JSON is also easy enough to work with in many programming languages. When not dealing with large binary data the message structure works fine – smallish binary data can be base64 encoded and treated as text.
Example Authentication Message: { Service: "Sessions", Feature: "Manager", Op: "Start", Body: { Credentials: { UserName: "fredsmith", Password: "temp123" } } }
When dealing with large binary data though, base64 is not the way to go. With UPD (ACTP), TCP (SCTP) and Web Sockets (WCTP) we can take advantage of the fact that there are data types and build that awareness into the network layer. The network layer then sees a sort of “binary JSON” or more accurately a “binary YAML”. Web Sockets and JavaScript have binary capability (in HTML5), which is why WCTP is listed above. HTTP, however, is still better off in text (XML/JSON) based calls to the server (XHR) – especially for legacy support. Therefore HTTP will use JSON/XHR to perform all CTP commands, and will open a separate connection for pure upload or download of binary content. Although this exception for HTTP makes the implementation of CTP more complex, it does ensure highest compatibility since any HTTP 1.1 library can access this service. It also avoids some of the biggest mistakes of FTP because there is no “active mode” data transfer and there is no separate port for data transfer – use the same port just a different URL. So proxies and fire walls are not as big an issue here as with FTP.
When making a request message (client) there are some fields that will appear frequently in the following examples. A quick overview: Service – the name of the destination service Feature – a part of the service Op – operation to perform with Feature
RespondTag – only present if you want a response to the message, provides a way for the client to associate a response message with a callback Body – the payload (or parameters) for the feature/operation requested
Typical Fields: { Service: "XXX", Feature: "YYY", Op: "ZZZ", RespondTag: "010A0D0502", Body: { ... } }
DivConq CTP
Example Message as seen by Session Handler:
Server RPC
{
UDP
Service: "Sessions", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn" } }
Common Security
RPC
Common Handler
TCP
Session Handler
Web Socket Web Security HTTP Server
HTTP Handler
When the Message gets to Session Handler an Origin field has been injected by Common Security. If no Domain was provided by the original sender, then that too is injected (based on the default Host for that network connection).
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
RPC
Web Socket
Session Handler
Web Security HTTP Server
HTTP Handler
When the Message gets to Session Handler the Origin, Domain and SslThumb (optional) fields have been injected by Web Security.
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
HTTP Handler
When the Message gets to Session Handler the Origin, Domain and SslThumb (optional) fields have been injected by Web Security. [more next slide]
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC + Web
HTTP Server
HTTP Handler
}
Both browser requests for web pages as well as XHR, require a Session Context. To support both models the session context is extracted by HTTP handler from a secure HTTP [server only] cookie. Server only is used to protect the cookie from cross domain attacks and such.
DivConq Collab
App Server UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
A HTTP Handler
Initial RPC/Web calls cause a Session to be created in Session Handler. Calls with a SessionId cookie will be connected to an existing Session. Although the App Server cannot push messages over HTTP, it can queue messages for an (HTTP originated) Actor of a Session.
Example RPC Request Message:
App Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Credentials: { UserName: "nnnn", Password: "mmmm" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
}
Web Security RPC
HTTP Server
A HTTP Handler
Example of an Initial RPC call. Credentials are optional, if not present then the session will be a guest session. It takes Trust points to create a session, too many attempts will block the client.
Example Response Message:
Stored in Session:
App Server
{
UDP Result: 0,
RPC
// success Body: { Common UserId: "119", UserName:Security "awhite", Common TCP FirstName: "Andy", Handler LastName: "White", Email: "[email protected]", Chronology: "/America/Chicago", Web Locale: "en-US", Socket Session: { Web Id: "nnnn", Security Key: "010A0D0502" } HTTP HTTP Server } Handler }
Session Handler
A
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White", Email: "[email protected]", AuthToken: "010A0D0502", Chronology: "/America/Chicago", Locale: "en-US", Credentials: { UserName: "nnnn", Password: "mmmm" } }
Response Session Cookie: Session: "nnnn_mmmm_010A0D0502"
Client gets back User info (JSON) and Session info (cookie). An Authentication Token as well as the User info are stored in the Session. Anyone with the session Id and Key can now act as this authenticated user (see guest next slide)
Example Response Message:
App Server
{ Result: 0, // success Body: { Chronology: "/America/Chicago", Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
A HTTP Handler
} Response Session Cookie: Session: "nnnn_mmmm_010A0D0502“
Stored in Session:
If a guest session (no credentials), then all we get back is session info. The default (guest) Chronology and Locale are returned.
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", Chronology: "/America/Chicago", Locale: "en-US" }
Example RPC Request Message:
App Server
{ Service: "X", Feature: "Y", Op: "Z", RespondTag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
Session Cookie:
Web Security RPC
HTTP Server
}
A HTTP Handler
Session: "nnnn_mmmm_010A0D0502"
X
Message requesting service X is processed through the session. The session becomes the proxy for the client. User Context is restored from session cookie by a visit to the session state before passing the message on to service X. Service X has context and a destination for response, thanks to the session. When the session gets the response it then sends it back to the client (with HTTP the client is blocking until the response comes).
Example RPC Response Message:
App Server
{ Result: 0, // success Tag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
}
A HTTP Handler
X
Nothing special is returned. Result is non-zero if error. Tag for the RespondTag field sent. Message for any error message. Body for the data of the response.
App Server UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security HTTP Server
A HTTP Handler
ABC
Someone sends a message for session A. A is configured (by HTTP Handler) to queue incoming messages.
Example Request Message:
App Server
{ Service: "Session", Feature: "Mail", Op: "Check", RespondTag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
Session Cookie:
Web Security RPC
HTTP Server
}
A HTTP Handler
Session: "nnnn_mmmm_010A0D0502"
ABC
Client must actively decide to check mail (queue) by issuing a request. The request may be long polling. Messages on the queue are popped and returned in a list.
Example Response Message:
App Server
{ Body: [ { Service: "Desktop", Feature: "App", Op: "Open", Body: ... } ]
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
}
A HTTP Handler
ABC
Each response has a target Service, Feature, Op, Body. FromHub, RespondTo and RespondTag are allowed. ToHub and Tag are removed as those where values associated with the Session.
DivConq CTP
App Server RPC UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
Each request via Common Handler must contain a session context or be requesting a session context.
Example RPC Request Message:
App Server
{
RPC
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Credentials: { UserName: "nnnn", Password: "mmmm" } }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
}
HTTP Handler
Example of an Initial RPC call. Credentials are optional, if not present then the session will be a guest session. It takes Trust points to create a session, too many attempts will (deplete Trust) block the client.
Example Response Message:
RPC
Stored in Session:
App Server
{
UDP Result: 0,
RPC
RPC
// success Tag: "010A0D0502", Common Body: { Security UserId: "119", Common TCP UserName: "awhite", Handler FirstName: "Andy", LastName: "White", Email: "[email protected]", Web Chronology: "/America/Chicago", Socket Locale: "en-US", Session: { Web Security Id: "nnnn", Key: "010A0D0502" HTTP HTTP } Server Handler } }
Session Handler
A
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White", Email: "[email protected]", AuthToken: "010A0D0502", Chronology: "/America/Chicago", Locale: "en-US", Credentials: { UserName: "nnnn", Password: "mmmm" } }
Client gets back User and Session info. Authentication Token and User info are stored in Session. Anyone with the session Id and Key can now act as this authenticated user (see guest next slide)
Example Response Message:
App Server
{
RPC
Result: 0, // success Tag: "010A0D0502", Body: { Chronology: "/America/Chicago", Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" } }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
}
HTTP Handler
If a guest session, then all we get back is session info. The default (guest) Chronology and Locale are returned.
Stored in Session: Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", Chronology: "/America/Chicago", Locale: "en-US" }
Example RPC Request Message:
App Server
{
RPC
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
}
X
User Context is restored from session field before passing the message on to service X. Service X sees a RespondTo and RespondTag corresponding to the HubId and Session_Actor. If the incoming request has a RespondTag, it is added to the new RespondTag as in Session_Actor_Tag. The RespondTag is properly restored to Tag on response.
Example RPC Response Message:
App Server
{
RPC
Result: 0, Tag: "abcd", Body: { ... }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
}
A
Web Security HTTP Server
Session Handler
// success
HTTP Handler
X
Nothing special is returned. Tag is present if a RespondTag was sent. Body for the response payload. Service X does not have to know anything about the client.
App Server RPC UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
ABC
Someone sends a message for session A. A is configured (by Common Handler) to directly route incoming messages over the full duplex network layer. Tag is adjusted to remove the session id, but otherwise the message is fairly consistent with what ABC sent.
DivConq CTP
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "abcd", Body: { Credentials: { UserName: "nnnn", Password: "mmmm", SslThumb: "mmmm" } } }
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "abcd", Body: { Credentials: { UserName: "nnnn", Password: "mmmm", SslThumb: "mmmm" } }
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Credentials: { UserName: "nnnn", Password: "mmmm", SignedKey: "mmmm" } }
} }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Email: "[email protected]", Chronology: "/America/Chicago",
Email: "[email protected]", Chronology: "/America/Chicago",
Email: "[email protected]", Chronology: "/America/Chicago",
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
} } Response Session Cookie: Session: "nnnn_mmmm_010A0D0502"
} }
} }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Body: { ... }
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... }
} Session Cookie: Session: "nnnn_mmmm_010A0D0502" }
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... } }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Result: 0, Tag: "abcd", Body: { ... } }
// success
Result: 0, Tag: "abcd", Body: { ... } }
// success
Result: 0, Tag: "abcd", Body: { ... } }
// success
DivConq RPC
Since the “real” user context/authentication is stored in the session, and because other vital data may be stored with a session, if a session is lost then the client must start a new one. 1) When client makes a call it will fail because the session id cannot be matched (or server is down). 2) The client should raise an event “session lost” or “connectivity lost”. Every part of the client app that relies on session data should clear their flags/states/cache. This should not trigger new RPC’s, wait for “session connected” 3) Try to reconnect and start session 4) Whether 3 works or not, return error to caller because we don’t know if caller relied on session info 5) If 3 worked then trigger “session connected” event.
DivConq CTP
Client App using API
App Server App Service
Web App
RPC Connector
Upload Download
A
Session
One Connector represents one user (set of credentials), one or more connections and one session on server. Connections pooled by browser, just simple SendWait or SendForget support. Uploads and Downloads have their own URLs.
Client App using API
App Server App Service
App
RPC Connector
A
Session
Each Connector represents one user (set of credentials), one connection and one session on server. No connection pool, just simple Send or SendForget support. App Server may issue a “Session Moving” message to inform client to use a new server (e.g. from a server farm).
Client App using API
App Server A
App
B
C
Sessions
App Services
RPC Sessions
Pool
App Server M
N
O
Sessions
Each client Session represents one user (set of credentials) and one session on server. Connections are pooled, message from client session to server session may use any connection to that server. New client session may start on any connected server, but session stays with that server. App Server may issue a “Session Moving” message to inform client to use a new server (e.g. from a server farm).
App Services
App Server
X
Y
Sessions
Z App Services
DivConq CTP
This first example (just one server) may appear overly complex until we review the second example (three servers). CTP is designed to work well within a distributed application environment where some parts may be running on separate servers. But in this first example all parts are running on one server.
App Server RPC
A
Session
Service Z
RPC caller sends service message to service Z asking for a download. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
App Server RPC
A
Session
Service Z
Plan
If Z approves the download, then it creates a Plan which will coordinate all tasks required for the download(s). Plan holds File Store Access Code, download Milestones and all details on sessions participating in download.
App Server RPC
A
Session
Session
Service Z
Plan
B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the RPC client told that transfer was approved and which Plan holds the milestones.
App Server RPC
A
Session
Session
Service Z
Plan
B
Session A registers with Plan, waits for Milestone 1. Meanwhile Session B prepares the download and then registers Milestone 1. This means that the Source (B) is ready.
App Server RPC
A
Session
Session
Service Z
Plan
B
Session A (Destination) calls Session B (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by A, it immediately asks for the next block from B and pushes the current block to the client. A will never have more than two blocks in holding – if client is slow to accept blocks then B waits until at least one is accepted by client before proceeding to the next block from B.
App Server RPC
A
Session
Session
Service Z
Plan
B
B sends “final block” flag with final block of file. A passing that flag on to client when possible (with HTTP download, the response is streamed (chunked) so we just finish the response to indicate done).
App Server A
Session
Session
Service Z
Plan
B
After client has file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq Database
DMZ Server RPC
DMZ Server
A
LAN Firewall
Session
Service Z
File Store Server
RPC caller sends service message to service Z asking for a download. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server
If Z approves the download, then it creates a Plan which will coordinate all tasks required for the download(s). Plan holds File Store Access Code, download Milestones and all details on sessions participating in download.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the RPC client told that transfer was approved and which Plan holds the milestones.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A registers with Plan, waits for Milestone 1. Meanwhile Session B prepares the download and then registers Milestone 1. This means that the Source (B) is ready.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A (Destination) calls Session B (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by A, it immediately asks for the next block from B and pushes the current block to the client. A will never have more than two blocks in holding – if client is slow to accept blocks then B waits until at least one is accepted by client before proceeding to the next block from B.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
B sends “final block” flag with final block of file. A passing that flag on to client when possible (with HTTP download, the response is streamed (chunked) so we just finish the response to indicate done).
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
After client has file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq Database
DMZ Server RPC
DMZ Server
A
LAN Firewall
Session
Service Z
File Store Server
Client sends service Z a message asking for an upload. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server
If Z approves the upload, then it creates a Plan which will coordinate all tasks required for the upload. Plan holds File Store Access Code, upload Milestones and all details on sessions participating in upload.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the client is told that transfer was approved and which Plan holds the milestones.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A registers with Plan, sets Milestone 1. This means that the Source (A) is ready. Meanwhile Session B prepares for the upload, registers with Plan and then waits for Milestone 1.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session B (Destination) calls Session A (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by B, it immediately asks for the next block from S and writes the current block to the store. B will never have more than two blocks in holding – if the store is slow to accept blocks then B waits until at least one is written before proceeding to the next block from A.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
A sends “final block” flag with final block of file. B closes the file after final write.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
After client has finished upload file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq CTP
Client App
App Server A
FT App
RPC A M X Sessions
M X Collab Session
File Store
Pool
To speed up a large file upload, use “Parts”. In this case we’ll transfer three parts at the same time. To transfer parts the requests go to the same server (or same squad/farm) for each session. Only one file name is given for upload, each session transfers a different set of blocks for the file.
DivConq CTP
Client App
App Server A
FT App
Collab Session
File Store
RPC A M X Sessions
Pool
App Server M
Collab Session
Each client Session represents one session on a different server (squad/farm). One file name is given. Partial files are verified independently – client knows that all of part A is intact, all of part M is intact and that all of part X is intact. Also knows that the offsets are correct. It all adds up to one complete file even if that file is in multiple geographies.
File Store
App Server
X
Collab Session
File Store
Common Transfer Protocol can meet its goals. • A complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT) • It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket • It does not assume full duplex communications • It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues • It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities • It is open and free
A file and message transfer protocol that transcends the network protocol. • Common Transfer Protocol provides a complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT) • It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket • It does not assume full duplex communications • It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues • It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities • It is open and free
Client
Server CTP Service
App CTP Session Connector
Network Layer
CTP Session
CTP Session provides the desired abstraction by living just outside of the network protocol. A CTP Session represents one authenticated user, it can accept duplex or half-duplex interactions and otherwise doesn’t care about the network layer. The CTP Service processes the commands sent to the session and provides support for the file systems the server exposes.
Client
Server CTP Service
App CTP Session Connector
Network Layer
CTP Session
The CTP Session Connector may be JavaScript in a web page, a Python or Java library, or otherwise. It keeps track of session context for all calls to the server. The session connector may be paired with a full-duplex or a half-duplex networking library. To the App developer the API calls will look pretty much the same either way (with a few exceptions for straight HTTP support).
The past has many examples of file transfer protocols. • FTP – commands and network completely intertwined • SFTP – the file transfer protocol *is* separate from the network protocol, a good model but • Effectively stuck on TCP/IP because of SSH • Assumes full duplex communications • Associates the communications client credentials with the file transfer, which can lead to scalability issues • Can be challenging to debug/trace, especially if custom extensions are present • HTTP POST/GET – commands and network not separated • Existing accelerated transfer protocols – commands and network not separated No existing candidates for a Common Transfer Protocol, but a lot to learn from all of these.
DivConq CTP
Client 1
CTP Session Connector
UDP
App
CTP Session
CTP Service
UDP
Client 2
HTTP
CTP Session Connector
HTTP
App
Server HTTPS Port 443
CTP Session
CTP Session
WSS
UDP Port 2020
Web Socket Port 2022 Client 3
Session Connectors paired with networking protocols. A single server can listen for different networking protocols on different ports – but again CTP Session and CTP Service can be mostly ignorant of the network protocol.
WebSocket App
CTP Session Connector
Client 2
CTP Service
TCP Port 2021
TCP
15 CTP Session Connectors
TCP
15 App Threads
Server
15 CTP Sessions
3 Connections
Full duplex communications allow for much more efficient use of a TCP connection. Traditionally (think FTP or HTTP) we follow a send request then wait for response (half-duplex) approach to issuing commands. If you just send a request and assume a response will come back when it is ready, then you can keep sending more commands and data blocks. Because of this efficiency, and because we do not tie a user to the network connection, we could be servicing 15 “users” (Sessions) on the client on as little as 3 TCP connections.
Server CTP Service
HTTPS 443 (HCTP)
WSS 443 (WCTP)
UDP 2020 (ACTP)
TCP 2021 (SCTP)
Though not limited to only these options, the design of CTP has support the following options within its design: • • • •
HTTP/HTTPS – henceforth called H(yperText)CTP WS/WSS – henceforth called W(ebSocket)CTP [using WS Binary mode] TCP - henceforth called S(tandard)CTP UDP - henceforth called A(ccelerated)CTP
DivConq CTP
Many of the following slides present details and examples of a reference implementation of CTP. As long as the results are the same over the wire, it doesn’t really matter how the CTP server or client is implemented. But understanding a reference implementation will help expose the design ideas.
CTP Connectors
Server UDP (ACTP)
TCP (SCTP) WS (WCTP) HTTP (HCTP)
Common Security
Web Security
Common Handler
Session Handler
HTTP Handler
There are four options for network connectivity, but there are really only three paths through the system. Each message is required to have a session identity by the time it reaches the Session Handler. Typically this means each message contains a session identity, except with HTTP which has the identity in a secure HTTP only cookie.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Common Security the host name provided on initial network connection becomes the default host name, but individual messages may override that name when authenticating a user. Although encryption of these connections resembles SSH, the client key given at connection is not a user key – it is just a way to confirm that the client app is allowed. User keys are provided during user authentication. A single network connection may support multiple users/sessions.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Web Security the host name provided on connection may not be overridden by a message. Encryption of these connections uses SSL, the client key (if any) *is* considered to be the user’s key. There can only be one user and one session per connection. An Authenticate message must still be sent, however, before a Session is ready to use.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With Common Handler communications are expected to be full duplex. The server or client may push messages at any time. There is no need to wait for responses. Messages entering Session Handler can be either “send and forget” or “send”, there is no “send and wait”.
Server
CTP Connectors UDP Common Security TCP Web Socket HTTP Server
Web Security
Common Handler
Session Handler
HTTP Handler
With HTTP Handler communications are expected to be half duplex, the server may NOT push messages at all. Caller must wait on responses. Messages entering Session Handler can be “send and forget” or “send and wait”. HTTP Handler sets a SessionId cookie to track the session identity.
DivConq CTP
To the client and server message structure is JSONlike (exact details vary depending on network layer). Actually messages are YAML-like in that data types may be associated with fields, but for practical purposes messages are JSON-like. Messages contain fields. Fields may hold a scalar value (string, number, true, false or null) or an object or an array. Just like JSON. Structures may be nested. Web page scripts may deal with messages “natively” using JSON. But JSON is also easy enough to work with in many programming languages. When not dealing with large binary data the message structure works fine – smallish binary data can be base64 encoded and treated as text.
Example Authentication Message: { Service: "Sessions", Feature: "Manager", Op: "Start", Body: { Credentials: { UserName: "fredsmith", Password: "temp123" } } }
When dealing with large binary data though, base64 is not the way to go. With UPD (ACTP), TCP (SCTP) and Web Sockets (WCTP) we can take advantage of the fact that there are data types and build that awareness into the network layer. The network layer then sees a sort of “binary JSON” or more accurately a “binary YAML”. Web Sockets and JavaScript have binary capability (in HTML5), which is why WCTP is listed above. HTTP, however, is still better off in text (XML/JSON) based calls to the server (XHR) – especially for legacy support. Therefore HTTP will use JSON/XHR to perform all CTP commands, and will open a separate connection for pure upload or download of binary content. Although this exception for HTTP makes the implementation of CTP more complex, it does ensure highest compatibility since any HTTP 1.1 library can access this service. It also avoids some of the biggest mistakes of FTP because there is no “active mode” data transfer and there is no separate port for data transfer – use the same port just a different URL. So proxies and fire walls are not as big an issue here as with FTP.
When making a request message (client) there are some fields that will appear frequently in the following examples. A quick overview: Service – the name of the destination service Feature – a part of the service Op – operation to perform with Feature
RespondTag – only present if you want a response to the message, provides a way for the client to associate a response message with a callback Body – the payload (or parameters) for the feature/operation requested
Typical Fields: { Service: "XXX", Feature: "YYY", Op: "ZZZ", RespondTag: "010A0D0502", Body: { ... } }
DivConq CTP
Example Message as seen by Session Handler:
Server RPC
{
UDP
Service: "Sessions", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn" } }
Common Security
RPC
Common Handler
TCP
Session Handler
Web Socket Web Security HTTP Server
HTTP Handler
When the Message gets to Session Handler an Origin field has been injected by Common Security. If no Domain was provided by the original sender, then that too is injected (based on the default Host for that network connection).
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
RPC
Web Socket
Session Handler
Web Security HTTP Server
HTTP Handler
When the Message gets to Session Handler the Origin, Domain and SslThumb (optional) fields have been injected by Web Security.
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
HTTP Handler
When the Message gets to Session Handler the Origin, Domain and SslThumb (optional) fields have been injected by Web Security. [more next slide]
}
Example Message:
Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Domain: "divconq.com", Origin: "rpc:[ipaddress]", Credentials: { UserName: "nnnn", Password: "nnnn", SslThumb: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC + Web
HTTP Server
HTTP Handler
}
Both browser requests for web pages as well as XHR, require a Session Context. To support both models the session context is extracted by HTTP handler from a secure HTTP [server only] cookie. Server only is used to protect the cookie from cross domain attacks and such.
DivConq Collab
App Server UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
A HTTP Handler
Initial RPC/Web calls cause a Session to be created in Session Handler. Calls with a SessionId cookie will be connected to an existing Session. Although the App Server cannot push messages over HTTP, it can queue messages for an (HTTP originated) Actor of a Session.
Example RPC Request Message:
App Server
{ Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "SendWait", Body: { Credentials: { UserName: "nnnn", Password: "mmmm" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
}
Web Security RPC
HTTP Server
A HTTP Handler
Example of an Initial RPC call. Credentials are optional, if not present then the session will be a guest session. It takes Trust points to create a session, too many attempts will block the client.
Example Response Message:
Stored in Session:
App Server
{
UDP Result: 0,
RPC
// success Body: { Common UserId: "119", UserName:Security "awhite", Common TCP FirstName: "Andy", Handler LastName: "White", Email: "[email protected]", Chronology: "/America/Chicago", Web Locale: "en-US", Socket Session: { Web Id: "nnnn", Security Key: "010A0D0502" } HTTP HTTP Server } Handler }
Session Handler
A
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White", Email: "[email protected]", AuthToken: "010A0D0502", Chronology: "/America/Chicago", Locale: "en-US", Credentials: { UserName: "nnnn", Password: "mmmm" } }
Response Session Cookie: Session: "nnnn_mmmm_010A0D0502"
Client gets back User info (JSON) and Session info (cookie). An Authentication Token as well as the User info are stored in the Session. Anyone with the session Id and Key can now act as this authenticated user (see guest next slide)
Example Response Message:
App Server
{ Result: 0, // success Body: { Chronology: "/America/Chicago", Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" } }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
A HTTP Handler
} Response Session Cookie: Session: "nnnn_mmmm_010A0D0502“
Stored in Session:
If a guest session (no credentials), then all we get back is session info. The default (guest) Chronology and Locale are returned.
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", Chronology: "/America/Chicago", Locale: "en-US" }
Example RPC Request Message:
App Server
{ Service: "X", Feature: "Y", Op: "Z", RespondTag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
Session Cookie:
Web Security RPC
HTTP Server
}
A HTTP Handler
Session: "nnnn_mmmm_010A0D0502"
X
Message requesting service X is processed through the session. The session becomes the proxy for the client. User Context is restored from session cookie by a visit to the session state before passing the message on to service X. Service X has context and a destination for response, thanks to the session. When the session gets the response it then sends it back to the client (with HTTP the client is blocking until the response comes).
Example RPC Response Message:
App Server
{ Result: 0, // success Tag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
}
A HTTP Handler
X
Nothing special is returned. Result is non-zero if error. Tag for the RespondTag field sent. Message for any error message. Body for the data of the response.
App Server UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security HTTP Server
A HTTP Handler
ABC
Someone sends a message for session A. A is configured (by HTTP Handler) to queue incoming messages.
Example Request Message:
App Server
{ Service: "Session", Feature: "Mail", Op: "Check", RespondTag: "SendWait", Body: { ... }
UDP Common Security Common Handler
TCP
Session Handler
Web Socket
Session Cookie:
Web Security RPC
HTTP Server
}
A HTTP Handler
Session: "nnnn_mmmm_010A0D0502"
ABC
Client must actively decide to check mail (queue) by issuing a request. The request may be long polling. Messages on the queue are popped and returned in a list.
Example Response Message:
App Server
{ Body: [ { Service: "Desktop", Feature: "App", Op: "Open", Body: ... } ]
UDP Common Security Common Handler
TCP
Session Handler
Web Socket Web Security RPC
HTTP Server
}
A HTTP Handler
ABC
Each response has a target Service, Feature, Op, Body. FromHub, RespondTo and RespondTag are allowed. ToHub and Tag are removed as those where values associated with the Session.
DivConq CTP
App Server RPC UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
Each request via Common Handler must contain a session context or be requesting a session context.
Example RPC Request Message:
App Server
{
RPC
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Credentials: { UserName: "nnnn", Password: "mmmm" } }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
}
HTTP Handler
Example of an Initial RPC call. Credentials are optional, if not present then the session will be a guest session. It takes Trust points to create a session, too many attempts will (deplete Trust) block the client.
Example Response Message:
RPC
Stored in Session:
App Server
{
UDP Result: 0,
RPC
RPC
// success Tag: "010A0D0502", Common Body: { Security UserId: "119", Common TCP UserName: "awhite", Handler FirstName: "Andy", LastName: "White", Email: "[email protected]", Web Chronology: "/America/Chicago", Socket Locale: "en-US", Session: { Web Security Id: "nnnn", Key: "010A0D0502" HTTP HTTP } Server Handler } }
Session Handler
A
Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White", Email: "[email protected]", AuthToken: "010A0D0502", Chronology: "/America/Chicago", Locale: "en-US", Credentials: { UserName: "nnnn", Password: "mmmm" } }
Client gets back User and Session info. Authentication Token and User info are stored in Session. Anyone with the session Id and Key can now act as this authenticated user (see guest next slide)
Example Response Message:
App Server
{
RPC
Result: 0, // success Tag: "010A0D0502", Body: { Chronology: "/America/Chicago", Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" } }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
}
HTTP Handler
If a guest session, then all we get back is session info. The default (guest) Chronology and Locale are returned.
Stored in Session: Context: { Domain: "divconq.com", Origin: "http:[ipaddress]", Chronology: "/America/Chicago", Locale: "en-US" }
Example RPC Request Message:
App Server
{
RPC
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
}
X
User Context is restored from session field before passing the message on to service X. Service X sees a RespondTo and RespondTag corresponding to the HubId and Session_Actor. If the incoming request has a RespondTag, it is added to the new RespondTag as in Session_Actor_Tag. The RespondTag is properly restored to Tag on response.
Example RPC Response Message:
App Server
{
RPC
Result: 0, Tag: "abcd", Body: { ... }
UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
}
A
Web Security HTTP Server
Session Handler
// success
HTTP Handler
X
Nothing special is returned. Tag is present if a RespondTag was sent. Body for the response payload. Service X does not have to know anything about the client.
App Server RPC UDP Common Security
RPC
Common Handler
TCP
RPC
Web Socket
A
Web Security HTTP Server
Session Handler
HTTP Handler
ABC
Someone sends a message for session A. A is configured (by Common Handler) to directly route incoming messages over the full duplex network layer. Tag is adjusted to remove the session id, but otherwise the message is fairly consistent with what ABC sent.
DivConq CTP
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "abcd", Body: { Credentials: { UserName: "nnnn", Password: "mmmm", SslThumb: "mmmm" } } }
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "abcd", Body: { Credentials: { UserName: "nnnn", Password: "mmmm", SslThumb: "mmmm" } }
Service: "Session", Feature: "Manager", Op: "Start", RespondTag: "010A0D0502", Body: { Domain: "divconq.com", Credentials: { UserName: "nnnn", Password: "mmmm", SignedKey: "mmmm" } }
} }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Result: 0, // success Tag: "010A0D0502", Body: { UserId: "119", UserName: "awhite", FirstName: "Andy", LastName: "White",
Email: "[email protected]", Chronology: "/America/Chicago",
Email: "[email protected]", Chronology: "/America/Chicago",
Email: "[email protected]", Chronology: "/America/Chicago",
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
Locale: "en-US", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }
} } Response Session Cookie: Session: "nnnn_mmmm_010A0D0502"
} }
} }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Body: { ... }
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... }
} Session Cookie: Session: "nnnn_mmmm_010A0D0502" }
Service: "X", Feature: "Y", Op: "Z", RespondTag: "abcd", Session: { Id: "nnnn", Actor: "mmmm", Key: "010A0D0502" }, Body: { ... } }
HTTP Path Request Message:
WS Path Request Message:
Common Path Request Message:
{
{
{
Result: 0, Tag: "abcd", Body: { ... } }
// success
Result: 0, Tag: "abcd", Body: { ... } }
// success
Result: 0, Tag: "abcd", Body: { ... } }
// success
DivConq RPC
Since the “real” user context/authentication is stored in the session, and because other vital data may be stored with a session, if a session is lost then the client must start a new one. 1) When client makes a call it will fail because the session id cannot be matched (or server is down). 2) The client should raise an event “session lost” or “connectivity lost”. Every part of the client app that relies on session data should clear their flags/states/cache. This should not trigger new RPC’s, wait for “session connected” 3) Try to reconnect and start session 4) Whether 3 works or not, return error to caller because we don’t know if caller relied on session info 5) If 3 worked then trigger “session connected” event.
DivConq CTP
Client App using API
App Server App Service
Web App
RPC Connector
Upload Download
A
Session
One Connector represents one user (set of credentials), one or more connections and one session on server. Connections pooled by browser, just simple SendWait or SendForget support. Uploads and Downloads have their own URLs.
Client App using API
App Server App Service
App
RPC Connector
A
Session
Each Connector represents one user (set of credentials), one connection and one session on server. No connection pool, just simple Send or SendForget support. App Server may issue a “Session Moving” message to inform client to use a new server (e.g. from a server farm).
Client App using API
App Server A
App
B
C
Sessions
App Services
RPC Sessions
Pool
App Server M
N
O
Sessions
Each client Session represents one user (set of credentials) and one session on server. Connections are pooled, message from client session to server session may use any connection to that server. New client session may start on any connected server, but session stays with that server. App Server may issue a “Session Moving” message to inform client to use a new server (e.g. from a server farm).
App Services
App Server
X
Y
Sessions
Z App Services
DivConq CTP
This first example (just one server) may appear overly complex until we review the second example (three servers). CTP is designed to work well within a distributed application environment where some parts may be running on separate servers. But in this first example all parts are running on one server.
App Server RPC
A
Session
Service Z
RPC caller sends service message to service Z asking for a download. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
App Server RPC
A
Session
Service Z
Plan
If Z approves the download, then it creates a Plan which will coordinate all tasks required for the download(s). Plan holds File Store Access Code, download Milestones and all details on sessions participating in download.
App Server RPC
A
Session
Session
Service Z
Plan
B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the RPC client told that transfer was approved and which Plan holds the milestones.
App Server RPC
A
Session
Session
Service Z
Plan
B
Session A registers with Plan, waits for Milestone 1. Meanwhile Session B prepares the download and then registers Milestone 1. This means that the Source (B) is ready.
App Server RPC
A
Session
Session
Service Z
Plan
B
Session A (Destination) calls Session B (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by A, it immediately asks for the next block from B and pushes the current block to the client. A will never have more than two blocks in holding – if client is slow to accept blocks then B waits until at least one is accepted by client before proceeding to the next block from B.
App Server RPC
A
Session
Session
Service Z
Plan
B
B sends “final block” flag with final block of file. A passing that flag on to client when possible (with HTTP download, the response is streamed (chunked) so we just finish the response to indicate done).
App Server A
Session
Session
Service Z
Plan
B
After client has file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq Database
DMZ Server RPC
DMZ Server
A
LAN Firewall
Session
Service Z
File Store Server
RPC caller sends service message to service Z asking for a download. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server
If Z approves the download, then it creates a Plan which will coordinate all tasks required for the download(s). Plan holds File Store Access Code, download Milestones and all details on sessions participating in download.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the RPC client told that transfer was approved and which Plan holds the milestones.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A registers with Plan, waits for Milestone 1. Meanwhile Session B prepares the download and then registers Milestone 1. This means that the Source (B) is ready.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A (Destination) calls Session B (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by A, it immediately asks for the next block from B and pushes the current block to the client. A will never have more than two blocks in holding – if client is slow to accept blocks then B waits until at least one is accepted by client before proceeding to the next block from B.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
B sends “final block” flag with final block of file. A passing that flag on to client when possible (with HTTP download, the response is streamed (chunked) so we just finish the response to indicate done).
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
After client has file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq Database
DMZ Server RPC
DMZ Server
A
LAN Firewall
Session
Service Z
File Store Server
Client sends service Z a message asking for an upload. Hub and Session Id are included as part of the request so that Z knows where to route the download to.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server
If Z approves the upload, then it creates a Plan which will coordinate all tasks required for the upload. Plan holds File Store Access Code, upload Milestones and all details on sessions participating in upload.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Plan starts a session on the File Store (same server in the example) by passing the FS Access Code. Meanwhile the client is told that transfer was approved and which Plan holds the milestones.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session A registers with Plan, sets Milestone 1. This means that the Source (A) is ready. Meanwhile Session B prepares for the upload, registers with Plan and then waits for Milestone 1.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
Session B (Destination) calls Session A (Source) requesting the next (starting with first) block of the file to transfer. As a block is received by B, it immediately asks for the next block from S and writes the current block to the store. B will never have more than two blocks in holding – if the store is slow to accept blocks then B waits until at least one is written before proceeding to the next block from A.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
A sends “final block” flag with final block of file. B closes the file after final write.
DMZ Server RPC
DMZ Server
A
Plan
LAN Firewall
Session
Service Z
File Store Server Session B
After client has finished upload file session A sets Milestone 2 with the Plan. Plan knows that this means it is done, so it kills session B and itself.
DivConq CTP
Client App
App Server A
FT App
RPC A M X Sessions
M X Collab Session
File Store
Pool
To speed up a large file upload, use “Parts”. In this case we’ll transfer three parts at the same time. To transfer parts the requests go to the same server (or same squad/farm) for each session. Only one file name is given for upload, each session transfers a different set of blocks for the file.
DivConq CTP
Client App
App Server A
FT App
Collab Session
File Store
RPC A M X Sessions
Pool
App Server M
Collab Session
Each client Session represents one session on a different server (squad/farm). One file name is given. Partial files are verified independently – client knows that all of part A is intact, all of part M is intact and that all of part X is intact. Also knows that the offsets are correct. It all adds up to one complete file even if that file is in multiple geographies.
File Store
App Server
X
Collab Session
File Store
Common Transfer Protocol can meet its goals. • A complete spec for messaging commands as well as file transfers (SIGN ON, DIR, GET, PUT) • It is abstracted from network layer, could be implemented in TCP, UDP or through either HTTP or Web Socket • It does not assume full duplex communications • It does not (necessarily) associate network layer client credentials with the file transfer, which can lead to scalability and efficiency issues • It is platform and language neutral • Server could be implemented in Java, C, Python or similarly suitable languages • Client can be implemented in any language with TCP or HTTP capabilities • It is open and free
