The ingraph project and incremental evaluation of Cypher queries
The ingraph project and incremental evaluation of Cypher queries Gábor Szárnyas, József Marton
Incremental Queries
Live railway model
Live railway model
Live railway model
Live railway model
Live railway model Proximity detection
Live railway model Proximity detection
Live railway model Proximity detection
Live railway model Proximity detection Trailing the switch
Live railway model
Live railway model
Live railway model
1
a
2
b
c
d
div
e
g
f
Live railway model
1
a
ON
ON
2
NEXT
b
NEXT
c
NEXT
d
NEXT
STRAIGHT
e
g
div
TOP
f
Proximity detection Proximity detection
≤ 𝟏 segments
Proximity detection
ON
≤ 𝟏 segments
t2
t1 ON
Proximity detection
seg1
seg2
NEXT: 1..2
Proximity detection
ON
≤ 𝟏 segments
t2
t1 ON
Proximity detection
seg1
seg2
NEXT: 1..2
MATCH (t1:Train)-[:ON]->(seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) (seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment)
(sw:Switch)−[:STRAIGHT]−>(seg:Segment)
t, seg t, seg, t.number
sw, seg sw, seg, sw.position
Works, but fragile
Nested Relational Algebra (NRA) Additional operators o Nest (𝜈) ~ collect o Unnest (𝜇) ~ UNWIND
Catch: incrementality requires Flat Relational Algebra (FRA)
name John
works year
company
1982 Big Biz, Inc. 2010 Fusion Power Plant, Ltd.
name
works.year
works.company
John
1982
Big Biz, Inc.
John
2010
Fusion Power Plant, Ltd.
Roth, M.A., Korth, H.F. and Silberschatz, A.: Extended algebra and calculus for nested relational databases. ACM Transactions on Database Systems (TODS), 1988 http://dl.acm.org/citation.cfm?id=49347
Property graphs as nested relations Node/relationship properties: id
name 1
John
age 32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
John
age 32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
age
John
32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Map: id
name
1
John
age
32
favColours
beerRatings
id
value
key
0
blue
lager
5
1
green
ale
3
value
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
age
John
32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Map: id
name
1
Paths: […]
John
age
32
favColours
beerRatings
id
value
key
0
blue
lager
5
1
green
ale
3
value
Flattening NRA to FRA It is possible to transform NRA to flat algebra expressions Research questions: o Does it solve the schema calculation problem? o Is it fast enough for practical implementations?
Paredaens, J. and Van Gucht, D.: Converting nested algebra expressions into flat algebra expressions. ACM Transactions on Database Systems (TODS), 1992 http://dl.acm.org/citation.cfm?id=128768
Incremental maintenance of FRA For a change Δ𝑠 on the input define change Δ𝑡 on the output update internal data structures
Maintenance of the antijoin operator
Szárnyas, G., Maginecz, J. and Varró, D.: Evaluation of optimization strategies for incremental graph queries. Periodica Polytechnica EECS, 2017 http://docs.inf.mit.bme.hu/preprints/perpol2016-gqo.pdf
IRE – Incremental Relational Engine Incremental (flat) relational engine built on Akka Independent from Cypher and property graphs
Source code: https://github.com/ftsrg/ingraph/tree/master/ire
OCIM1 revisited Composite data structures o Lists o Maps o Paths
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
o Lists o Maps o Paths
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]}
]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, CIR-2017-220 e2, {k: […]} ]
Current challenges
Update operations Presume a perfectly working incremental query engine How to perform updates? o Low-level API operations: indexer.addTuple() o Adding new nodes: CREATE (…) o Matching and creating: MATCH (n) CREATE (n)-[:REL]->(:Label) o Loading CSVs (legacy construct): LOAD CSV FROM … AS line CREATE (:Label {prop1: toInt(line[2]), …})
Update operations Presume a perfectly working incremental query engine How to perform updates? o Low-level API operations: indexer.addTuple() o Adding new nodes: CREATE (…) o Matching and creating: Not well suited to Rete MATCH (n) CREATE (n)-[:REL]->(:Label) o Loading CSVs (legacy construct): LOAD CSV FROM … AS line CREATE (:Label {prop1: toInt(line[2]), …})
Roadmap Research o Formalise openCypher using Nested Relational Algebra o Transform nested expressions to Flat Relational Algebra
Development o Support for LDBC’s Social Network Benchmark / BI workload o Use TCK for testing o Implement NRA to FRA transformation • See if it works • Run benchmarks
o Use Akka clustering and Docker Compose for deployment o Discover more use cases
Related resources Repository: https://github.com/ftsrg/ingraph Technical report: http://docs.inf.mit.bme.hu/ingraph/pub/opencypher-report.pdf Formalisation (preprint): https://arxiv.org/abs/1705.02844
Incremental Queries
Live railway model
Live railway model
Live railway model
Live railway model
Live railway model Proximity detection
Live railway model Proximity detection
Live railway model Proximity detection
Live railway model Proximity detection Trailing the switch
Live railway model
Live railway model
Live railway model
1
a
2
b
c
d
div
e
g
f
Live railway model
1
a
ON
ON
2
NEXT
b
NEXT
c
NEXT
d
NEXT
STRAIGHT
e
g
div
TOP
f
Proximity detection Proximity detection
≤ 𝟏 segments
Proximity detection
ON
≤ 𝟏 segments
t2
t1 ON
Proximity detection
seg1
seg2
NEXT: 1..2
Proximity detection
ON
≤ 𝟏 segments
t2
t1 ON
Proximity detection
seg1
seg2
NEXT: 1..2
MATCH (t1:Train)-[:ON]->(seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) (seg1:Segment) -[:NEXT*1..2]->(seg2:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment) (seg:Segment)
(sw:Switch)−[:STRAIGHT]−>(seg:Segment)
t, seg t, seg, t.number
sw, seg sw, seg, sw.position
Works, but fragile
Nested Relational Algebra (NRA) Additional operators o Nest (𝜈) ~ collect o Unnest (𝜇) ~ UNWIND
Catch: incrementality requires Flat Relational Algebra (FRA)
name John
works year
company
1982 Big Biz, Inc. 2010 Fusion Power Plant, Ltd.
name
works.year
works.company
John
1982
Big Biz, Inc.
John
2010
Fusion Power Plant, Ltd.
Roth, M.A., Korth, H.F. and Silberschatz, A.: Extended algebra and calculus for nested relational databases. ACM Transactions on Database Systems (TODS), 1988 http://dl.acm.org/citation.cfm?id=49347
Property graphs as nested relations Node/relationship properties: id
name 1
John
age 32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
John
age 32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
age
John
32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Map: id
name
1
John
age
32
favColours
beerRatings
id
value
key
0
blue
lager
5
1
green
ale
3
value
Property graphs as nested relations Node/relationship properties: id
name 1
age
John
32
favColours
beerRatings
[blue, green]
{lager: 5, ale: 3}
favColours
beerRatings
List: id
name 1
age
John
32
id
value
0
blue
1
green
{lager: 5, ale: 3}
Map: id
name
1
Paths: […]
John
age
32
favColours
beerRatings
id
value
key
0
blue
lager
5
1
green
ale
3
value
Flattening NRA to FRA It is possible to transform NRA to flat algebra expressions Research questions: o Does it solve the schema calculation problem? o Is it fast enough for practical implementations?
Paredaens, J. and Van Gucht, D.: Converting nested algebra expressions into flat algebra expressions. ACM Transactions on Database Systems (TODS), 1992 http://dl.acm.org/citation.cfm?id=128768
Incremental maintenance of FRA For a change Δ𝑠 on the input define change Δ𝑡 on the output update internal data structures
Maintenance of the antijoin operator
Szárnyas, G., Maginecz, J. and Varró, D.: Evaluation of optimization strategies for incremental graph queries. Periodica Polytechnica EECS, 2017 http://docs.inf.mit.bme.hu/preprints/perpol2016-gqo.pdf
IRE – Incremental Relational Engine Incremental (flat) relational engine built on Akka Independent from Cypher and property graphs
Source code: https://github.com/ftsrg/ingraph/tree/master/ire
OCIM1 revisited Composite data structures o Lists o Maps o Paths
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
o Lists o Maps o Paths
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]} ]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, e2, {k: […]}
]
OCIM1 revisited Composite data structures
(
o Lists o Maps o Paths
)
Nested data structures [
e1, CIR-2017-220 e2, {k: […]} ]
Current challenges
Update operations Presume a perfectly working incremental query engine How to perform updates? o Low-level API operations: indexer.addTuple() o Adding new nodes: CREATE (…) o Matching and creating: MATCH (n) CREATE (n)-[:REL]->(:Label) o Loading CSVs (legacy construct): LOAD CSV FROM … AS line CREATE (:Label {prop1: toInt(line[2]), …})
Update operations Presume a perfectly working incremental query engine How to perform updates? o Low-level API operations: indexer.addTuple() o Adding new nodes: CREATE (…) o Matching and creating: Not well suited to Rete MATCH (n) CREATE (n)-[:REL]->(:Label) o Loading CSVs (legacy construct): LOAD CSV FROM … AS line CREATE (:Label {prop1: toInt(line[2]), …})
Roadmap Research o Formalise openCypher using Nested Relational Algebra o Transform nested expressions to Flat Relational Algebra
Development o Support for LDBC’s Social Network Benchmark / BI workload o Use TCK for testing o Implement NRA to FRA transformation • See if it works • Run benchmarks
o Use Akka clustering and Docker Compose for deployment o Discover more use cases
Related resources Repository: https://github.com/ftsrg/ingraph Technical report: http://docs.inf.mit.bme.hu/ingraph/pub/opencypher-report.pdf Formalisation (preprint): https://arxiv.org/abs/1705.02844
