on-demand - Semantic Scholar
Dynamic Cloud Resource Reservation via Cloud Brokerage
Wei Wang*, Di Niu+, Baochun Li*, Ben Liang* * Department of Electrical and Computer Engineering, University of Toronto + Department of Electrical and Computer Engineering, University of Alberta July 10, 2013 Wednesday, 7 August, 13
Growing Cloud Computing Costs Drastic increase in enterprise spending on Infrastructure-as-aService (IaaS) clouds 41.7% annual growth rate by 2016 [CloudTimes’12] IaaS cloud will be the fastest-growing segment among all cloud services
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
2
Auto Scaling
On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments. This frees you from the costs and complexities of planning, purchasing, and maintaining hardware and transforms what are commonly large fixed costs into much smaller variable costs.
Tradeoffs in Cloud Pricing Options Elastic Load Balancing High Performance Computing
The pricing below includes the cost to run private and public AMIs on the specified operating system (“Windows Usage” prices apply to Windows Server® 2003 R2, 2008 and 2008 R2). Amazon also provides you with additional instances
VM Import/Export
for Amazon EC2 running Microsoft Windows with SQL Server, Amazon EC2 running SUSE Linux Enterprise Server, Amazon EC2 running Red Hat Enterprise Linux and Amazon EC2 running IBM that are priced differently.
On-demand instances Related Resources
No commitment
Region:
US East (N. Virginia)
AWS Management Console Documentation
Pay-as-you-go
Linux/UNIX Usage
Windows Usage
Standard On-Demand Instances
Release Notes
Small (Default)
$0.080 per Hour
$0.115 per Hour
Developer Tools
Medium
$0.160 per Hour
$0.230 per Hour
Large
$0.320 per Hour
$0.460 per Hour
Extra Large
$0.640 per Hour
$0.920 per Hour
$0.020 per Hour
$0.020 per Hour
Sample Code & Libraries Developer Tools
Micro On-Demand Instances Articles & Tutorials
Reserved instances Amazon Machine Images (AMIs)
Micro High-Memory On-Demand Instances
Public Data Sets on AWS
Extra Large
$0.450 per Hour
$0.570 per Hour
Community Forum
Double Extra Large
$0.900 per Hour
$1.140 per Hour
Quadruple Extra Large
$1.800 per Hour
$2.280 per Hour
Reservation fee + discounted price Suitable for long-term usage commitment http://aws.amazon.com/ec2/pricing/
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Page 1 of 9
3
On-demand v.s. Reservation Pros
Cons
1. Flexible Expensive for longOn-demand 2. Fits sporadic workload term usage
Cost efficient for longReservation term usage
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
1. Long-term usage commitment 2. Expensive for sporadic workload
4
User’s Problem Hard to choose among different pricing options Lacks sufficient expertise
Cost savings due to the reservation option are not always possible Depends on the user’s own demand pattern Must be long-term and heavy usage
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
5
Can we go beyond the limitation of demand pattern of a single user and lower the cost?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
A Cloud Brokerage Service A cloud broker reserves a large pool of instances Users purchase instances from the broker in an “on-demand” fashion User
Broker
Providers
User cost
User ...
Broker cost
"On-demand" Instances
...
Reserved/On-demand Instances IaaS Cloud
User
Figure 1: The proposed cloud broker. Solid arrows show the direction of instance provisioning; dashed arrows show the direction of money flow. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Billing cycle (an instance-hour)
7
Why cloud broker?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Better Exploiting Reservation Options Statistical multiplexing increases the utilization of reserved instances Aggregating all users’ demands smoothes out the “bursts” A flat demand curve is more friendly to reserved instances The “true cost” of reserved instance is reduced due to the increased instance utilization
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
9
cloud
"On-demand" Instances
.
Reserved/On-demand Instances IaaS Cloud
Broker User ReducingProviders Wasted Cost Due to Partial Usage ...
roker Broker cost User cost hould User Alleviate the pricing inefficiency of on-demand instances mand, Figure 1: The proposed cloud broker. Solid arrows show the ically Partial usage is counted as a full billing cycle direction of instance provisioning; dashed arrows show the prob- The broker can time-multiplex partial usage direction of money flow. data, c proBilling cycle (an instance-hour) ng is User 1 User 1 Instance 1 nt apWithout broker User 2 Instance 2 e cost rithm With broker User 1 User 2 User 1 ess to Figure 2: The broker can time-multiplex partial usage from different users in the same instance-hour. In this case, serving two users only takes one instance-hour, instead of two.
GB of users’ te the y the discounted hourly rate charged over the entire reservation ur Wei reWang, Department of Electrical and Computer Engineering, University of Toronto period, no matter whether it is used or not. EC2 also ofusers Wednesday, 7 August, 13
10
Enjoying Volume Discounts Most IaaS clouds offer significant volume discounts Amazon provides 20% or even higher volume discounts in EC2 The sheer volume of the aggregated demand makes cloud broker easily qualify for such discounts
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
11
A Win-Win Solution Users receive a lower price when trading with the broker No upfront payment for reservation No money wasted on idled reservation instances
Broker makes profit by leveraging the wholesale (reservation) model A significant price gap between on-demand and reserved instances Aggregate demand is more amenable to the reservation option
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
12
How many instances should a broker reserve?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
estimate its future demand to decide how many instances reserve at a Reserved particular time. In the case where users are On-demand and Pricing able to estimate demand at all, we propose an online reser strategy in Sec. 5.3 to make decisions based on histo On-demandtion instances Suppose cloud users submit to the broker their dema Fixed hourly rate p estimates up to time T into the future (in terms of bill Reserved instances cycles). The broker aggregate all the demands. Suppos requires dfee: t instances in total to accommodate all the reque Upfront reservation at time t, t = 1, 2, . . . , T . The broker makes a decision Reservation period: ⌧ reserve rt instances at time t, with rt 0. Each reserv rt Instances reserved time instance at will be t:effective from t to t + ⌧ 1, with ⌧ be theinstances reservation period. # of reserved that are effective at time t At time t, the number of reserved instances that rem Pt effective is nt = i=t ⌧ +1 ri . Note that these nt reserv instances may not be sufficient to accommodate the agg gate demand dt . The broker thus needs to launch (dt nt additional on-demand instances at time t, where X + max{0, X}. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
14
thus to launch (dtat all, nt )we propose the second element is 2 because tw le toneeds estimate demand an online reservainstances may not +be sufficient to accommodate the aggreper nces at time t, where X := 4.1 no later than stage 2 and remain effe + on strategy in Sec. 5.3 to make decisions based on history. Dynamic Resource Reservation gate demand dt . The broker thus needs to launch (dt nt ) the We note that given state s := + t 1 Suppose cloud users submit to the broker their demand We additional on-demand instances at time t, where X := nos me reservation fee for each rethe previous stage, the current state max{0, X}. timates up to time T into the future (in terms of billing lem (2)W Cloud users submit demand predictions to the broker te the price of running an onindependent sthe 2, A ts Let denote the one-time reservation fee of forthe each re-stcle. cles). The broker aggregate all the is demands. Suppose itstates Broker reserves instances based on the aggregate demand cycle. Hence, the total cost to characterized by s , with state tra t 1 served instance, and p denote the price of running an onis i quires dt instances in total to accommodate all the requests where ds d1 , . . demand . , dT is instance per billing cycle. Hence, the total cost to cha time t, t = 1, 2, . . . , T . The broker makes axdecision to no late i= i = yi+1 + x⌧ 1 , accommodate all the demands d , . . . , d is 1 T rtcost instances at time t, with rt 0. Eachcost reserved Pserve 1, . . . , T Total = +Reservation cost + On-demand (d n ) p , (1) To see the rationale behind (3), let t t t=1 stance will be effective 1, with ⌧ being PT from t toPtT+ ⌧ cause n + rt + t=1 (dt 4. n p , t + ⌧ (1) Fig. At 1, there are To t ) stage t=1 e reservation period. effectiv Figr tal cost of reservations and the remain effective. Clearly, all these where, At time t, the number of reserved instances that remain x1 · P where the first term is the totalstage cost of reservations and the rem t demand instances. The broker’s t (because the reservations m fective is nt = i=t ⌧ +1 ri . Note that these nt reserved reserve second is the cost instances. reservation decisions r1 , . of . . ,all rTon-demand all expired at tThe + ⌧broker’s 1), i.e., rsta t stances problem may not be sufficient to accommodate the aggre# of reserved that are decisions effectiver1at, + is to makeinstances dynamic reservation . t. . , rT period all te demand dt . The broker thus needs to launch (dt nt ) the sec ditional on-demand instances at3 time t, where X + := no late 3 ax{0, X}. We n Let denote the one-time reservation fee for each rethe pre rved instance, and p denote the price of running an onis indep Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
15
Fig. 4. A max{0, We no hased a X}. st term is the total cost of reservations and the remain ef Stage rovides The Cost Minimization Problem Let denote the one-time reservation fee for each rethe prev cost allFigure on-demand instances. broker’s t (b eerved to theof 3: State illustration. The The reservation period is ⌧ =stage 4.is indepe instance, and p denote the price of running an onbroker All four reservations are highlighted as, total the shaded area. all Atcharacte make reservation decisions . . . , r expire demand instance per billing cycle. Hence, rthe cost to Makedynamic dynamic reservation decisions to 1 T 1
reduces
2
3
4
5
6
7
8
9
10
stage 2, s2 = (2, 2, 2, 1).
ccommodate all the demands d1 , . . . , dT is accommodate demands
okerage PT cost, i.e., + toP minimize its total T 3 r + (d n ) p , (1) To see th s at the t t t t=1 t=1PT PT + min cost = r + (d n ) p . (2)Fig. 4. A t king int=1 t t=1 t {r1 ,...,rT } nd datathe first term is the total cost of reservations and the where remain e Problem (2) is integer programming. In general, complex llowing econd is the cost of all on-demand instances. The broker’s stage t ( methods are needed to solve it. This is ancombinatorial integer program!
problem is to make dynamic reservation decisions r1 , . . . , rT
TION
4.
DYNAMIC PROGRAMMING: THE OPTIMALITY AND LIMITATIONS 3
In this section, we resort to dynamic programming to charnstance acterize the optimal solution to problem (2). Using a set an obof recursive Bellman equations, the original combinatorial users to optimization problem can be decomposed into a number of , based Wang, Department subproblems, of Electrical and Computer Engineering, of Toronto can be solved efficiently. Howeach University of which oteWeithat ever, we also point out that computing such a dynamic proWednesday, 7 August, 13
all expir
16
Optimal Solution: Dynamic Programming
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
ic1 The Programming FormulationFormulation Dynamic Programming Curse of Dimensionality
fining stages and states. The and decision probWe start by defining stages states. The decision probHigh dimensional dynamic programming of T stages, each representing a billing cy- a billing cym (2) consists of T stages, each representing geAt is denoted by at ⌧is-tuple st := ), x1 , . . . , x⌧ High dimensional state: e. state at stage denoted by (t, a ⌧ x-tuple 1 , . . . s, tx:= ⌧ 1(t, s the xnumber of the instances that reserved here number ofare instances that aret and reserved # of instances that are reserved no later than remain i :denotes effective t+i remain remain at stage t + i, for i = t + i, for i = ondlater thaneffective tatand effective at stage ere, ⌧ -tuple to represent a state be. .Exponential . we , ⌧ use1.a Here, we use a ⌧ -tuple to represent a state betime and space complexity euse reserved beforereserved or at stage t will no instance before or remain at stage t will remain The curse of dimensionality age t + after ⌧ 1. Andt + it is⌧ easy check fective stage 1. to And it isthat easy to check that . For example, in Fig. 3, three instances are 1 · · · x . For example, in Fig. 3, three instances are 1 ⌧ 1 1, 2, and 3, respectively, with a reservation served at stage 1, 2, and 3, respectively, with a reservation e see ⌧that 2, sthat (2, 2, 2,2, 1),s2where 2 =at eriod = at 4. stage We see stage = (2, 2, 2, 1), where ent is 2 because e second elementtwo is 2instances becauseare tworeserved instances are reserved 2 and remain at stage 3. ogelater than stageeffective 2 and remain effective at stage 3. given state s := (t 1, y , . . . , y ) at t 1 1 ⌧ 1 We note that given state st 1 := (t 1, y1 , . . . , y⌧ 1 ) at ge, the current statethe st current := (t, xstate ) x ,...,x 1 , . . .s, x:= ⌧ 1(t, e previous stage, t 1 ⌧ 1) Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
1 ),
18
Approximate Solution
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
A 2-Competitive Heuristic Segment the demand into intervals each spanning one reservation period
Demand
Interval
⌧
2⌧
3⌧
4⌧
Time
Make optimal instance reservation decisions per interval
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
20
Optimal Instance Reservation within an Interval Stratify demand into levels For each level, decide if a reserved instance should be used Example
Should reserve when instance usage >= 3 hours
On-demand rate: $1 per hour 6 5 4 3 2 1 0
6
L5 L4
1
2
3
5
6
Interval 1
5 4 3 2 1 0
1
2
3
4
5
6
Time (hour)
Time (hou
(a) T = 5, ⌧ = 6.
(b) T = 8, ⌧ =
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
4
L3 L2 L1 7
Demand
Demand
Reservation: $2.5 for 6 hours
21
Cost Performance Per-interval reservation is 2-competitive Incurs at most twice the optimal cost in the worst case
Demand
Interval
⌧
2⌧
3⌧
4⌧
Time
All reservations are made at the beginning of the interval
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
22
the demand at time t in level l 1. We then step to level , l 1 and make optimal reservations there, taking into account the “leftover” reserved instances passed over from the he last inupper level. Undoubtedly, the algorithm becomes more effi1)). Equacient, since each level triesintervals to utilize such “leftover” reserved Do not segment demand into intervalinstances from upper levels. Note that this is not possible y. Stratify into levels for a bottom-up approach, where no “leftover” reserved inthm 1 in- demands stances can be passed from a lower level up. strategies, L6 reservations In each level of the above procedure, optimal 1. By (8), L5 can be efficiently made via dynamic programming. Suppose incurs no L4 L3 before we make reservations in level l, mlt reserved instances L2 L1 of which can are passed over from upper levels at time t, all rth noting be utilized. Let Vl (t) be the minimum costTime of serving deMake reservations top-down estimates mands dl1 , . . . , dlt in level l up to time t . The Bellman equaup its contion isapply givendynamic by At each level, programming by AlgoVl (t) = min{Vl (t ⌧ ) + , Vl (t 1) + cl (t)}. (10) each state o be whichthan chooses the minimumReservation, between two options. first Strictly better Per-Interval and isThe also 2l (9) is to serve the demand d t with an instance reserved in the competitive Demand
An Improved Greedy Algorithm
6
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
23
When demand predictions are unavailable
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Online Algorithm
# Instances
The best that we can do [Wang et al. ICAC’13]
Algorithm 2 summarizes the greedy reservation strategy ¯ ) and space complexabove, which has time complexity O( dT 2-competitiveness for the deterministic online algorithm ity O(T ). As has been analyzed earlier in this subsection, in each level, Algorithm 2 is more cost efficient than Algorithm 1, leading to the following proposition: Proposition 2: Algorithm 2 incurs a cost no more than Algorithm 1, and is thus 2-competitive.
600 400 200 0 0
# Instances (k)
Algorithm 3 Online Reservation Made at Time t 1. Let gi = (di ni )+ for all i = t ⌧ + 1, . . . , t. 2. Run Algorithm 1 with gt ⌧ +1 , . . . , gt as the input demands. Let x be its output. 3. Reserve rt = x instances at time t. 4. Update ni = ni + rt for all i = t ⌧ + 1, . . . , t + ⌧ 1.
# Instances
Make instance reservation decisions without future information
1.5 1 0.5 0 0
5
600 400 200 0 0
Fig. 6.
C. An Online Reservation Strategy
5
The d
250
d Std
Previous algorithms apply to users who predict their future demands. For those who cannot, we propose an online strategy Wei Wang, Department of Electrical and Computer Engineering, University of Toronto that reserves instances based only on history, without accessing Wednesday, 7 August, 13 to any future information. Recall that n is the number of
5
200 High 150
y = 5x 25
M
Trace-Driven Simulations
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
# Instanc
1
0.5 (11) Dataset and Preprocessing 0 0
50
150
# Instances
and instance 600 traces noGoogle reserved cluster-usage 400 t, i.e., mlt =
100 Time (hour)
200 User 187
900+ users’ usage traces on a 12K-node Google datacenter 200 We convert users’ computing demand data to IaaS instance demand 0 (12)
0
50
100 Time (hour)
150
200
Users are classified into 3 groups based on demand fluctuation level
ervation stratce complexrogramming ) space. As each level, m 1, leading
Demand Std
, we update Figure 7: The demand curves of three typical users. Standard deviation vs. mean in hourly demand ssed to level n instance is 250 := mlt 1, 200 High y = 5x y=x passed over 150
Medium
100 50 0 0
Low 50
100 150 200 Demand Mean
250
300
Figure 8: Demand statistics and the division of users into 3 Wei Wang, Department of Electrical and Computer Engineering, University of Toronto groups according to demand fluctuation level. Wednesday, 7 August, 13
27
and instance no reserved t, i.e.,Relatively mlt =
stable (12)
600
User 863
400 200 0 0
# Instances (k)
,
Medium (11) fluctuation
# Instances
ly serve deance at Highly time The fluctuated second 1 and serve with a cost
# Instances
Demand Curve
50
100 Time (hour)
150
1.5
200 User 48
1 0.5 0 0
50
100 Time (hour)
150
600
200 User 187
400 200 0 0
50
100 Time (hour)
150
200
l, we update Figure 7: The demand curves of three typical users. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto ssed to level Wednesday, 7 August, 13
28
Aggregation Smoothes Out Demand Bursts 200
90 60 30 0 0
y = 1.774x 5
250
150 100 50 0 0
10 15 20 25 30 Demand Mean
(a) Group 1: high fluctuation
300 Demand Std
120
Demand Std
Demand Std
150
y = 0.363x 20 40 60 80 Demand Mean
200 150 100 50
100
(b) Group 2: medium fluctuation
0 0
y
200 400 60 Demand M
(c) Group 3: low fl
200
Demand Std
Demand Std
Figure 300 9: Aggregation suppresses the 300 demand fluctuation of individual users. Ea the250 demand fluctuation level (the ratio 250 between the demand standard deviation an 200
each reservatio Before aggregation discount of 50 100 After aggregation y = 0.058x y = 0.061x y = 0.363x on-demand ins 50 −23.4% general pricing 0 600 0 0 40 60 80 100 0 200 400 600 800 1000 0 200 400 600 800 1000 Demand Mean Demand Mean Demand Mean Group Divi 400 −5.6%(d) All the users. tics of users, w 2: medium fluctuation (c) Group 3: low fluctuation −30.5% deviation for e 200 mand fluctuation of individual users.−16.5% Each circle represents a user. The line indicatesbeen mentione Wei Wang, Department of Electrical and Computer Engineering, University of Toronto 29 0 tween the demand standard deviation and mean) in the aggregate demand curve. vations critical High Medium Low All Wednesday, 7 August, 13 Wasted instance−hours (k)
150 1000 100 800 50
150
(a) Group 1: high fluctuation
(b) Group 2: medium fluctuation
The Reduction of Partial Usage
(c) G
Wasted instance−hours (k)
Figure 9: Aggregation suppresses the demand fluctuation of individu the demand fluctuation level (the ratio between the demand standard 1000 800
Before aggregation After aggregation −23.4%
600 400
−5.6% −30.5%
200 0
−16.5% High
Medium Low Demand Fluctuation
All
Figure 10: Aggregation reduces the wasted instance-hours due to partial usage. disk, memory, etc. Instance We take such a dataset as input, Wei Wang, Department of Electrical and Computer Scheduling. Engineering, University of Toronto and ask the question: How many computing instances would
Wednesday, 7 August, 13
ea dis on ge
tic de be va flu sta thr rat
de
30
(a) Impact of initial estimates. (b) Reservation granularities. hat it reduces 1200 usage. To see 1000 W/O broker W/O Figure 11: The convergence of ADP inbroker different scenarios. uring which it W/ broker W/ broker 1000 800 50 this user purHeuristic 800 up, we do the 600 Greedy 40 600 Online mpare it with users in that 400 No volume 30 400 d, we observe discount 200 r cases. Inter- 200 Heuristic 20Greedy Online eedy Online Heuristic Greedy Online ithms Algorithms Algorithms ficant for users 10 um fluctuation (d) All the users ctuated users. (c) Group 3: low fluctuation 0 ount of highly High Medium Low All ce costs with and without broker in different user groups. Demand Fluctuation Cost (k $)
Saving Percentage (%)
Cost w/ Broker (k $)
Figure 12: Aggregate cost savings in different user groups 8 to the brokerage service. 8 due
Cost w/ Broker (k $)
ional ADP
Cost (k $)
Cost Savings Due to the Broker
6 6 r under differy = x→ Cost Savings y = x→ 6.3 Aggregate neffectiveness 4 We now evaluate the aggregate 4 cost savings offered by the problem. We broker, under three different reservation strategies, namely, of ADP. First, 2 2 the Heuristic (Algorithm 1), Greedy (Algorithm 2) and On(Algorithm 1) 0 0 line. Assuming we the 0 2 4 a specific 6 8strategy 0 is used, 2 4 compare 6 8 0oarse-grained 50 60 70 Cost w/o cost Broker $) are using the Costbroker w/o Broker $) sum age (%) total service if (kusers with(k the ation is made, fluctuation (b) All the users without costs2:if medium each user individually makes reservations ers a multiple of (a)ofGroup using the broker. Fig. 13 shows such comparisons in each n granularity. 15: Cost without broker with theofbroker for alategy users due userand group, while Fig. 12the shows the vs. percentage cost savWei Wang, Department ofFigure Electrical Computer Engineering, University of Toronto leads to individual users, using ms. ings due to the use of aGreedy broker.strategy. Each circle is a user. Wednesday, 7 August, 13 accelerate the
31
Conclusions We propose a smart cloud brokerage service Reserves a pool of instances to serve the aggregated demand Leverages the price gap between the wholesale and retail model to reap the profit while offering lower price to cloud users Cloud users purchase instances from the broker as if instances were offered on demand
Design and analyze three instance reservation algorithms for the broker and evaluate them via trace-driven simulations More detailed analysis of online algorithms are given in our follow-up work [Wang et al. ICAC’13]
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
32
Thanks! http://iqua.ece.toronto.edu/~weiwang/
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Wei Wang*, Di Niu+, Baochun Li*, Ben Liang* * Department of Electrical and Computer Engineering, University of Toronto + Department of Electrical and Computer Engineering, University of Alberta July 10, 2013 Wednesday, 7 August, 13
Growing Cloud Computing Costs Drastic increase in enterprise spending on Infrastructure-as-aService (IaaS) clouds 41.7% annual growth rate by 2016 [CloudTimes’12] IaaS cloud will be the fastest-growing segment among all cloud services
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
2
Auto Scaling
On-Demand Instances let you pay for compute capacity by the hour with no long-term commitments. This frees you from the costs and complexities of planning, purchasing, and maintaining hardware and transforms what are commonly large fixed costs into much smaller variable costs.
Tradeoffs in Cloud Pricing Options Elastic Load Balancing High Performance Computing
The pricing below includes the cost to run private and public AMIs on the specified operating system (“Windows Usage” prices apply to Windows Server® 2003 R2, 2008 and 2008 R2). Amazon also provides you with additional instances
VM Import/Export
for Amazon EC2 running Microsoft Windows with SQL Server, Amazon EC2 running SUSE Linux Enterprise Server, Amazon EC2 running Red Hat Enterprise Linux and Amazon EC2 running IBM that are priced differently.
On-demand instances Related Resources
No commitment
Region:
US East (N. Virginia)
AWS Management Console Documentation
Pay-as-you-go
Linux/UNIX Usage
Windows Usage
Standard On-Demand Instances
Release Notes
Small (Default)
$0.080 per Hour
$0.115 per Hour
Developer Tools
Medium
$0.160 per Hour
$0.230 per Hour
Large
$0.320 per Hour
$0.460 per Hour
Extra Large
$0.640 per Hour
$0.920 per Hour
$0.020 per Hour
$0.020 per Hour
Sample Code & Libraries Developer Tools
Micro On-Demand Instances Articles & Tutorials
Reserved instances Amazon Machine Images (AMIs)
Micro High-Memory On-Demand Instances
Public Data Sets on AWS
Extra Large
$0.450 per Hour
$0.570 per Hour
Community Forum
Double Extra Large
$0.900 per Hour
$1.140 per Hour
Quadruple Extra Large
$1.800 per Hour
$2.280 per Hour
Reservation fee + discounted price Suitable for long-term usage commitment http://aws.amazon.com/ec2/pricing/
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Page 1 of 9
3
On-demand v.s. Reservation Pros
Cons
1. Flexible Expensive for longOn-demand 2. Fits sporadic workload term usage
Cost efficient for longReservation term usage
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
1. Long-term usage commitment 2. Expensive for sporadic workload
4
User’s Problem Hard to choose among different pricing options Lacks sufficient expertise
Cost savings due to the reservation option are not always possible Depends on the user’s own demand pattern Must be long-term and heavy usage
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
5
Can we go beyond the limitation of demand pattern of a single user and lower the cost?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
A Cloud Brokerage Service A cloud broker reserves a large pool of instances Users purchase instances from the broker in an “on-demand” fashion User
Broker
Providers
User cost
User ...
Broker cost
"On-demand" Instances
...
Reserved/On-demand Instances IaaS Cloud
User
Figure 1: The proposed cloud broker. Solid arrows show the direction of instance provisioning; dashed arrows show the direction of money flow. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Billing cycle (an instance-hour)
7
Why cloud broker?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Better Exploiting Reservation Options Statistical multiplexing increases the utilization of reserved instances Aggregating all users’ demands smoothes out the “bursts” A flat demand curve is more friendly to reserved instances The “true cost” of reserved instance is reduced due to the increased instance utilization
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
9
cloud
"On-demand" Instances
.
Reserved/On-demand Instances IaaS Cloud
Broker User ReducingProviders Wasted Cost Due to Partial Usage ...
roker Broker cost User cost hould User Alleviate the pricing inefficiency of on-demand instances mand, Figure 1: The proposed cloud broker. Solid arrows show the ically Partial usage is counted as a full billing cycle direction of instance provisioning; dashed arrows show the prob- The broker can time-multiplex partial usage direction of money flow. data, c proBilling cycle (an instance-hour) ng is User 1 User 1 Instance 1 nt apWithout broker User 2 Instance 2 e cost rithm With broker User 1 User 2 User 1 ess to Figure 2: The broker can time-multiplex partial usage from different users in the same instance-hour. In this case, serving two users only takes one instance-hour, instead of two.
GB of users’ te the y the discounted hourly rate charged over the entire reservation ur Wei reWang, Department of Electrical and Computer Engineering, University of Toronto period, no matter whether it is used or not. EC2 also ofusers Wednesday, 7 August, 13
10
Enjoying Volume Discounts Most IaaS clouds offer significant volume discounts Amazon provides 20% or even higher volume discounts in EC2 The sheer volume of the aggregated demand makes cloud broker easily qualify for such discounts
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
11
A Win-Win Solution Users receive a lower price when trading with the broker No upfront payment for reservation No money wasted on idled reservation instances
Broker makes profit by leveraging the wholesale (reservation) model A significant price gap between on-demand and reserved instances Aggregate demand is more amenable to the reservation option
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
12
How many instances should a broker reserve?
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
estimate its future demand to decide how many instances reserve at a Reserved particular time. In the case where users are On-demand and Pricing able to estimate demand at all, we propose an online reser strategy in Sec. 5.3 to make decisions based on histo On-demandtion instances Suppose cloud users submit to the broker their dema Fixed hourly rate p estimates up to time T into the future (in terms of bill Reserved instances cycles). The broker aggregate all the demands. Suppos requires dfee: t instances in total to accommodate all the reque Upfront reservation at time t, t = 1, 2, . . . , T . The broker makes a decision Reservation period: ⌧ reserve rt instances at time t, with rt 0. Each reserv rt Instances reserved time instance at will be t:effective from t to t + ⌧ 1, with ⌧ be theinstances reservation period. # of reserved that are effective at time t At time t, the number of reserved instances that rem Pt effective is nt = i=t ⌧ +1 ri . Note that these nt reserv instances may not be sufficient to accommodate the agg gate demand dt . The broker thus needs to launch (dt nt additional on-demand instances at time t, where X + max{0, X}. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
14
thus to launch (dtat all, nt )we propose the second element is 2 because tw le toneeds estimate demand an online reservainstances may not +be sufficient to accommodate the aggreper nces at time t, where X := 4.1 no later than stage 2 and remain effe + on strategy in Sec. 5.3 to make decisions based on history. Dynamic Resource Reservation gate demand dt . The broker thus needs to launch (dt nt ) the We note that given state s := + t 1 Suppose cloud users submit to the broker their demand We additional on-demand instances at time t, where X := nos me reservation fee for each rethe previous stage, the current state max{0, X}. timates up to time T into the future (in terms of billing lem (2)W Cloud users submit demand predictions to the broker te the price of running an onindependent sthe 2, A ts Let denote the one-time reservation fee of forthe each re-stcle. cles). The broker aggregate all the is demands. Suppose itstates Broker reserves instances based on the aggregate demand cycle. Hence, the total cost to characterized by s , with state tra t 1 served instance, and p denote the price of running an onis i quires dt instances in total to accommodate all the requests where ds d1 , . . demand . , dT is instance per billing cycle. Hence, the total cost to cha time t, t = 1, 2, . . . , T . The broker makes axdecision to no late i= i = yi+1 + x⌧ 1 , accommodate all the demands d , . . . , d is 1 T rtcost instances at time t, with rt 0. Eachcost reserved Pserve 1, . . . , T Total = +Reservation cost + On-demand (d n ) p , (1) To see the rationale behind (3), let t t t=1 stance will be effective 1, with ⌧ being PT from t toPtT+ ⌧ cause n + rt + t=1 (dt 4. n p , t + ⌧ (1) Fig. At 1, there are To t ) stage t=1 e reservation period. effectiv Figr tal cost of reservations and the remain effective. Clearly, all these where, At time t, the number of reserved instances that remain x1 · P where the first term is the totalstage cost of reservations and the rem t demand instances. The broker’s t (because the reservations m fective is nt = i=t ⌧ +1 ri . Note that these nt reserved reserve second is the cost instances. reservation decisions r1 , . of . . ,all rTon-demand all expired at tThe + ⌧broker’s 1), i.e., rsta t stances problem may not be sufficient to accommodate the aggre# of reserved that are decisions effectiver1at, + is to makeinstances dynamic reservation . t. . , rT period all te demand dt . The broker thus needs to launch (dt nt ) the sec ditional on-demand instances at3 time t, where X + := no late 3 ax{0, X}. We n Let denote the one-time reservation fee for each rethe pre rved instance, and p denote the price of running an onis indep Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
15
Fig. 4. A max{0, We no hased a X}. st term is the total cost of reservations and the remain ef Stage rovides The Cost Minimization Problem Let denote the one-time reservation fee for each rethe prev cost allFigure on-demand instances. broker’s t (b eerved to theof 3: State illustration. The The reservation period is ⌧ =stage 4.is indepe instance, and p denote the price of running an onbroker All four reservations are highlighted as, total the shaded area. all Atcharacte make reservation decisions . . . , r expire demand instance per billing cycle. Hence, rthe cost to Makedynamic dynamic reservation decisions to 1 T 1
reduces
2
3
4
5
6
7
8
9
10
stage 2, s2 = (2, 2, 2, 1).
ccommodate all the demands d1 , . . . , dT is accommodate demands
okerage PT cost, i.e., + toP minimize its total T 3 r + (d n ) p , (1) To see th s at the t t t t=1 t=1PT PT + min cost = r + (d n ) p . (2)Fig. 4. A t king int=1 t t=1 t {r1 ,...,rT } nd datathe first term is the total cost of reservations and the where remain e Problem (2) is integer programming. In general, complex llowing econd is the cost of all on-demand instances. The broker’s stage t ( methods are needed to solve it. This is ancombinatorial integer program!
problem is to make dynamic reservation decisions r1 , . . . , rT
TION
4.
DYNAMIC PROGRAMMING: THE OPTIMALITY AND LIMITATIONS 3
In this section, we resort to dynamic programming to charnstance acterize the optimal solution to problem (2). Using a set an obof recursive Bellman equations, the original combinatorial users to optimization problem can be decomposed into a number of , based Wang, Department subproblems, of Electrical and Computer Engineering, of Toronto can be solved efficiently. Howeach University of which oteWeithat ever, we also point out that computing such a dynamic proWednesday, 7 August, 13
all expir
16
Optimal Solution: Dynamic Programming
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
ic1 The Programming FormulationFormulation Dynamic Programming Curse of Dimensionality
fining stages and states. The and decision probWe start by defining stages states. The decision probHigh dimensional dynamic programming of T stages, each representing a billing cy- a billing cym (2) consists of T stages, each representing geAt is denoted by at ⌧is-tuple st := ), x1 , . . . , x⌧ High dimensional state: e. state at stage denoted by (t, a ⌧ x-tuple 1 , . . . s, tx:= ⌧ 1(t, s the xnumber of the instances that reserved here number ofare instances that aret and reserved # of instances that are reserved no later than remain i :denotes effective t+i remain remain at stage t + i, for i = t + i, for i = ondlater thaneffective tatand effective at stage ere, ⌧ -tuple to represent a state be. .Exponential . we , ⌧ use1.a Here, we use a ⌧ -tuple to represent a state betime and space complexity euse reserved beforereserved or at stage t will no instance before or remain at stage t will remain The curse of dimensionality age t + after ⌧ 1. Andt + it is⌧ easy check fective stage 1. to And it isthat easy to check that . For example, in Fig. 3, three instances are 1 · · · x . For example, in Fig. 3, three instances are 1 ⌧ 1 1, 2, and 3, respectively, with a reservation served at stage 1, 2, and 3, respectively, with a reservation e see ⌧that 2, sthat (2, 2, 2,2, 1),s2where 2 =at eriod = at 4. stage We see stage = (2, 2, 2, 1), where ent is 2 because e second elementtwo is 2instances becauseare tworeserved instances are reserved 2 and remain at stage 3. ogelater than stageeffective 2 and remain effective at stage 3. given state s := (t 1, y , . . . , y ) at t 1 1 ⌧ 1 We note that given state st 1 := (t 1, y1 , . . . , y⌧ 1 ) at ge, the current statethe st current := (t, xstate ) x ,...,x 1 , . . .s, x:= ⌧ 1(t, e previous stage, t 1 ⌧ 1) Wei Wang, Department of Electrical and Computer Engineering, University of Toronto
Wednesday, 7 August, 13
1 ),
18
Approximate Solution
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
A 2-Competitive Heuristic Segment the demand into intervals each spanning one reservation period
Demand
Interval
⌧
2⌧
3⌧
4⌧
Time
Make optimal instance reservation decisions per interval
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
20
Optimal Instance Reservation within an Interval Stratify demand into levels For each level, decide if a reserved instance should be used Example
Should reserve when instance usage >= 3 hours
On-demand rate: $1 per hour 6 5 4 3 2 1 0
6
L5 L4
1
2
3
5
6
Interval 1
5 4 3 2 1 0
1
2
3
4
5
6
Time (hour)
Time (hou
(a) T = 5, ⌧ = 6.
(b) T = 8, ⌧ =
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
4
L3 L2 L1 7
Demand
Demand
Reservation: $2.5 for 6 hours
21
Cost Performance Per-interval reservation is 2-competitive Incurs at most twice the optimal cost in the worst case
Demand
Interval
⌧
2⌧
3⌧
4⌧
Time
All reservations are made at the beginning of the interval
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
22
the demand at time t in level l 1. We then step to level , l 1 and make optimal reservations there, taking into account the “leftover” reserved instances passed over from the he last inupper level. Undoubtedly, the algorithm becomes more effi1)). Equacient, since each level triesintervals to utilize such “leftover” reserved Do not segment demand into intervalinstances from upper levels. Note that this is not possible y. Stratify into levels for a bottom-up approach, where no “leftover” reserved inthm 1 in- demands stances can be passed from a lower level up. strategies, L6 reservations In each level of the above procedure, optimal 1. By (8), L5 can be efficiently made via dynamic programming. Suppose incurs no L4 L3 before we make reservations in level l, mlt reserved instances L2 L1 of which can are passed over from upper levels at time t, all rth noting be utilized. Let Vl (t) be the minimum costTime of serving deMake reservations top-down estimates mands dl1 , . . . , dlt in level l up to time t . The Bellman equaup its contion isapply givendynamic by At each level, programming by AlgoVl (t) = min{Vl (t ⌧ ) + , Vl (t 1) + cl (t)}. (10) each state o be whichthan chooses the minimumReservation, between two options. first Strictly better Per-Interval and isThe also 2l (9) is to serve the demand d t with an instance reserved in the competitive Demand
An Improved Greedy Algorithm
6
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
23
When demand predictions are unavailable
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
Online Algorithm
# Instances
The best that we can do [Wang et al. ICAC’13]
Algorithm 2 summarizes the greedy reservation strategy ¯ ) and space complexabove, which has time complexity O( dT 2-competitiveness for the deterministic online algorithm ity O(T ). As has been analyzed earlier in this subsection, in each level, Algorithm 2 is more cost efficient than Algorithm 1, leading to the following proposition: Proposition 2: Algorithm 2 incurs a cost no more than Algorithm 1, and is thus 2-competitive.
600 400 200 0 0
# Instances (k)
Algorithm 3 Online Reservation Made at Time t 1. Let gi = (di ni )+ for all i = t ⌧ + 1, . . . , t. 2. Run Algorithm 1 with gt ⌧ +1 , . . . , gt as the input demands. Let x be its output. 3. Reserve rt = x instances at time t. 4. Update ni = ni + rt for all i = t ⌧ + 1, . . . , t + ⌧ 1.
# Instances
Make instance reservation decisions without future information
1.5 1 0.5 0 0
5
600 400 200 0 0
Fig. 6.
C. An Online Reservation Strategy
5
The d
250
d Std
Previous algorithms apply to users who predict their future demands. For those who cannot, we propose an online strategy Wei Wang, Department of Electrical and Computer Engineering, University of Toronto that reserves instances based only on history, without accessing Wednesday, 7 August, 13 to any future information. Recall that n is the number of
5
200 High 150
y = 5x 25
M
Trace-Driven Simulations
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
# Instanc
1
0.5 (11) Dataset and Preprocessing 0 0
50
150
# Instances
and instance 600 traces noGoogle reserved cluster-usage 400 t, i.e., mlt =
100 Time (hour)
200 User 187
900+ users’ usage traces on a 12K-node Google datacenter 200 We convert users’ computing demand data to IaaS instance demand 0 (12)
0
50
100 Time (hour)
150
200
Users are classified into 3 groups based on demand fluctuation level
ervation stratce complexrogramming ) space. As each level, m 1, leading
Demand Std
, we update Figure 7: The demand curves of three typical users. Standard deviation vs. mean in hourly demand ssed to level n instance is 250 := mlt 1, 200 High y = 5x y=x passed over 150
Medium
100 50 0 0
Low 50
100 150 200 Demand Mean
250
300
Figure 8: Demand statistics and the division of users into 3 Wei Wang, Department of Electrical and Computer Engineering, University of Toronto groups according to demand fluctuation level. Wednesday, 7 August, 13
27
and instance no reserved t, i.e.,Relatively mlt =
stable (12)
600
User 863
400 200 0 0
# Instances (k)
,
Medium (11) fluctuation
# Instances
ly serve deance at Highly time The fluctuated second 1 and serve with a cost
# Instances
Demand Curve
50
100 Time (hour)
150
1.5
200 User 48
1 0.5 0 0
50
100 Time (hour)
150
600
200 User 187
400 200 0 0
50
100 Time (hour)
150
200
l, we update Figure 7: The demand curves of three typical users. Wei Wang, Department of Electrical and Computer Engineering, University of Toronto ssed to level Wednesday, 7 August, 13
28
Aggregation Smoothes Out Demand Bursts 200
90 60 30 0 0
y = 1.774x 5
250
150 100 50 0 0
10 15 20 25 30 Demand Mean
(a) Group 1: high fluctuation
300 Demand Std
120
Demand Std
Demand Std
150
y = 0.363x 20 40 60 80 Demand Mean
200 150 100 50
100
(b) Group 2: medium fluctuation
0 0
y
200 400 60 Demand M
(c) Group 3: low fl
200
Demand Std
Demand Std
Figure 300 9: Aggregation suppresses the 300 demand fluctuation of individual users. Ea the250 demand fluctuation level (the ratio 250 between the demand standard deviation an 200
each reservatio Before aggregation discount of 50 100 After aggregation y = 0.058x y = 0.061x y = 0.363x on-demand ins 50 −23.4% general pricing 0 600 0 0 40 60 80 100 0 200 400 600 800 1000 0 200 400 600 800 1000 Demand Mean Demand Mean Demand Mean Group Divi 400 −5.6%(d) All the users. tics of users, w 2: medium fluctuation (c) Group 3: low fluctuation −30.5% deviation for e 200 mand fluctuation of individual users.−16.5% Each circle represents a user. The line indicatesbeen mentione Wei Wang, Department of Electrical and Computer Engineering, University of Toronto 29 0 tween the demand standard deviation and mean) in the aggregate demand curve. vations critical High Medium Low All Wednesday, 7 August, 13 Wasted instance−hours (k)
150 1000 100 800 50
150
(a) Group 1: high fluctuation
(b) Group 2: medium fluctuation
The Reduction of Partial Usage
(c) G
Wasted instance−hours (k)
Figure 9: Aggregation suppresses the demand fluctuation of individu the demand fluctuation level (the ratio between the demand standard 1000 800
Before aggregation After aggregation −23.4%
600 400
−5.6% −30.5%
200 0
−16.5% High
Medium Low Demand Fluctuation
All
Figure 10: Aggregation reduces the wasted instance-hours due to partial usage. disk, memory, etc. Instance We take such a dataset as input, Wei Wang, Department of Electrical and Computer Scheduling. Engineering, University of Toronto and ask the question: How many computing instances would
Wednesday, 7 August, 13
ea dis on ge
tic de be va flu sta thr rat
de
30
(a) Impact of initial estimates. (b) Reservation granularities. hat it reduces 1200 usage. To see 1000 W/O broker W/O Figure 11: The convergence of ADP inbroker different scenarios. uring which it W/ broker W/ broker 1000 800 50 this user purHeuristic 800 up, we do the 600 Greedy 40 600 Online mpare it with users in that 400 No volume 30 400 d, we observe discount 200 r cases. Inter- 200 Heuristic 20Greedy Online eedy Online Heuristic Greedy Online ithms Algorithms Algorithms ficant for users 10 um fluctuation (d) All the users ctuated users. (c) Group 3: low fluctuation 0 ount of highly High Medium Low All ce costs with and without broker in different user groups. Demand Fluctuation Cost (k $)
Saving Percentage (%)
Cost w/ Broker (k $)
Figure 12: Aggregate cost savings in different user groups 8 to the brokerage service. 8 due
Cost w/ Broker (k $)
ional ADP
Cost (k $)
Cost Savings Due to the Broker
6 6 r under differy = x→ Cost Savings y = x→ 6.3 Aggregate neffectiveness 4 We now evaluate the aggregate 4 cost savings offered by the problem. We broker, under three different reservation strategies, namely, of ADP. First, 2 2 the Heuristic (Algorithm 1), Greedy (Algorithm 2) and On(Algorithm 1) 0 0 line. Assuming we the 0 2 4 a specific 6 8strategy 0 is used, 2 4 compare 6 8 0oarse-grained 50 60 70 Cost w/o cost Broker $) are using the Costbroker w/o Broker $) sum age (%) total service if (kusers with(k the ation is made, fluctuation (b) All the users without costs2:if medium each user individually makes reservations ers a multiple of (a)ofGroup using the broker. Fig. 13 shows such comparisons in each n granularity. 15: Cost without broker with theofbroker for alategy users due userand group, while Fig. 12the shows the vs. percentage cost savWei Wang, Department ofFigure Electrical Computer Engineering, University of Toronto leads to individual users, using ms. ings due to the use of aGreedy broker.strategy. Each circle is a user. Wednesday, 7 August, 13 accelerate the
31
Conclusions We propose a smart cloud brokerage service Reserves a pool of instances to serve the aggregated demand Leverages the price gap between the wholesale and retail model to reap the profit while offering lower price to cloud users Cloud users purchase instances from the broker as if instances were offered on demand
Design and analyze three instance reservation algorithms for the broker and evaluate them via trace-driven simulations More detailed analysis of online algorithms are given in our follow-up work [Wang et al. ICAC’13]
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
32
Thanks! http://iqua.ece.toronto.edu/~weiwang/
Wei Wang, Department of Electrical and Computer Engineering, University of Toronto Wednesday, 7 August, 13
