Automation
A Day in the life of a Network Engineer - Day 2 with ACI Ishan Mehta, Sr.Network Engineer Mani Govindasamy, Sr.Network Engineer BRKCOC-2012
Cisco Spark Questions? Use Cisco Spark to chat with the speaker after the session
How 1.
Find this session in the Cisco Live Mobile App
2.
Click “Join the Discussion”
3.
Install Spark or go directly to the space
4.
Enter messages/questions in the space
Cisco Spark spaces will be available until July 3, 2017.
cs.co/ciscolivebot#BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda •
Architecture & Design
•
Day 2 with ACI
•
Lessons Learned
Cisco IT’s: ACI Architecture & Design
Complexity of Application Delivery thru IT Performance
Security
Reseliency
Scale
IT Organization
Application Compute Team
Security
Network Team
Storage Team
Web
App Tier Physical, Virtual Servers
Virtualization Team
DB Tier
Physical, Virtual Servers
Storage
Physical Servers
Firewall
Web Security Appliance
Application Delivery Controller SW ITCH VLAN IP QoS ACLs
ADC Services Rules
Web cache VIRTUALIZED SERVICES VLAN IP QoS ACLs
Intrusion Detection
Intrusion Detection
Application Delivery Controller
Application Delivery Controller Storage Firewall
Firewall FIREW ALL Security Policy
VIRTUALIZED SERVICES VLAN IP QoS ACLs
FIREW ALL Security Policy
BRKCOC-2012
VIRTUALIZED SERVICES VLAN IP QoS ACLs
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
6
Where we are? •
7 ACI Fabrics, biggest has 8 Spine, 40 Leaf and 3 Controller
•
Over 70+ Virtual Network Appliance
•
Over 10K Endpoints, 9k Virtual Machines
•
Over 20 PB storage behind ACI
•
100+ BM Migrated
•
100+ Application Migrated
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
Cisco IT’s ACI Design •
Design Goals • • • •
Flexible Design with Automation in Mind
Flexible Topology Any workload anywhere Near zero downtime Enhanced Security
Components • Border Leafs Leaf Pairs
Bare Metal Compute
Virtual Compute
multiple IP Storage Leaf Pairs 42
42
42
41
41
41
42
42
41
41
40
40
39
39
38
38
37
37
37
36
36
36
42 42
42
42
42
42
42
42
42 42
41 41
41
41
40
40
40
40
39
39
39
39
38
38
38
38
38
37
37
37
37
37
36
36
36
36
41
41
41
41
40
40
39
39
38
37
36
36
40
40
40
39
39
39
40
38
39
38
38
37
37
37
36
36
36
36
38
35
35
35
35
35
34
34
34
34
34
34
33
33
33
33
33
33
35
35
35
34
34
34
34
34
34
34
34
33
33
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
31
31
35
NAS CDOT
32
32
32
32
31
31
31
31
31
31
30
30
32
30
30
30
29
29
35
35
35
29
29
28
28
29
38
37
37
36
36
27
27
26
26
25
26
26
25
25
25
24
24
24
24
23
23
23
23
23
22
22
22
22
22
24
32
32
31
31
30
30
29
29
29
29
28
28
28
27
27
27
26
26
26
26
25
25
25
25
30
30
34
35
35
33
34
34
30
30
32
32
33
31
31
32
32
31
31
21
30
27
26
26
26
26
25
25
25
25
29
29
28
28
28
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
24
23
23
23
23
23
23
22
22
22
22
22
22
22
21
20
20
19
19
24
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
18
18
17
17
15
15
15
15
15
14
14
14
14
14
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
21
21
21
20
20
20
20
20
20
19
21
19
19
19
19
21
19
27
27
25
26
26
18
18
18
18
18
18
17
17
17
17
17
17
24
42
41
41
40
40
39
39
39
42
41
•
40
39
38
38
38
37
37
37
37
36
36
38
36
35
34
36
36
35
35
34
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
34
33
33
32
32
35
35
34
34
33
33
32
32
31
31
31 31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
28
25 25
24
24
23
23
21
21
22
22
20
20
21
21
20
25
25
25
31
30
30
29
29
30
24
24
24
24
23
23
23
23
20
22
22
22
30
29
28
28
27
27
27
26
26
26
26
25
24
23
22
29
28
27
28
25
24
23
22
19
25
25
23
22
16
16
16
15
15
15
15
15
15
15
14
14
14
14
14
14
14
16
13 13
16
16
18
18
19
19
17
17
18
18
16
16
16
25
24
24
23
23
22
22
21
21
25
24
23
25
24
23
22
21
21
21
21
21
21
20
20
20
20
20
20 20
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
12
12
11
11
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
9
9
8
8
7
7
6
6
5
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
17
15
16
16
14
15
15
5
5
5
4
4
4
4
3
3
3
2
2
2
1
1
1
5
5
5
4
4
4
3
3
3
2
2
2
1
1
1
13
14
14
12
12
13
13
11
11
12
12
10
10
11
11
9
9
10
10
8
9
9
7
7
8
8
6
6
7
7
5
6
6
4
5
3
4
2
3
1
2
2
1
1
8
5
4 4
10
19
19
19
19
19
19
18
18
18
18
18
18
17 17
15
14
13
7
42
41
40
39
38
37
36
35
34
30
28
26
23
22
21
14
13
7
42
41
40
39
38
37
36
35
34
29
27
26
24
15
16
15
7
42
41
40
39
38
37
36
35
34
20
22
22
21
21
20
20
18
14
7
42
41
40
39
38
37
36
35
34
31
30
29
27
25
24
19
L3 IP out to IP Core
30
29
28
27
24
23
22
24
24
27
24
23
21
Netapp backend network
42
41
40
39
38
37
35
34
33
30
29
28
27
27
24
21
42
41
40
38
37
28 28
27
42
41
36 35
34
33
29
28
27
26
25
23
22
24
42
40
39
38
36
35
28
29
28
27
26
25
27
39
37
36
30 30
29
28
41
40
38
37
35
32
42
41
40
39 38
35
42
40
39
39 39
38
37
36
41
40 40
38
37
42
41 41
40
39
5
4
17
17
17
17
19
19
19
18
18
18
17
16
16
16
16
16
15
15
15
15
15
14
14
14
14
14
19
18
17
16
15
17
16
16
15
15
14 14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
5
14
17
17
16
16
15
14
15
14
13
13
13
12
12
12
12
13
11
11
11
11
10
10
10
10
9
9
9
9
8
8
8
8
7
7
7
7
6
6
6
6
5
5
4
4
3
3
2
2
1
1
5
5
3 3
3
2
2
1
1
3
2
1
Citrix VPX
4
3
Citrix VPX APIC
APIC
APIC
BRKCOC-2012
4
3
4
3
2
2
1
1
• • •
ACI: 9508, 9396PX, 93180YCEX, APIC UCS: B420m3, B200m4, B200m3 ESX + AVS NetApp CDoT Citrix VPX & ASAv
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
Logical Mapping EPG to BD to Subnets to VRFs to External (L3 Out) Tenant 1
Tenant 2
Tenant 3
Internet
Tenant Common EPG-DMZ
EPG-11
EPG-21
EPG-31 EPGInternet
1.1.1.0/24 2.2.2.0/24
DC Core (External)
BD-Ext-1 VRF-dmz BD-Ext-2 EPG-12
EPG-22
3.3.3.0/24
9396 9396
EPG-32 4.4.4.0/24 BD-sec-1
VRF-Secured
DC Core (Internal) BD-int-1 5.5.5.0/24 EPG-13
EPG-23
EPG-33
VRF-Int EPG-Corp EPGOther-DC
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
ACI Security Architecture
Tenant Common
Private Cloud view
Tenant 1
EPGs consume or provide directly Foundational Services Contracts
EPG-11
Tenant 2 EPG-21
C
Contractsspecific
C
EPG-Internet
Contractsspecific
C
EPG-Corporate
Contracts-OCM
P
EPG-OCM
Contracts-OAM
P
EPG-OAM
Contract-LDAP
P
EPG-LDAP
Contract-…
P
Tenant 3
L3 Out
C
C
P EPG-13
P EPG-23
P
EPG-32
VRF Level
EPG-22
EPG-33
C C C C
Infra Services Contracts
P EPG-12
EPG-…
L3-out
ContractMonitoring
C
EPG-Monitoring
Contract-NTP
P
EPG-NTP
Contract-DNS
P
EPG-DNS
Contract-PostOSInstall
P
EPG-PostOSInstall
Contracts-…
P
EPG--…
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
ACI Consumption Model
Consume Provide
Syslog
DNS
NTP
Infra Service bundle
External Network LoadBalancer
Firewall
APP EPG
SQL Contract
Web EPG
Java Contract
Web Contract
Firewall
LoadBalancer
DB EPG
Internal Network Management Bundle
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
Any workload anywhere
Virtual Load Balancer
40+ Citrix VPX deployed on UCS 1 VNIC for mgmt, 9 VNIC for data Each VNIC maps to one EPG on ACI Fabric Spine Switches
WAF deployment for DMZ services
Leaf Switches Mgmt vNIC
Data vNIC
Mgmt vNIC
Data vNIC
NS1000v Citrix VPX (Active) (active)
ESXi host 1
NS1000v Citrix VPX (Standby) (standby)
ESXi host 2
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
Consume Provide Bindings
Citrix VPX & ACI Tenant 1
Tenant 2 VIP & sNAT
External Network
VIP EPG
Web Contract
VIP EPG
Contract
Contract
Server-1
Server-1
Server-2
Server-2
Web-EPG
App-EPG
Citrix VPX Citrix VPX
VIP-BD1 (10.10.10.1/24)
BD-VPX-Mgmt (10.100.11.1/24)
EPG within Fabric VIP + sNAT Citrix VPX Citrix VPX
VIP-BD2 (20.20.20.1/24) Tenant Common
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
Firewall Design
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
FW Architecture
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
Firewall Traffic Flow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
Firewall Traffic Flow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
Day 2 with ACI
Day 2 with ACI Automation
Maintenance • Software • Hardware • Organic Growth
• Greenfield • Brownfield Migrations • Operations
Troubleshooting
Monitoring • Enterprise Monitor • Splunk - Syslog • Programmability
• VM Connectivity loss • Contract Issue • Vlan Duplication
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Automation
Greenfield Provisioning
Brownfield Migration
Operations
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
Automation Greenfield Application Developers
Cloud Orchestration
DC Resources
Automation Packs C
O A M / S S O
C P r o x y
TCP: *,443
C
W e b
C
S e r v i c e
M e m c a c h e
Internet
C
C O r a c l e D B
C
I n t e g r a t i o n
Self-describing
C
Self-Service Catalog
Provisioning Automation
R a b b i t M Q
C a s s a n d r a
E l a s t i c S e a r c h
Packaging
Model
Click
Manageability
Deploy Fault-tolerant
BRKCOC-2012
Self-optimizing
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
Automation
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
Automation Brownfield Migration Compute Storage
95%
Network
70%
ACI Portal Unmapped Hosts
60%
Booking ID
App. View
Booking Tool
NW View
Portal Admin
Metrics
100%
CMDBServiceNow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Migration Process
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
Migration Brownfield Migration 1
Build the ACI Fabric in Parallel
2
Install Seed Compute/Storage into ACI Fabric
3
Connect Fabric to Traditional Network
4
Configure EPGs, Contracts, etc.
5
Move Applications to ACI
6
Move Compute/Storage Unit
7
Iterate: Identify next App(s) to set up and migrate
Traditional
ACI
Application Profile
Infrastructure
LIF LIF
Compute
Nexus
Storage
ASA
ACE
BRKC0C-2012
Seed Compute
ACI SW
Seed Storage
ASAv
Netscaler
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Maintenance
Software Upgrade
Hardware Upgrade
Network Maintenance
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
• Read the release notes • Export the backups • Download Firmware from controller
Hitless Upgrades
• Create Firmware upgrade group for odd and even
• Upgrade APIC • Upgrade odd Leaf and Spine • Upgrade even leaf and Spine
ACI Fabric ACI Switch Border Leafs Leaf Pairs
Multiple Compute & Services Leaf Pairs
multiple IP Storage Leaf Pairs
42 42
42
42
42
42
42
41
40
39
39
38
38
37
37
36
36
35
41
41
40
40
39
39
38
38
37
35
37
37
36
36
35
35
34
34
34
34
33
33
33
33
36
36
35
35
34
34
33
33
39
38
38
37
37
36
36
Internal
IP Network
Cisco+Corp
Internet
NAS
NAS
32
32
32
32
32
31
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
24
24
24
24
24
24
23
23
23
23
23
23
22
22
22
22
22
22
21
21
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
18
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
Compute
APIC
41
41
41
40
40
40
39
39
39
39
38
38
38
38
38
37
37
37
37
37
40
39
38
37
36
36
36
36
36
36
35
35
35
35
35
35
42
35
35
34
34
33
33
32
32
31
31
30
30
29
29
34
34
34
34
34
34
33
33
33
33
33
33
32
32
32
32
32
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
37
28
27
26
25
25
24
24
23
23
22
22
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
25
24
24
24
24
24
24
32
23
23
23
23
23
22
22
22
22
22
21
21
21
21
21
21
21
20
19
19
20
20
20
20
20
20
19
19
19
19
19
19
18
18
17
17
27
18
18
18
18
18
17
17
17
17
17
16
16
15
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
22
13
13
13
13
13
13
12
12
12
12
12
12
13
12
12
11
11
10
9
9
8
8
7
7
6
6
5
5
4
4
3
3
2
2
1
1
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
8
8
8
8
8
7
7
7
7
17
Application Virtual Switch
15
14
13
12
12
11
11
10
10
9
9
8
8
7
7
9
8
7
7
6
6
6
6
6
6
5
5
5
5
5
4
4
4
4
4
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
Services (physical or virtual)
18
16
13
10
19
16
14 14
13
20
17
15
14
23
21
18
15
24
21
19
18
25
22
20
17
28
26
23
20
29
26
24
21
30
27
25
22
33
31
28
26
34
31
29
27
35
32
30
28
38
36
33
31
39
36
34
32
41
40
37
35
28
25
Services (physical or virtual)
41
40
39
38
23
DMZ
42
41
41 41
40
39
39
32
28
42
42
41
40
39
37
40
41
40
38 38
42
42
41 41
40
40
39
42
42 42
41
40
6
6
5
5
4
4
3
3
2
2
1
1
5
4
3
ESXi/Vmware
Compute
APIC
APIC vCenter
BRKCOC-2012
UCS Compute (B-Series) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
Hardware Upgrades Traditional DC Network
ACI Fabric
DC Layer 3 Core
Layer 2
Layer 2
Layer 2
Layer 2
DC wide mobility domain
Limited mobility domains BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
Resource Stripe - ACI Fabric leafpair 2
leafpair1
leafpair 4
leafpair 3
42
42
42
42
42
42
41
41
41
41
41
41
40
40
40
40
40
40
39
39
39
39
39
39
38
38
38
38
38
38
37
37
37
37
37
37
36
36
36
36
36
36
35
35
35
35
35
35
34
34
34
34
34
34
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
25
24
24
24
24
24
24
23
23
23
23
23
23
22
22
22
22
22
22
21
21
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
18
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
42
42
41
41
40
40
39
39
38
37
36
36
35
35
34
34
33
32
31
31
30
30
28
29
28
27
27
26
26
25
25
24
24
23
23
22
22
21
21
20
20
19
19
18
18
17
17
16
16
15
15
14
13
13
12
11
11
10
10
9
9
8
8
7
7
6
6
5
5
3
Resource pools and applications striped across multiple pods
14
12
4
Easier Maintenance
33
32
29
38
37
Greater Resiliency
Infrastructure “hold-back”: 13%
4
3
2
2
1
1
BRKC0C-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
Network Maintenance
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
34
Network Maintenance Traditional
Time for Rack, Stack & Patch
Basic Configuration Build Code Upgrade & Configuration Push
Same
ACI Same
60 mins
5 mins
30 mins
10 mins Auto Upgrade & Configuration Push
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
Monitoring
Enterprise Monitor
Syslog
Homegrown Automation
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
36
Enterprise Monitor
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
Splunk:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
Capacity Dashboard Operations Capacity Dashboard
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
Monitoring Scripts
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
40
Troubleshooting
Connectivity Issues
VM Outage
Endpoint Tracking
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
Identify Contract Issues - GUI
173.x.x.x
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
42
Troubleshooting Wizard - Contracts
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
43
Contract Checker
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
Identify Contract Issues - CLI
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
45
VM Outage:
Condition •
This issue is trigged when we have an overlapping subnet configured in the Bridge Domain and L3Out External Network
Symptom • Lost Connectivity to VM • Endpoint learning flaps • MAC learned on the leaf Solution •
Remove the subnets from the L3Out networks.
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
VM Outage: Condition •
Stale endpoint after changing BD VRF association
•
Ingress Policy Feature can impact existing endpoints
Symptom •
Intermittent connectivity to VM
•
Stale Endpoint
Solution •
Clear the stale endpoint entry
•
Intermediate VRF
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
47
Visore – Its Italian for Viewer ! https://fabric-apic/visore.html
admin@apic1:~>moquery -c l3extSubnet | grep 10.40.125.50 admin@apic1:~>moquery -c fvCEp | grep 10.40.125.50 BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
Class Lookup
Current Screen:insieme.stromboli.layout.Tab [fv:infoAEPg:center:b ] | Current Mo:insieme.stromboli.model.def.fvAEPg [uni/tn-common/ap-ceph/epg-ceph-cluster-01 ] BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
Homegrown Script:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
VLAN Mapping for Endpoint
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
Endpoint Tracking
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
52
EP Tracker Operations EP Tracker
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
Find VLAN Script: Bare-metals moquery -c fvRsPathAtt -f "fv.RsPathAtt.encap==\"vlan-158\""
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
Iping:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Itrace:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
Itrace vs Traceroute:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Itrace: SPINE-1 TO LEAF 1042
SPINE-2 TO LEAF 1042
SPINE-1 TO LEAF 1042
SPINE-2 TO LEAF 1042 BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Network Engineer Skillset
Python
CLI
GUI
SDK / Toolkit
API
Management information tree APIC
CLI
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Lessons Learned
Lessons Learned – Design & Implementation •
Feature and Code certification process
•
Document naming conventions for various objects to make readability and troubleshooting easier
•
Traffic segregation [External Network]
•
Consistent naming convention & configuration
•
Dedicated leaf pairs for different work loads/function
•
Avoid full mesh contracts where possible
•
Use different contracts on each VRF
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Lessons Learned - Operations •
Check health scores to narrow down affected scope
•
Get familiar with reviewing faults and audit logs. If anything fails deployment then faults are raised
•
Get familiar with utilizing CLI and APIC (possibly Visore or MOQuery)
•
Check the resolved object model is present on all the APIC and relevant leafs
•
Check the concrete objects are present on the relevant leafs
•
Leverage tools like Endpoint Tracker and Troubleshooting Wizard
•
Document/Track issues faced
•
Change Management Review Process
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Cheat Sheet
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
References Title
Links
Cisco IT ACI Design
Cisco IT ACI Design
Cisco IT AVS & Vmware - Vsphere Distributed Switch Failover Convergence
Cisco IT AVS & Vmware DVS Case Study
Cisco IT Storage Design
Cisco IT ACI Storage Design
Cisco IT Compute at Scale on Cisco ACI
Cisco IT Compute at Scale on Cisco ACI
Cisco IT ACI Migration
Cisco IT ACI Migration
Tetration Analytics : IDC Business Value
Cisco IT case study
ACITool Kit
https://github.com/datacenter/acitoolkit/
DevNet
https://developer.cisco.com
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Continue Your Education •
Demos in the Cisco campus
•
Walk-in Self-Paced Labs
•
Lunch & Learn
•
Meet the Engineer 1:1 meetings
•
Related sessions
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
Complete Your Online Session Evaluation •
Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.
•
Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us. Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco on Cisco Breakout Sessions SESSION
SPEAKER(s)
BRKCOC-2006 - Inside Cisco IT: ACI & Tetration Analytics
Anitha Parimi & Laxmi Vellanki
BRKCOC-2019 - Inside Cisco IT: Leveraging Cisco WAAS to Improve Network Performance
Rubens Lima & Hara Bandhakavi
BRKCOC-2016 - Inside Cisco IT: Containers on Enterprise Compute and Networks
Michael Duarte
BRKCOC-2014 - Inside Cisco IT: Increasing the Speed of Business using AppDynamics
Yatin Wadhavkar & Clement Joseph
BRKCOC-2012 - Inside Cisco IT: A Day in the life of a Network Engineer - Day 2 with ACI
Mani Govindasamy & Ishan Mehta
BRKCOC-2018 - Inside Cisco IT: How Cisco Deployed ISE and TrustSec throughout the Enterprise
David Iacobacci
BRKCOC-2013 - Inside Cisco IT: Embedding Collaboration in Business Workflows using Cisco Spark
Paul Anholt & David deMilo
BRKCOC-2023 - Inside Cisco IT: Security Overview - Making it Work
Gil Daudistel
BRKCOC-2021 - Inside Cisco IT: DNA and the Next Generation Network
John Moe
BRKCOC-2017 - Inside Cisco IT: Using Machine Learning Technologies to Drive Digital Transformation
PLAMEN NEDELTCHEV
BRKCOC-2012
& Bassem Khalife
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
Thank you
Cisco Spark Questions? Use Cisco Spark to chat with the speaker after the session
How 1.
Find this session in the Cisco Live Mobile App
2.
Click “Join the Discussion”
3.
Install Spark or go directly to the space
4.
Enter messages/questions in the space
Cisco Spark spaces will be available until July 3, 2017.
cs.co/ciscolivebot#BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Agenda •
Architecture & Design
•
Day 2 with ACI
•
Lessons Learned
Cisco IT’s: ACI Architecture & Design
Complexity of Application Delivery thru IT Performance
Security
Reseliency
Scale
IT Organization
Application Compute Team
Security
Network Team
Storage Team
Web
App Tier Physical, Virtual Servers
Virtualization Team
DB Tier
Physical, Virtual Servers
Storage
Physical Servers
Firewall
Web Security Appliance
Application Delivery Controller SW ITCH VLAN IP QoS ACLs
ADC Services Rules
Web cache VIRTUALIZED SERVICES VLAN IP QoS ACLs
Intrusion Detection
Intrusion Detection
Application Delivery Controller
Application Delivery Controller Storage Firewall
Firewall FIREW ALL Security Policy
VIRTUALIZED SERVICES VLAN IP QoS ACLs
FIREW ALL Security Policy
BRKCOC-2012
VIRTUALIZED SERVICES VLAN IP QoS ACLs
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
6
Where we are? •
7 ACI Fabrics, biggest has 8 Spine, 40 Leaf and 3 Controller
•
Over 70+ Virtual Network Appliance
•
Over 10K Endpoints, 9k Virtual Machines
•
Over 20 PB storage behind ACI
•
100+ BM Migrated
•
100+ Application Migrated
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
Cisco IT’s ACI Design •
Design Goals • • • •
Flexible Design with Automation in Mind
Flexible Topology Any workload anywhere Near zero downtime Enhanced Security
Components • Border Leafs Leaf Pairs
Bare Metal Compute
Virtual Compute
multiple IP Storage Leaf Pairs 42
42
42
41
41
41
42
42
41
41
40
40
39
39
38
38
37
37
37
36
36
36
42 42
42
42
42
42
42
42
42 42
41 41
41
41
40
40
40
40
39
39
39
39
38
38
38
38
38
37
37
37
37
37
36
36
36
36
41
41
41
41
40
40
39
39
38
37
36
36
40
40
40
39
39
39
40
38
39
38
38
37
37
37
36
36
36
36
38
35
35
35
35
35
34
34
34
34
34
34
33
33
33
33
33
33
35
35
35
34
34
34
34
34
34
34
34
33
33
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
31
31
35
NAS CDOT
32
32
32
32
31
31
31
31
31
31
30
30
32
30
30
30
29
29
35
35
35
29
29
28
28
29
38
37
37
36
36
27
27
26
26
25
26
26
25
25
25
24
24
24
24
23
23
23
23
23
22
22
22
22
22
24
32
32
31
31
30
30
29
29
29
29
28
28
28
27
27
27
26
26
26
26
25
25
25
25
30
30
34
35
35
33
34
34
30
30
32
32
33
31
31
32
32
31
31
21
30
27
26
26
26
26
25
25
25
25
29
29
28
28
28
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
24
23
23
23
23
23
23
22
22
22
22
22
22
22
21
20
20
19
19
24
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
18
18
17
17
15
15
15
15
15
14
14
14
14
14
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
21
21
21
20
20
20
20
20
20
19
21
19
19
19
19
21
19
27
27
25
26
26
18
18
18
18
18
18
17
17
17
17
17
17
24
42
41
41
40
40
39
39
39
42
41
•
40
39
38
38
38
37
37
37
37
36
36
38
36
35
34
36
36
35
35
34
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
34
33
33
32
32
35
35
34
34
33
33
32
32
31
31
31 31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
28
25 25
24
24
23
23
21
21
22
22
20
20
21
21
20
25
25
25
31
30
30
29
29
30
24
24
24
24
23
23
23
23
20
22
22
22
30
29
28
28
27
27
27
26
26
26
26
25
24
23
22
29
28
27
28
25
24
23
22
19
25
25
23
22
16
16
16
15
15
15
15
15
15
15
14
14
14
14
14
14
14
16
13 13
16
16
18
18
19
19
17
17
18
18
16
16
16
25
24
24
23
23
22
22
21
21
25
24
23
25
24
23
22
21
21
21
21
21
21
20
20
20
20
20
20 20
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
12
12
11
11
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
9
9
8
8
7
7
6
6
5
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
17
15
16
16
14
15
15
5
5
5
4
4
4
4
3
3
3
2
2
2
1
1
1
5
5
5
4
4
4
3
3
3
2
2
2
1
1
1
13
14
14
12
12
13
13
11
11
12
12
10
10
11
11
9
9
10
10
8
9
9
7
7
8
8
6
6
7
7
5
6
6
4
5
3
4
2
3
1
2
2
1
1
8
5
4 4
10
19
19
19
19
19
19
18
18
18
18
18
18
17 17
15
14
13
7
42
41
40
39
38
37
36
35
34
30
28
26
23
22
21
14
13
7
42
41
40
39
38
37
36
35
34
29
27
26
24
15
16
15
7
42
41
40
39
38
37
36
35
34
20
22
22
21
21
20
20
18
14
7
42
41
40
39
38
37
36
35
34
31
30
29
27
25
24
19
L3 IP out to IP Core
30
29
28
27
24
23
22
24
24
27
24
23
21
Netapp backend network
42
41
40
39
38
37
35
34
33
30
29
28
27
27
24
21
42
41
40
38
37
28 28
27
42
41
36 35
34
33
29
28
27
26
25
23
22
24
42
40
39
38
36
35
28
29
28
27
26
25
27
39
37
36
30 30
29
28
41
40
38
37
35
32
42
41
40
39 38
35
42
40
39
39 39
38
37
36
41
40 40
38
37
42
41 41
40
39
5
4
17
17
17
17
19
19
19
18
18
18
17
16
16
16
16
16
15
15
15
15
15
14
14
14
14
14
19
18
17
16
15
17
16
16
15
15
14 14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
5
14
17
17
16
16
15
14
15
14
13
13
13
12
12
12
12
13
11
11
11
11
10
10
10
10
9
9
9
9
8
8
8
8
7
7
7
7
6
6
6
6
5
5
4
4
3
3
2
2
1
1
5
5
3 3
3
2
2
1
1
3
2
1
Citrix VPX
4
3
Citrix VPX APIC
APIC
APIC
BRKCOC-2012
4
3
4
3
2
2
1
1
• • •
ACI: 9508, 9396PX, 93180YCEX, APIC UCS: B420m3, B200m4, B200m3 ESX + AVS NetApp CDoT Citrix VPX & ASAv
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
Logical Mapping EPG to BD to Subnets to VRFs to External (L3 Out) Tenant 1
Tenant 2
Tenant 3
Internet
Tenant Common EPG-DMZ
EPG-11
EPG-21
EPG-31 EPGInternet
1.1.1.0/24 2.2.2.0/24
DC Core (External)
BD-Ext-1 VRF-dmz BD-Ext-2 EPG-12
EPG-22
3.3.3.0/24
9396 9396
EPG-32 4.4.4.0/24 BD-sec-1
VRF-Secured
DC Core (Internal) BD-int-1 5.5.5.0/24 EPG-13
EPG-23
EPG-33
VRF-Int EPG-Corp EPGOther-DC
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
ACI Security Architecture
Tenant Common
Private Cloud view
Tenant 1
EPGs consume or provide directly Foundational Services Contracts
EPG-11
Tenant 2 EPG-21
C
Contractsspecific
C
EPG-Internet
Contractsspecific
C
EPG-Corporate
Contracts-OCM
P
EPG-OCM
Contracts-OAM
P
EPG-OAM
Contract-LDAP
P
EPG-LDAP
Contract-…
P
Tenant 3
L3 Out
C
C
P EPG-13
P EPG-23
P
EPG-32
VRF Level
EPG-22
EPG-33
C C C C
Infra Services Contracts
P EPG-12
EPG-…
L3-out
ContractMonitoring
C
EPG-Monitoring
Contract-NTP
P
EPG-NTP
Contract-DNS
P
EPG-DNS
Contract-PostOSInstall
P
EPG-PostOSInstall
Contracts-…
P
EPG--…
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
ACI Consumption Model
Consume Provide
Syslog
DNS
NTP
Infra Service bundle
External Network LoadBalancer
Firewall
APP EPG
SQL Contract
Web EPG
Java Contract
Web Contract
Firewall
LoadBalancer
DB EPG
Internal Network Management Bundle
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
Any workload anywhere
Virtual Load Balancer
40+ Citrix VPX deployed on UCS 1 VNIC for mgmt, 9 VNIC for data Each VNIC maps to one EPG on ACI Fabric Spine Switches
WAF deployment for DMZ services
Leaf Switches Mgmt vNIC
Data vNIC
Mgmt vNIC
Data vNIC
NS1000v Citrix VPX (Active) (active)
ESXi host 1
NS1000v Citrix VPX (Standby) (standby)
ESXi host 2
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
Consume Provide Bindings
Citrix VPX & ACI Tenant 1
Tenant 2 VIP & sNAT
External Network
VIP EPG
Web Contract
VIP EPG
Contract
Contract
Server-1
Server-1
Server-2
Server-2
Web-EPG
App-EPG
Citrix VPX Citrix VPX
VIP-BD1 (10.10.10.1/24)
BD-VPX-Mgmt (10.100.11.1/24)
EPG within Fabric VIP + sNAT Citrix VPX Citrix VPX
VIP-BD2 (20.20.20.1/24) Tenant Common
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
Firewall Design
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
FW Architecture
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
Firewall Traffic Flow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
Firewall Traffic Flow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
Day 2 with ACI
Day 2 with ACI Automation
Maintenance • Software • Hardware • Organic Growth
• Greenfield • Brownfield Migrations • Operations
Troubleshooting
Monitoring • Enterprise Monitor • Splunk - Syslog • Programmability
• VM Connectivity loss • Contract Issue • Vlan Duplication
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Automation
Greenfield Provisioning
Brownfield Migration
Operations
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
Automation Greenfield Application Developers
Cloud Orchestration
DC Resources
Automation Packs C
O A M / S S O
C P r o x y
TCP: *,443
C
W e b
C
S e r v i c e
M e m c a c h e
Internet
C
C O r a c l e D B
C
I n t e g r a t i o n
Self-describing
C
Self-Service Catalog
Provisioning Automation
R a b b i t M Q
C a s s a n d r a
E l a s t i c S e a r c h
Packaging
Model
Click
Manageability
Deploy Fault-tolerant
BRKCOC-2012
Self-optimizing
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
Automation
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
Automation Brownfield Migration Compute Storage
95%
Network
70%
ACI Portal Unmapped Hosts
60%
Booking ID
App. View
Booking Tool
NW View
Portal Admin
Metrics
100%
CMDBServiceNow
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Migration Process
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
Migration Brownfield Migration 1
Build the ACI Fabric in Parallel
2
Install Seed Compute/Storage into ACI Fabric
3
Connect Fabric to Traditional Network
4
Configure EPGs, Contracts, etc.
5
Move Applications to ACI
6
Move Compute/Storage Unit
7
Iterate: Identify next App(s) to set up and migrate
Traditional
ACI
Application Profile
Infrastructure
LIF LIF
Compute
Nexus
Storage
ASA
ACE
BRKC0C-2012
Seed Compute
ACI SW
Seed Storage
ASAv
Netscaler
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
Operations
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Maintenance
Software Upgrade
Hardware Upgrade
Network Maintenance
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
• Read the release notes • Export the backups • Download Firmware from controller
Hitless Upgrades
• Create Firmware upgrade group for odd and even
• Upgrade APIC • Upgrade odd Leaf and Spine • Upgrade even leaf and Spine
ACI Fabric ACI Switch Border Leafs Leaf Pairs
Multiple Compute & Services Leaf Pairs
multiple IP Storage Leaf Pairs
42 42
42
42
42
42
42
41
40
39
39
38
38
37
37
36
36
35
41
41
40
40
39
39
38
38
37
35
37
37
36
36
35
35
34
34
34
34
33
33
33
33
36
36
35
35
34
34
33
33
39
38
38
37
37
36
36
Internal
IP Network
Cisco+Corp
Internet
NAS
NAS
32
32
32
32
32
31
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
24
24
24
24
24
24
23
23
23
23
23
23
22
22
22
22
22
22
21
21
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
18
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
Compute
APIC
41
41
41
40
40
40
39
39
39
39
38
38
38
38
38
37
37
37
37
37
40
39
38
37
36
36
36
36
36
36
35
35
35
35
35
35
42
35
35
34
34
33
33
32
32
31
31
30
30
29
29
34
34
34
34
34
34
33
33
33
33
33
33
32
32
32
32
32
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
37
28
27
26
25
25
24
24
23
23
22
22
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
25
24
24
24
24
24
24
32
23
23
23
23
23
22
22
22
22
22
21
21
21
21
21
21
21
20
19
19
20
20
20
20
20
20
19
19
19
19
19
19
18
18
17
17
27
18
18
18
18
18
17
17
17
17
17
16
16
15
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
22
13
13
13
13
13
13
12
12
12
12
12
12
13
12
12
11
11
10
9
9
8
8
7
7
6
6
5
5
4
4
3
3
2
2
1
1
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
8
8
8
8
8
7
7
7
7
17
Application Virtual Switch
15
14
13
12
12
11
11
10
10
9
9
8
8
7
7
9
8
7
7
6
6
6
6
6
6
5
5
5
5
5
4
4
4
4
4
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
Services (physical or virtual)
18
16
13
10
19
16
14 14
13
20
17
15
14
23
21
18
15
24
21
19
18
25
22
20
17
28
26
23
20
29
26
24
21
30
27
25
22
33
31
28
26
34
31
29
27
35
32
30
28
38
36
33
31
39
36
34
32
41
40
37
35
28
25
Services (physical or virtual)
41
40
39
38
23
DMZ
42
41
41 41
40
39
39
32
28
42
42
41
40
39
37
40
41
40
38 38
42
42
41 41
40
40
39
42
42 42
41
40
6
6
5
5
4
4
3
3
2
2
1
1
5
4
3
ESXi/Vmware
Compute
APIC
APIC vCenter
BRKCOC-2012
UCS Compute (B-Series) © 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
Hardware Upgrades Traditional DC Network
ACI Fabric
DC Layer 3 Core
Layer 2
Layer 2
Layer 2
Layer 2
DC wide mobility domain
Limited mobility domains BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
Resource Stripe - ACI Fabric leafpair 2
leafpair1
leafpair 4
leafpair 3
42
42
42
42
42
42
41
41
41
41
41
41
40
40
40
40
40
40
39
39
39
39
39
39
38
38
38
38
38
38
37
37
37
37
37
37
36
36
36
36
36
36
35
35
35
35
35
35
34
34
34
34
34
34
33
33
33
33
33
33
32
32
32
32
32
32
31
31
31
31
31
31
30
30
30
30
30
30
29
29
29
29
29
29
28
28
28
28
28
28
27
27
27
27
27
27
26
26
26
26
26
26
25
25
25
25
25
25
24
24
24
24
24
24
23
23
23
23
23
23
22
22
22
22
22
22
21
21
21
21
21
21
20
20
20
20
20
20
19
19
19
19
19
19
18
18
18
18
18
18
17
17
17
17
17
17
16
16
16
16
16
16
15
15
15
15
15
15
14
14
14
14
14
14
13
13
13
13
13
13
12
12
12
12
12
12
11
11
11
11
11
11
10
10
10
10
10
10
9
9
9
9
9
9
8
8
8
8
8
8
7
7
7
7
7
7
6
6
6
6
6
6
5
5
5
5
5
5
4
4
4
4
4
4
3
3
3
3
3
3
2
2
2
2
2
2
1
1
1
1
1
1
42
42
41
41
40
40
39
39
38
37
36
36
35
35
34
34
33
32
31
31
30
30
28
29
28
27
27
26
26
25
25
24
24
23
23
22
22
21
21
20
20
19
19
18
18
17
17
16
16
15
15
14
13
13
12
11
11
10
10
9
9
8
8
7
7
6
6
5
5
3
Resource pools and applications striped across multiple pods
14
12
4
Easier Maintenance
33
32
29
38
37
Greater Resiliency
Infrastructure “hold-back”: 13%
4
3
2
2
1
1
BRKC0C-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
Network Maintenance
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
34
Network Maintenance Traditional
Time for Rack, Stack & Patch
Basic Configuration Build Code Upgrade & Configuration Push
Same
ACI Same
60 mins
5 mins
30 mins
10 mins Auto Upgrade & Configuration Push
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
Monitoring
Enterprise Monitor
Syslog
Homegrown Automation
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
36
Enterprise Monitor
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
Splunk:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
Capacity Dashboard Operations Capacity Dashboard
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
Monitoring Scripts
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
40
Troubleshooting
Connectivity Issues
VM Outage
Endpoint Tracking
ACI Operational Best Practices BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
Identify Contract Issues - GUI
173.x.x.x
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
42
Troubleshooting Wizard - Contracts
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
43
Contract Checker
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
Identify Contract Issues - CLI
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
45
VM Outage:
Condition •
This issue is trigged when we have an overlapping subnet configured in the Bridge Domain and L3Out External Network
Symptom • Lost Connectivity to VM • Endpoint learning flaps • MAC learned on the leaf Solution •
Remove the subnets from the L3Out networks.
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
VM Outage: Condition •
Stale endpoint after changing BD VRF association
•
Ingress Policy Feature can impact existing endpoints
Symptom •
Intermittent connectivity to VM
•
Stale Endpoint
Solution •
Clear the stale endpoint entry
•
Intermediate VRF
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
47
Visore – Its Italian for Viewer ! https://fabric-apic/visore.html
admin@apic1:~>moquery -c l3extSubnet | grep 10.40.125.50 admin@apic1:~>moquery -c fvCEp | grep 10.40.125.50 BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
Class Lookup
Current Screen:insieme.stromboli.layout.Tab [fv:infoAEPg:center:b ] | Current Mo:insieme.stromboli.model.def.fvAEPg [uni/tn-common/ap-ceph/epg-ceph-cluster-01 ] BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
Homegrown Script:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
VLAN Mapping for Endpoint
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
Endpoint Tracking
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
52
EP Tracker Operations EP Tracker
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
Find VLAN Script: Bare-metals moquery -c fvRsPathAtt -f "fv.RsPathAtt.encap==\"vlan-158\""
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
Iping:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Itrace:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
Itrace vs Traceroute:
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Itrace: SPINE-1 TO LEAF 1042
SPINE-2 TO LEAF 1042
SPINE-1 TO LEAF 1042
SPINE-2 TO LEAF 1042 BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Network Engineer Skillset
Python
CLI
GUI
SDK / Toolkit
API
Management information tree APIC
CLI
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Lessons Learned
Lessons Learned – Design & Implementation •
Feature and Code certification process
•
Document naming conventions for various objects to make readability and troubleshooting easier
•
Traffic segregation [External Network]
•
Consistent naming convention & configuration
•
Dedicated leaf pairs for different work loads/function
•
Avoid full mesh contracts where possible
•
Use different contracts on each VRF
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Lessons Learned - Operations •
Check health scores to narrow down affected scope
•
Get familiar with reviewing faults and audit logs. If anything fails deployment then faults are raised
•
Get familiar with utilizing CLI and APIC (possibly Visore or MOQuery)
•
Check the resolved object model is present on all the APIC and relevant leafs
•
Check the concrete objects are present on the relevant leafs
•
Leverage tools like Endpoint Tracker and Troubleshooting Wizard
•
Document/Track issues faced
•
Change Management Review Process
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Cheat Sheet
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
References Title
Links
Cisco IT ACI Design
Cisco IT ACI Design
Cisco IT AVS & Vmware - Vsphere Distributed Switch Failover Convergence
Cisco IT AVS & Vmware DVS Case Study
Cisco IT Storage Design
Cisco IT ACI Storage Design
Cisco IT Compute at Scale on Cisco ACI
Cisco IT Compute at Scale on Cisco ACI
Cisco IT ACI Migration
Cisco IT ACI Migration
Tetration Analytics : IDC Business Value
Cisco IT case study
ACITool Kit
https://github.com/datacenter/acitoolkit/
DevNet
https://developer.cisco.com
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Continue Your Education •
Demos in the Cisco campus
•
Walk-in Self-Paced Labs
•
Lunch & Learn
•
Meet the Engineer 1:1 meetings
•
Related sessions
BRKCOC-2012
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
Complete Your Online Session Evaluation •
Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 gift card.
•
Complete your session surveys through the Cisco Live mobile app or on www.CiscoLive.com/us. Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at www.CiscoLive.com/Online.
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco on Cisco Breakout Sessions SESSION
SPEAKER(s)
BRKCOC-2006 - Inside Cisco IT: ACI & Tetration Analytics
Anitha Parimi & Laxmi Vellanki
BRKCOC-2019 - Inside Cisco IT: Leveraging Cisco WAAS to Improve Network Performance
Rubens Lima & Hara Bandhakavi
BRKCOC-2016 - Inside Cisco IT: Containers on Enterprise Compute and Networks
Michael Duarte
BRKCOC-2014 - Inside Cisco IT: Increasing the Speed of Business using AppDynamics
Yatin Wadhavkar & Clement Joseph
BRKCOC-2012 - Inside Cisco IT: A Day in the life of a Network Engineer - Day 2 with ACI
Mani Govindasamy & Ishan Mehta
BRKCOC-2018 - Inside Cisco IT: How Cisco Deployed ISE and TrustSec throughout the Enterprise
David Iacobacci
BRKCOC-2013 - Inside Cisco IT: Embedding Collaboration in Business Workflows using Cisco Spark
Paul Anholt & David deMilo
BRKCOC-2023 - Inside Cisco IT: Security Overview - Making it Work
Gil Daudistel
BRKCOC-2021 - Inside Cisco IT: DNA and the Next Generation Network
John Moe
BRKCOC-2017 - Inside Cisco IT: Using Machine Learning Technologies to Drive Digital Transformation
PLAMEN NEDELTCHEV
BRKCOC-2012
& Bassem Khalife
© 2017 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
Thank you
