LOGICAL @
US007451285B2
(12) Ulllted States Patent
(10) Patent N0.:
Maki et a]. (54)
(75)
US 7,451,285 B2
(45) Date of Patent:
COMPUTER SYSTEMS, MANAGEMENT
7,130,975 B2 *
Nov. 11, 2008
10/2006 Suishu et a1. ............. .. 711/162
COMPUTERS AND STORAGE SYSTEM
2003/0229764 A1 *
12/2003
Ohno et a1. ............... .. 711/147
MANAGEMENT METHOD
2005/0050115 A1 *
3/2005
Kekre ....................... .. 707/204
2007/0198790 A1 *
8/2007
Asano et a1. .... ..
Inventorsi Nobllhiro Maki, Yokohama (JP);
2007/0198791 A1 *
8/2007 lwamura et a1. ........... .. 711/162
7ll/l62
Makoto Uchikado, Yokohama (JP); Kazuhiko Watanabe, Yokohama (JP);
Yuri HiraiWa, Sagamihara (JP) FOREIGN PATENT DOCUMENTS
(73) Assignee: Hitachi, Ltd., Tokyo (JP) JP
(*)
Notice:
2005-18506
1/2005
Subject to any disclaimer, the term of this patent is extended or adjusted under 35
U.S.C. 154(b) by 190 days. * cited by examiner
(21) App1.No.: 11/485,971 (22)
Filed;
Primary ExamineriReginald G Bragdon
JUL 14, 2006
Assistant ExamineriMehdi NamaZi
(74) Attorney, Agent, or FirmiMattingly, Stanger, Malur & Brundidge, RC.
(65)
Prior Publication Data
(30)
Foreign Application Priority Data
May 16, 2006
(JP)
........................... .. 2006- 1 36239
A computer includes a management computer connected to a
(51)
Int- ClG06F 12/00
(2006-01)
host computer, ?rst and second primary storage systems, ?rst and second secondary storage systems, a primary external storage system connected to the ?rst and second primary
(52) (58)
US. Cl. ..................................... .. 711/162; 71 l/l6l Field of Classi?cation Search ............... .. 711/162, 71 1/ 161
storage systems and a secondary external storage system connected to the ?rst and second secondary storage systems. The management computer controls to start copying as pro
See application ?le for complete search history.
cessing for storing copied data in the second storage area into
_
(56)
a fourth storage area in response to a command for starting
References Cited
processing for storing copied data in the ?rst storage area into a third storage area.
U.S. PATENT DOCUMENTS 6,209,002 B1 *
3/2001
Gagne et a1. .............. .. 707/204
6,745,303 B2 *
6/2004
Watanabe ................. .. 7ll/l6l
1601
21 Claims, 24 Drawing Sheets
CONSISTENC‘I GROUP
LOGICAL VOLUME
3501 LOGICAL VOLUME
1602
3502
LOGICAL
@
VOLUME
VOLUME
$10006
3000a~
3503 -
VOLUME PRIMARY STORAGE SYSTEM
SEOOND STORAG SYSTEM
1603 LOGICAL
_/—
1604 VIRTUAL
REMOTE comm; m DEFINITION
IUNPROCESSEDI
/ VOLUME
3504
VIRTUAL
/ VOLUME 4000
PRIMARY STORAGE SYSTEM
SECONDARY STORAGE SYSTEM
2001 -
2°00“
1605 VIRTUAL VOLUME
gsRlrlém STORAGE
(PRIMARY DEPUTY
VOLUME
3505
PRIMARY
‘
EXTERNAL STORAGE SYSTEM DEPUTY INITIAL
COPYING
STORAGE SYSTEM) "4000c PRIMARY SITE
4001 -
LOGICAL
VIRTUAL
VOLUME
DEPUTY STORAGE SECO
QOODC'» I) EPUTY
RY
STORAGE SYSTEM) SECONDARY SITE
LOGICAL VOLUME
SECONDARY EXTERNAL
gtgagi
US. Patent
Nov. 11,2008
f
Sheet 1 0f 24
US 7,451,285 B2
100
200
200
S
S
S
PRIMARY MANAGEMENT COMPUTER
PRIMARY HOST COMPUTER
PRIMARY HOST COMPUTER
500
500
1 000a
1000b
5
5
PRIMARY
SET VALUE 1?
$5710 PERFORMANCE DIFFERENCE > 0
REMOVE PERTINENT DEPUTY STORAGE
w S5720
SYSTEM FROM CANDIDATE
vv4 S5730 PERFORMANCE
(CACHE OCCUPANCY RATE) > SET VALUE 2'?
ADD SET VALUE 3 TO DEPUTY TOTAL OF
S5740
PERTINENT DEPUTY STORAGE SYSTEM
N
85750 ALL DEPUTY
STORAGE SYSTEMS CONFIRMED?
NO
US. Patent
Nov. 11,2008
Sheet 14 0124
US 7,451,285 B2
F|G.21 DEPUTY COPYING PROCEDURE OF STORAGE SYSTEM
@ MONITORING OF WRITE DATA OF
w S5800
VIRTUAL STORAGE SYSTEM
T STARTING OF DEPUTY COPYING BY
m S5810
DEPUTY STORAGE SYSTEM
T ENDING OF DEPUTY COPYING BY DEPUTY sTORAGE SYSTEM
V“ 55820
STARTING OF REGULAR COPYING
w S5830
BY VIRTUAL sTORAGE SYSTEM
@
US. Patent
Nov. 11,2008
Sheet 17 0f 24
US 7,451,285 B2
FIG.24 ENDING PROCESSING OF DEPUTY COPYING BY DEPUTY STORAGE SYSTEM
PRIMARY STORAGE SYSTEM 1000b
I
START
SECONDARY STORAGE SYSTEM 3000b
I
I
START
I
II
TRANSFER DATA THAT IS NOT YET
N 86100
COPYING-TRANSFERRED IN SYSTEM ITSELF TO STORAGE SYSTEM
II
DELETE CONTENTS OF COPYING PAIR
~ S6110
MANAGEMENT INFORMATION AND END
COPYING PROCESSING
S6120
\
I
I
WRITE RECEIVED DATA INTO CACHE PART
S6130
II
S
WRITE DATA IN DIRTY STATUS TEMPORARILY HELD IN CACHE PART OF SYSTEM ITSELF INTO SECONDARY EXTERNAL STORAGE
SYSTEM
END
END
US. Patent
Nov. 11,2008
Sheet 18 0124
US 7,451,285 B2
FIG.25 STARTING PROCEDURE OF REGULAR COPYING BY STORAGE SYSTEM
PRIMARY STORAGE SYSTEM 1000a
I
START
SECONDARY STORAGE SYSTEM 3000a
I
I
START
I
II
READ OUT DIRTY CACHE
~ 56200
INFORMATION sTORED IN SHARED MEMORY II
READ OUT RECORDED DATA FROM
~ S6210
CACHE PART OR DISK DRIVE WITH REFERENCE TO DIRTY CACHE INFORMATION II
CONVERT READ DATA INTO FORMAT
~ S6220
OF WRITE DATA MANAGEMENT
S6230
v
INFORMATION AND TRANSFER IT TO
/
\
SECONDARY STORAGE SYSTEM
ANALYZE RECEIVED WRITE DATA MANAGEMENT INFORMATION AND
~
WRITE DATA INTO LOGICAL VOLUME
S6240 ALL DATA REGISTERED IN DIRTY CACHE
INFORMATION TRANSFERRED?
COPY RECEIVED WRITE DATA AND CONVERT IT INTO WRITE DATA
S6250 N
MANAGEMENT INFORMATION FORMAT II
TRANSFER IT TO SECONDARY
STORAGE SYSTEM
~ S6260
\
"
S6270
/
ANALYZE REcEIvED WRITE DATA MANAGEMENT INFORMATION AND WRITE DATA INTO LOGICAL VOLUME
END
END
US. Patent
Nov. 11,2008
Sheet 19 0124
US 7,451,285 B2
100
S PRIMARY STORAGE COMPUTER
1510 MEMORY
117
I 115 w DEP
STOHXEE TABLE
116
MANAGEMENT
SWITCHING PROGRAM
vIRTuAL 111 w
STORAGE
PATH
w
112
PROGRAM
EXTERNAL
COPY DEFINITION $113
RELATION INFORMATION
TABLE
M DEP TY
DEFRHTISS
PY
STORAGE SYSTEM
INFORMATION
TABLE
“*1 14
TABLE
I/O PROOEssING UNIT
PROOEssOR
120
130
200
S 210
PRIMARY HOST OOMPuTER
S 211
212
213
S AP
S 05 ]
5
PROOEssOR
220
PATH SWITCHING PROGRAM
l/O PROOEssING UNIT
230
(12) Ulllted States Patent
(10) Patent N0.:
Maki et a]. (54)
(75)
US 7,451,285 B2
(45) Date of Patent:
COMPUTER SYSTEMS, MANAGEMENT
7,130,975 B2 *
Nov. 11, 2008
10/2006 Suishu et a1. ............. .. 711/162
COMPUTERS AND STORAGE SYSTEM
2003/0229764 A1 *
12/2003
Ohno et a1. ............... .. 711/147
MANAGEMENT METHOD
2005/0050115 A1 *
3/2005
Kekre ....................... .. 707/204
2007/0198790 A1 *
8/2007
Asano et a1. .... ..
Inventorsi Nobllhiro Maki, Yokohama (JP);
2007/0198791 A1 *
8/2007 lwamura et a1. ........... .. 711/162
7ll/l62
Makoto Uchikado, Yokohama (JP); Kazuhiko Watanabe, Yokohama (JP);
Yuri HiraiWa, Sagamihara (JP) FOREIGN PATENT DOCUMENTS
(73) Assignee: Hitachi, Ltd., Tokyo (JP) JP
(*)
Notice:
2005-18506
1/2005
Subject to any disclaimer, the term of this patent is extended or adjusted under 35
U.S.C. 154(b) by 190 days. * cited by examiner
(21) App1.No.: 11/485,971 (22)
Filed;
Primary ExamineriReginald G Bragdon
JUL 14, 2006
Assistant ExamineriMehdi NamaZi
(74) Attorney, Agent, or FirmiMattingly, Stanger, Malur & Brundidge, RC.
(65)
Prior Publication Data
(30)
Foreign Application Priority Data
May 16, 2006
(JP)
........................... .. 2006- 1 36239
A computer includes a management computer connected to a
(51)
Int- ClG06F 12/00
(2006-01)
host computer, ?rst and second primary storage systems, ?rst and second secondary storage systems, a primary external storage system connected to the ?rst and second primary
(52) (58)
US. Cl. ..................................... .. 711/162; 71 l/l6l Field of Classi?cation Search ............... .. 711/162, 71 1/ 161
storage systems and a secondary external storage system connected to the ?rst and second secondary storage systems. The management computer controls to start copying as pro
See application ?le for complete search history.
cessing for storing copied data in the second storage area into
_
(56)
a fourth storage area in response to a command for starting
References Cited
processing for storing copied data in the ?rst storage area into a third storage area.
U.S. PATENT DOCUMENTS 6,209,002 B1 *
3/2001
Gagne et a1. .............. .. 707/204
6,745,303 B2 *
6/2004
Watanabe ................. .. 7ll/l6l
1601
21 Claims, 24 Drawing Sheets
CONSISTENC‘I GROUP
LOGICAL VOLUME
3501 LOGICAL VOLUME
1602
3502
LOGICAL
@
VOLUME
VOLUME
$10006
3000a~
3503 -
VOLUME PRIMARY STORAGE SYSTEM
SEOOND STORAG SYSTEM
1603 LOGICAL
_/—
1604 VIRTUAL
REMOTE comm; m DEFINITION
IUNPROCESSEDI
/ VOLUME
3504
VIRTUAL
/ VOLUME 4000
PRIMARY STORAGE SYSTEM
SECONDARY STORAGE SYSTEM
2001 -
2°00“
1605 VIRTUAL VOLUME
gsRlrlém STORAGE
(PRIMARY DEPUTY
VOLUME
3505
PRIMARY
‘
EXTERNAL STORAGE SYSTEM DEPUTY INITIAL
COPYING
STORAGE SYSTEM) "4000c PRIMARY SITE
4001 -
LOGICAL
VIRTUAL
VOLUME
DEPUTY STORAGE SECO
QOODC'» I) EPUTY
RY
STORAGE SYSTEM) SECONDARY SITE
LOGICAL VOLUME
SECONDARY EXTERNAL
gtgagi
US. Patent
Nov. 11,2008
f
Sheet 1 0f 24
US 7,451,285 B2
100
200
200
S
S
S
PRIMARY MANAGEMENT COMPUTER
PRIMARY HOST COMPUTER
PRIMARY HOST COMPUTER
500
500
1 000a
1000b
5
5
PRIMARY
SET VALUE 1?
$5710 PERFORMANCE DIFFERENCE > 0
REMOVE PERTINENT DEPUTY STORAGE
w S5720
SYSTEM FROM CANDIDATE
vv4 S5730 PERFORMANCE
(CACHE OCCUPANCY RATE) > SET VALUE 2'?
ADD SET VALUE 3 TO DEPUTY TOTAL OF
S5740
PERTINENT DEPUTY STORAGE SYSTEM
N
85750 ALL DEPUTY
STORAGE SYSTEMS CONFIRMED?
NO
US. Patent
Nov. 11,2008
Sheet 14 0124
US 7,451,285 B2
F|G.21 DEPUTY COPYING PROCEDURE OF STORAGE SYSTEM
@ MONITORING OF WRITE DATA OF
w S5800
VIRTUAL STORAGE SYSTEM
T STARTING OF DEPUTY COPYING BY
m S5810
DEPUTY STORAGE SYSTEM
T ENDING OF DEPUTY COPYING BY DEPUTY sTORAGE SYSTEM
V“ 55820
STARTING OF REGULAR COPYING
w S5830
BY VIRTUAL sTORAGE SYSTEM
@
US. Patent
Nov. 11,2008
Sheet 17 0f 24
US 7,451,285 B2
FIG.24 ENDING PROCESSING OF DEPUTY COPYING BY DEPUTY STORAGE SYSTEM
PRIMARY STORAGE SYSTEM 1000b
I
START
SECONDARY STORAGE SYSTEM 3000b
I
I
START
I
II
TRANSFER DATA THAT IS NOT YET
N 86100
COPYING-TRANSFERRED IN SYSTEM ITSELF TO STORAGE SYSTEM
II
DELETE CONTENTS OF COPYING PAIR
~ S6110
MANAGEMENT INFORMATION AND END
COPYING PROCESSING
S6120
\
I
I
WRITE RECEIVED DATA INTO CACHE PART
S6130
II
S
WRITE DATA IN DIRTY STATUS TEMPORARILY HELD IN CACHE PART OF SYSTEM ITSELF INTO SECONDARY EXTERNAL STORAGE
SYSTEM
END
END
US. Patent
Nov. 11,2008
Sheet 18 0124
US 7,451,285 B2
FIG.25 STARTING PROCEDURE OF REGULAR COPYING BY STORAGE SYSTEM
PRIMARY STORAGE SYSTEM 1000a
I
START
SECONDARY STORAGE SYSTEM 3000a
I
I
START
I
II
READ OUT DIRTY CACHE
~ 56200
INFORMATION sTORED IN SHARED MEMORY II
READ OUT RECORDED DATA FROM
~ S6210
CACHE PART OR DISK DRIVE WITH REFERENCE TO DIRTY CACHE INFORMATION II
CONVERT READ DATA INTO FORMAT
~ S6220
OF WRITE DATA MANAGEMENT
S6230
v
INFORMATION AND TRANSFER IT TO
/
\
SECONDARY STORAGE SYSTEM
ANALYZE RECEIVED WRITE DATA MANAGEMENT INFORMATION AND
~
WRITE DATA INTO LOGICAL VOLUME
S6240 ALL DATA REGISTERED IN DIRTY CACHE
INFORMATION TRANSFERRED?
COPY RECEIVED WRITE DATA AND CONVERT IT INTO WRITE DATA
S6250 N
MANAGEMENT INFORMATION FORMAT II
TRANSFER IT TO SECONDARY
STORAGE SYSTEM
~ S6260
\
"
S6270
/
ANALYZE REcEIvED WRITE DATA MANAGEMENT INFORMATION AND WRITE DATA INTO LOGICAL VOLUME
END
END
US. Patent
Nov. 11,2008
Sheet 19 0124
US 7,451,285 B2
100
S PRIMARY STORAGE COMPUTER
1510 MEMORY
117
I 115 w DEP
STOHXEE TABLE
116
MANAGEMENT
SWITCHING PROGRAM
vIRTuAL 111 w
STORAGE
PATH
w
112
PROGRAM
EXTERNAL
COPY DEFINITION $113
RELATION INFORMATION
TABLE
M DEP TY
DEFRHTISS
PY
STORAGE SYSTEM
INFORMATION
TABLE
“*1 14
TABLE
I/O PROOEssING UNIT
PROOEssOR
120
130
200
S 210
PRIMARY HOST OOMPuTER
S 211
212
213
S AP
S 05 ]
5
PROOEssOR
220
PATH SWITCHING PROGRAM
l/O PROOEssING UNIT
230
