Mercurial > hg > Members > masakoha > masa

annotate 14/February/memo/03rd.txt @ 51:d8f499590d82

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
rename 201* to 1*
author Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
date Sun, 16 Mar 2014 13:36:04 +0900
parents 2014/February/memo/03rd.txt@37d843c14923
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
32
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
1 2014/02/03 (Mon)
29
444b0ca6ff2c add OUTLINE & 01st.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff changeset
2 [program]
32
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
3 とりあえず、manager->allocate の内部実装を int -> long long に変更した。
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
4 regex_mas 2GB 以上の file を読み込ますと、mmap と divide read の違いで結果が違ってしまう。 -> 多分、ちゃんと読み込みされていないのでは??
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
5
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
6 grep -c "doing" 2g.txt -> 5209072
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
7 ./regex -file 2g.txt -sw doing -cpu 4 -> 5209072
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
8 ./regex -file 2g.txt -sw doing -cpu 4 -dr -> 5190577
29
444b0ca6ff2c add OUTLINE & 01st.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff changeset
9
32
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
10 ./regex -file 3g.txt -sw doing -cpu 4 -> 7799680
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
11 ./regex -file 3g.txt -sw doing -cpu 4 -dr -> 5190577
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
12
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
13 やっぱり 2GB の壁くさい
30
7884e1b44177 add 02nd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 29
diff changeset
14
32
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
15 segmentation fault が起こらないということは、ちゃんと allocate はされているっぽい。
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
16 ということは、やはり read 関係のどこかで 2GB 制限がかかってるのではないか。
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
17 (API を int -> long long にかえないといけない部分がまだ存在すると思われる。
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
18
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
19 divide read mode 時、wbuf の address の動きが怪しい
07176dd2dddc add 03rd.txt
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 31
diff changeset
20 long だと MAX が約2GB。つまりどこかで long long にしないとこれは直らない。
33
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
21
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
22 read_run 内の
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
23 read->set_outData(0,w->file_mmap + w->read_task_number*w->read_division_size, w->read_division_size);
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
24 第二引数 の値がおかしくなっていることが発覚
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
25
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
26 実行結果
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
27
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
28 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
29 set_outData buf : 0x24367a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
30 w->file_mmap : 0x1c467a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
31 w->read_task_number : 127
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
32 w->read_division_size : 16777216
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
33 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
34 [read No 105 : wbuf addr 0x22d67a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
35 [read No 106 : wbuf addr 0x22e67a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
36 [read No 107 : wbuf addr 0x22f67a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
37 ・・・
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
38 [read No 120 : wbuf addr 0x23c67a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
39 [read No 121 : wbuf addr 0x23d67a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
40 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
41 set_outData buf : 0x14467a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
42 w->file_mmap : 0x1c467a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
43 w->read_task_number : 128
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
44 w->read_division_size : 16777216
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
45 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
46 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
47 set_outData buf : 0x14567a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
48 w->file_mmap : 0x1c467a000
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
49 w->read_task_number : 129
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
50 w->read_division_size : 16777216
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
51 ----------------------
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
52
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
53 再実行
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
54 [read No 124 : wbuf addr 0x24067a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
55 [read No 125 : wbuf addr 0x24167a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
56 [read No 126 : wbuf addr 0x24267a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
57 [read No 127 : wbuf addr 0x24367a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
58 [read No 128 : wbuf addr 0x14467a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
59 [read No 129 : wbuf addr 0x14567a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
60 [read No 130 : wbuf addr 0x14667a000]
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
61 876543210
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
62 16^8 = 2^4^8 = 2^32
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
63
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
64 Address が 2^32 分落ちてしまっている。
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
65 (オーバーフローしている)
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
66
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
67 原因究明
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
68 (char*)w->file_mmap + (int)w->read_task_number*(int)w->read_division_size
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
69 (int)w->read_task_number*(int)w->read_division_size
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
70
37d843c14923 resolve 2GB problem
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents: 32
diff changeset
71 int に 2GB 以上の数値を突っ込んでいたので、オーバーフローしていた