50
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
1 <!DOCTYPE html>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
2
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
3 <!--
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
4 Google HTML5 slide template
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
5
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
6 Authors: Luke Mahé (code)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
7 Marcin Wichary (code and design)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
8
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
9 Dominic Mazzoni (browser compatibility)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
10 Charles Chen (ChromeVox support)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
11
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
12 URL: http://code.google.com/p/html5slides/
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
13 -->
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
14
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
15 <html>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
16 <head>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
17 <title>slide</title>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
18
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
19 <meta charset='utf-8'>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
20 <script
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
21 src='http://html5slides.googlecode.com/svn/trunk/slides.js'></script>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
22 </head>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
23
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
24 <style>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
25 /* Your individual styles here, or just use inline styles if that’s
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
26 what you want. */
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
27 .slides article { background-image: none !important; background-color: white; }
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
28
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
29 </style>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
30
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
31 <body style='display: none'>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
32
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
33 <section class='slides layout-regular template-default'>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
34
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
35 <!-- Your slides (<article>s) go here. Delete or comment out the
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
36 slides below.-->
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
37
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
38 <article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
39 <h1>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
40 Cerium の並列処理向け I/O の設計と実装
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
41 </h1>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
42 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
43 Masataka Kohagura
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
44 <br>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
45 21st January , 2014
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
46 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
47 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
48
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
49 <article class='smaller'>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
50 <h3>研究背景と目的</h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
51 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
52 近年のCPUのほとんどはマルチコアであり、それらの性能を引き出すためには並列プログラミングが必須となっている。そこで当研究室では Cerium Library の開発を行い、提供することによって並列プログラミングを容易にしている。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
53 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
54 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
55 先行研究では Task の並列化によって、プログラム全体の処理速度は向上しているが、ファイル読み込み等の I/O に対して並列に Task が動作するようにはされていない。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
56 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
57 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
58 現状では、ファイルを全て memory に mapping を行ってから Task が走るようになっているので、非常に大きいサイズのファイルを読み込むと、ファイルを memory に mapping するまでの時間がオーバーヘッドになってしまう。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
59 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
60 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
61 本研究では I/O と Task の並列化の設計、実装によってプログラム全体の処理速度、処理効率を上げていく。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
62 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
63 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
64
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
65 <article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
66 <h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
67 したこと
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
68 </h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
69 <ul>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
70
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
71 <li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
72 2GB 以上のファイルが読み込めなかった -> 修正
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
73 </li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
74
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
75 <li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
76 fileread をちょっとだけ測定
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
77 </li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
78 <ul>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
79 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
80
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
81 <!--
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
82 <article class='smaller'>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
83 <h3>I/O並列化のシーケンス図(mmap)</h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
84 <div align="center">
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
85 <IMG SRC="mmap.png">
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
86 </div>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
87 <li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
88 codeがシンプル(readを書いて読み込まなくていいため)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
89 </li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
90 <li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
91 memoryより大きなファイルは開けない
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
92 </li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
93 <li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
94 readの先読みがOS依存
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
95 </li>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
96
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
97 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
98 -->
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
99
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
100 <article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
101
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
102 <h3> 2GB 以上のファイルが読み込めなかった -> 修正 </h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
103 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
104 manager->allocate(int) を使用したところ
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
105 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
106 <section><pre>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
107 if (divide_read_flag != 0) {
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
108 printf("[divide read mode]\n");
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
109 <font color="red">w->file_mmap = (char*)manager->allocate((long)w->read_filesize);</font>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
110 r_run = manager->create_task
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
111 (RUN_READ_BLOCKS, (memaddr)&w->self, sizeof(memaddr),0,0);
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
112 }else {
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
113 printf("[mmap mode]\n");
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
114 r_run = manager->create_task
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
115 (MMAP , (memaddr)&w->self, sizeof(memaddr),0,0);
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
116 }
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
117 </pre></section>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
118 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
119 divide read をする前に manager->alocate にて memory を確保する。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
120 この領域に、分割して読み込んだ file を mapping していく。
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
121 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
122 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
123
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
124 <article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
125 <h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
126 kernel/ppe/TaskManagerImpl.h
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
127 </h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
128 <p>修正前</p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
129
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
130 <section><pre>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
131 void* allocate(int size, int alignment)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
132 {
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
133 #if defined(__SPU__) || ! defined(HAS_POSIX_MEMALIGN)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
134 buff = malloc(size);
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
135 #else
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
136 posix_memalign(&buff, alignment, size);
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
137 #endif
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
138 return buff;
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
139 }
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
140 </pre></section>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
141
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
142 <p>修正後</p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
143 <section><pre>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
144 void* allocate(<font color="red">long size</font>, int alignment)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
145 {
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
146 ・・・
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
147 }
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
148 </pre></section>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
149
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
150 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
151
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
152 <article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
153 <h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
154 fileread
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
155 </h3>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
156 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
157 ファイルの読み込みだけを行うプログラム
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
158 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
159 <p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
160 firefly 上で実行<br>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
161 2*2.66 GHz, HHD 1TB, memory 16GB<br>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
162 file size : 10740996168 Byte (約10GB)
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
163 </p>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
164
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
165 <table>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
166 <tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
167 <table>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
168 <tr><td>mode</td><td>time(s)</td></tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
169 <tr><td>mmap</td><td>0.194788</td></tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
170 <tr><td>read_divide_size</td><td></td></tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
171 <tr><td>4*4096 Byte</td><td>391.734989</td></tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
172 <tr><td>4*4096*1024 Byte</td><td>123.563726</td></tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
173 </table>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
174 <tr>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
175 </table>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
176 </article>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
177
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
178 </body>
|
Masataka Kohagura <e085726@ie.u-ryukyu.ac.jp>
parents:
diff
changeset
|
179 </html>
|