# HG changeset patch # User Nozomi # Date 1455698226 -32400 # Node ID f786ffb2a198a7c493d7b187f2c208213eafd478 # Parent 58a32f966763aeab38e3097246245804d2c26340 add bm.h diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/Func.h --- a/example/bm_search/Func.h Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/Func.h Wed Feb 17 17:37:06 2016 +0900 @@ -1,13 +1,6 @@ enum { #include "SysTasks.h" TASK_EXEC, - BREAD_RUN_TASK_BLOCKS, - MMAP_RUN_TASK_BLOCKS, - READ_TASK, - PRINT_TASK, + TASK_EXEC_DATA_PARALLEL, + TASK_PRINT, }; - -#define DATA_NUM 12 -#define ADD_NUM 26 - -#define DATA_ID 0 diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/Makefile --- a/example/bm_search/Makefile Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/Makefile Wed Feb 17 17:37:06 2016 +0900 @@ -17,7 +17,7 @@ @$(MAKE) -f Makefile.gpu test: - ./regex -file c.txt -cpu 4 -sw doing -br + ./bm_search -file c.txt -cpu 4 -sw doing -br FORCE: diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/WordCount.h --- a/example/bm_search/WordCount.h Sat Feb 06 20:05:05 2016 +0900 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,34 +0,0 @@ - -typedef struct wordCount { - struct wordCount *self; - /* read task variable */ - long fd; - long long read_filesize; - CPU_TYPE read_cpu; - - long long size; // remaining file size - long division_size; // for each word count task - long division_out_size; - long out_size; - long out_size_; - long task_num; // remaining task count - long task_blocks; // spawn task one at a time - long status_num; - long task_spawned; - unsigned long long *o_data; - long out_task_num; - char *file_mmap; - long long file_size; - - /*set input Data variables*/ - long last_task_size; - unsigned char *search_word; - int *BMskip_table; - int search_word_len; - int extra_len; - - HTaskPtr t_print; - HTaskPtr t_exec; - -} WordCount, *WordCountPtr; - diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/bm.h --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/example/bm_search/bm.h Wed Feb 17 17:37:06 2016 +0900 @@ -0,0 +1,6 @@ +typedef struct bm { + int* skip_table; + unsigned char *search_word; + int search_word_len; + struct bm *next; +} BM, *BMPtr; diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/main.cc --- a/example/bm_search/main.cc Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/main.cc Wed Feb 17 17:37:06 2016 +0900 @@ -1,18 +1,11 @@ #include -#include #include -#include -#include -#include -#include -#include #include -#include -#include #include "TaskManager.h" #include "SchedTask.h" #include "Func.h" -#include "WordCount.h" +#include "FileMapReduce.h" +#include "bm.h" /* ;TODO * PS3でCPU数が2以上の時に、あまりが計算されてない @@ -22,24 +15,7 @@ void TMend(TaskManager *); static double st_time; static double ed_time; -int all = 0; -int use_task_array = 1; -int use_task_creater = 0; -int use_compat = 0; -int div_read = 0; -int array_task_num = 8; -int spe_num = 1; -int block_read_flag = 0; -int DIVISION_SIZE = 4*4096; - -unsigned char *sword = 0; -int task_count = 0; - -static int division = 128; // in Kbyte -CPU_TYPE spe_cpu = SPE_ANY; -CPU_TYPE read_spe_cpu = IO_0; - -const char *usr_help_str = "Usage: ./word_count [-a -c -s] [-cpu spe_num] [-sw search_word] [-file filename]\n Required filename & search_word\n"; +const char* usr_help_str = ""; static double getTime() { @@ -48,375 +24,52 @@ return tv.tv_sec + (double)tv.tv_usec*1e-6; } -typedef struct { - caddr_t file_mmap; - off_t size; -} st_mmap_t; - +//Boyer Moore法に使用するテーブルを作成 static void -my_mmap(char *filename, WordCount *w) +create_BMskiptable(BMPtr bm) { - /*マッピングだよ!*/ - int map = MAP_PRIVATE; - st_mmap_t st_mmap; - struct stat sb; - long fd = w->fd; - - if ((fd=open(filename,O_RDONLY,0666))==0) { - fprintf(stderr,"can't open %s\n",filename); - } - - if (fstat(fd,&sb)) { - fprintf(stderr,"can't fstat %s\n",filename); - } - - //madvise(w->file_mmap, w->read_filesize, POSIX_MADV_NORMAL); - w->file_mmap = (char*)mmap(NULL,w->read_filesize,PROT_READ,map,fd,(off_t)0); - - if (st_mmap.file_mmap == (caddr_t)-1) { - fprintf(stderr,"Can't mmap file\n"); - perror(NULL); - exit(0); + for (int i = 0; i < 256; ++i) { + bm->skip_table[i] = bm->search_word_len; } - return ; -} - - -static void -run_tasks(SchedTask *manager, WordCount *w, int task_count,HTaskPtr t_read, HTaskPtr t_next, int size) -{ - - if (task_count < array_task_num) { - array_task_num = task_count; - if (task_count<=0) return; - } - for (int i = 0; i < task_count; i += array_task_num) { - HTask *task_array; - if (use_task_array) { - int task_num = (w->size+size-1)/size; - if (task_num>array_task_num) task_num = array_task_num; - task_array = manager->create_task_array(TASK_EXEC,task_num,1,3,1); - if (t_read != 0) task_array->wait_for(t_read); - if (!all) { - t_next->wait_for(task_array); - } else { - w->t_print->wait_for(task_array); - } - } - - Task *t_exec = 0; - HTask *h_exec = 0; - for (int j = 0; j < array_task_num; j++) { - long i = w->task_spawned++; - if (w->size < size) size = w->size; - if (size==0) break; - if (use_task_array) { - t_exec = task_array->next_task_array(TASK_EXEC,t_exec); - t_exec->set_inData(0,w->file_mmap + i*w->division_size, size); - t_exec->set_inData(1,w->search_word, w->search_word_len); - t_exec->set_inData(2,w->BMskip_table, 256); - - t_exec->set_param(0,(long)i); - - t_exec->set_outData(0,w->o_data + i*w->out_size, w->division_out_size); - } else if (use_compat) { - h_exec = manager->create_task(TASK_EXEC); - if (t_read != 0) h_exec->wait_for(t_read); - h_exec->set_inData(0,w->file_mmap + i*w->division_size, size); - h_exec->set_outData(0,w->o_data + i*w->out_size, w->division_out_size); - - t_next->wait_for(h_exec); - - h_exec->set_cpu(spe_cpu); - h_exec->spawn(); - } else { - h_exec = manager->create_task(TASK_EXEC, - (memaddr)(w->file_mmap + i*w->division_size), size, - (memaddr)(w->o_data + i*w->out_size), w->division_out_size); - if (t_read != 0) h_exec->wait_for(t_read); - t_next->wait_for(h_exec); - h_exec->set_cpu(spe_cpu); - h_exec->spawn(); - } - w->size -= w->division_size; - w->task_num--; - } - if (use_task_array) { - task_array->spawn_task_array(t_exec->next()); - task_array->set_cpu(spe_cpu); - task_array->spawn(); - } else { - //if (!all) t_next->wait_for(h_exec); - } + for (int j = 0; j < bm->search_word_len - 1; ++j) { + bm->skip_table[bm->search_word[j]] = bm->search_word_len - j - 1; } } -/** - * このTaskは、PPE上で実行されるので、並列に実行されることはない - * 二つ実行されていて、Task が足りなくなることがないようにしている。 - */ - -SchedDefineTask1(BREAD_RUN_TASK_BLOCKS,bread_run16); - -static int -bread_run16(SchedTask *manager, void *in, void *out) -{ - WordCount *w = *(WordCount **)in; - - HTaskPtr t_read = manager->create_task(READ_TASK); - w->t_print->wait_for(t_read); - t_read->set_cpu(read_spe_cpu); - t_read->set_param(0,w->fd); - - if (w->task_num < w->task_blocks) { - t_read->set_param(1,w->task_spawned*w->division_size); // front blocked read position - t_read->set_outData(0, w->file_mmap + w->task_spawned * w->division_size, w->task_blocks * w->division_size); - - // last case - while (w->size >= w->division_size) - run_tasks(manager,w,w->task_num,t_read,w->t_print, w->division_size + w->extra_len); - // remaining data - while (w->size>0) - run_tasks(manager,w,1,t_read,w->t_print, w->size); - - t_read->set_param(2,w->task_spawned*w->division_size); // end blocked read position - t_read->spawn(); - } else { - HTaskPtr t_next = manager->create_task(BREAD_RUN_TASK_BLOCKS, - (memaddr)&w->self,sizeof(memaddr),0,0); - w->t_print->wait_for(t_next); - - t_read->set_param(1,w->task_spawned*w->division_size); - t_read->set_outData(0, w->file_mmap + w->task_spawned * w->division_size, w->task_blocks * w->division_size); - - run_tasks(manager,w, w->task_blocks, t_read, t_next, w->division_size + w->extra_len); - - t_read->set_param(2,w->task_spawned*w->division_size + w->extra_len); - - t_read->spawn(); - t_next->spawn(); - } - return 0; -} - -SchedDefineTask1(MMAP_RUN_TASK_BLOCKS,mmap_run16); - -static int -mmap_run16(SchedTask *manager, void *in, void *out) -{ - WordCount *w = *(WordCount **)in; - - if (w->task_num < w->task_blocks) { - // last case - while (w->size >= w->division_size) - run_tasks(manager,w,w->task_num,0, w->t_print, w->division_size + w->extra_len); - // remaining data - while (w->size>0) - run_tasks(manager,w,1,0,w->t_print, w->size); - } else { - HTaskPtr t_next = manager->create_task(MMAP_RUN_TASK_BLOCKS, - (memaddr)&w->self,sizeof(memaddr),0,0); - w->t_print->wait_for(t_next); - - run_tasks(manager,w, w->task_blocks,0, t_next, w->division_size + w->extra_len); - - t_next->spawn(); - } - return 0; -} - - -static int blocks = 48; -//static int blocks = 31 * 6 * 24; - -//Boyer Moore法に使用するテーブルを作成 -static int* -create_BMskiptable(unsigned char *search_word,int search_word_len,int *skip_table) -{ - for(int i = 0; i < 256; ++i){ - skip_table[i] = search_word_len; - } - - for(int j = 0; j < search_word_len - 1; ++j){ - skip_table[search_word[j]] = search_word_len - j - 1; - } - return skip_table; -} - - -static void -run_start(TaskManager *manager, char *filename,unsigned char *search_word, int search_word_len) -{ - long fd = (long)manager->allocate(sizeof(long)); - struct stat *sb = (struct stat*)manager->allocate(sizeof(struct stat)); - HTaskPtr t_exec; - - if ((fd=open(filename,O_RDONLY,0666))==0) { - fprintf(stderr,"can't open %s\n",filename); - return ; - } - - if (fstat(fd,sb)) { - fprintf(stderr,"can't fstat %s\n",filename); - return ; - } - - WordCountPtr w = (WordCountPtr)manager->allocate(sizeof(WordCount)); - - w->self = w; - w->fd = fd; - w->read_filesize = sb->st_size; - - - if (block_read_flag == 1) { - printf("[block read mode]\n"); - w->file_mmap = (char*)manager->allocate(w->read_filesize); - }else { - printf("[mmap mode]\n"); - my_mmap(filename, w); - } - - - /* prepare BMSearch*/ - int *skip = (int*)manager->allocate(256 * sizeof(int)); // 文字列に対応した table を用意 - w->search_word = search_word; - w->search_word_len = search_word_len; - w->BMskip_table = create_BMskiptable(w->search_word, w->search_word_len, skip); - w->extra_len = w->search_word_len - 1; - - /* original */ - HTaskPtr t_print; - - //w->task_blocks = blocks; - w->self = w; - w->task_spawned = 0; - - w->size = w->file_size = w->read_filesize; - printf("w %lx\n",(long)w); - - /* 1task分のデータサイズ(byte) */ - if (w->size >= 1024*division) { - w->division_size = 1024 * division;/*16kbyte*/ - } else { - w->division_size = w->size; - } - - printf("division_size %ld\n",w->division_size); - - /* "word num" and "line num" */ - w->status_num = 2; - /* taskの数 */ - w->task_num = w->size / w->division_size; - w->task_num = w->task_num + (w->division_size*w->task_num < w->size); - int out_task_num = w->task_num; - - if(!all) { - w->task_blocks = blocks; - } else { - w->task_blocks = w->task_num; - } - - w->out_task_num = out_task_num; - printf("task_num %ld\n",w->task_num); - printf("out_task_num %ld\n",w->out_task_num); - - /* out用のdivision_size. statusが2つなので、あわせて16byteになるように、long long(4byte)を使用 */ - - w->division_out_size = sizeof(unsigned long long)*1; - int out_size = w->division_out_size*out_task_num; - w->o_data = (unsigned long long *)manager->allocate(out_size); - w->out_size_ = out_size; - w->out_size = 1; - printf("out size %d\n",out_size); - - /*各SPEの結果を合計して出力するタスク*/ - - t_print = manager->create_task(PRINT_TASK, - (memaddr)&w->self,sizeof(memaddr),0,0); - w->t_print = t_print; - - for(int i=0;i<1;i++) { - /* Task を task_blocks ずつ起動する Task */ - /* serialize されていると仮定する... */ - if (block_read_flag == 1) { - t_exec = manager->create_task(BREAD_RUN_TASK_BLOCKS, - (memaddr)&w->self,sizeof(memaddr),0,0); - }else { - t_exec = manager->create_task(MMAP_RUN_TASK_BLOCKS, - (memaddr)&w->self,sizeof(memaddr),0,0); - } - - t_print->wait_for(t_exec); - // t_exec->iterate(4); - t_exec->spawn(); - } - w->t_exec = t_exec; - - t_print->spawn(); -} - -static char* -init(int argc, char **argv) -{ - - char *filename = 0; - - for (int i = 1; argv[i]; ++i) { - if (strcmp(argv[i], "-file") == 0) { - filename = argv[i+1]; - } else if (strcmp(argv[i], "-division") == 0) { - division = atoi(argv[i+1]); - } else if (strcmp(argv[i], "-block") == 0) { - blocks = atoi(argv[i+1]); - } else if (strcmp(argv[i], "-a") == 0) { - // create task all at once - all = 1; - } else if (strcmp(argv[i], "-c") == 0) { - use_task_array = 0; - use_compat = 1; - } else if (strcmp(argv[i], "-s") == 0) { - use_task_array = 0; - use_compat = 0; - } else if (strcmp(argv[i], "-t") == 0) { - use_task_creater = 1; - use_task_array = 0; - use_compat = 0; - } else if (strcmp(argv[i], "-anum") == 0) { - array_task_num = atoi(argv[i+1]); - } else if (strcmp(argv[i], "-g") == 0 ) { - spe_cpu = GPU_0; - } else if (strcmp(argv[i], "-cpu") == 0) { - spe_num = atoi(argv[i+1]); - if (spe_num==0) spe_num = 1; - } else if (strcmp(argv[i], "-sw") == 0) { - sword = (unsigned char*)argv[i+1]; - } else if (strcmp(argv[i], "-br") == 0) { - block_read_flag = 1; - } - } - if ((filename==0) || (sword==0)) { - puts(usr_help_str); - exit(1); - } - - return filename; -} - - int TMmain(TaskManager *manager, int argc, char *argv[]) { - char *filename = init(argc, argv); - int sw_len = strlen((const char *)sword); - - unsigned char *search_word = (unsigned char*)manager->allocate(sw_len + 1); - memcpy(search_word, sword, sw_len + 1); - + char *filename = 0; + FileMapReduce *fmp = new FileMapReduce(manager,TASK_EXEC,TASK_EXEC_DATA_PARALLEL,TASK_PRINT); + filename = fmp->init(argc, argv); + BMPtr bm = new BM; + bm->search_word = NULL; + bm->next = NULL; + bm->skip_table = (int*)malloc(sizeof(int)*256); + BM dummy; + dummy.next = NULL; + BMPtr *b = &dummy.next; + for (int i = 1; i < argc; i++) { + if (strcmp(argv[i],"-sw") == 0) { + BMPtr bm = *b = new BM; + bm->search_word = (unsigned char*)argv[i+1]; i++; + bm->search_word_len = strlen((const char*)bm->search_word); + create_BMskiptable(bm); + bm->next = NULL; + b = &bm->next; + } + } + BMPtr bmp = dummy.next; + if (filename < 0 || bmp == NULL) { + return -1; + } + fmp->w->global = (void*)bmp; + fmp->division_out_size = sizeof(unsigned long long); task_init(); st_time = getTime(); - run_start(manager, filename, search_word, sw_len); + fmp->run_start(manager, filename); manager->set_TMend(TMend); return 0; } diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/ppe/Exec.cc --- a/example/bm_search/ppe/Exec.cc Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/ppe/Exec.cc Wed Feb 17 17:37:06 2016 +0900 @@ -3,10 +3,12 @@ #include "Exec.h" #include "Func.h" #include "SchedTask.h" +#include "FileMapReduce.h" +#include "bm.h" #define max(a,b)((a)>(b)?a:b) /* これは必須 */ -SchedDefineTask1(TASK_EXEC,task_exec); +SchedDefineTask1(Exec,task_exec); //ボイヤームーア法による文字列検索アルゴリズム static int BM_method(unsigned char *text,int text_len, @@ -52,22 +54,18 @@ task_exec(SchedTask *s, void *rbuf, void *wbuf) { //get_input - unsigned char *i_data = (unsigned char *)s->get_input(rbuf, 0); - unsigned char *search_word = (unsigned char *)s->get_input(rbuf, 1); - int *skip_table = (int *)s->get_input(rbuf, 2); - + unsigned char *i_data = (unsigned char *)s->get_input(0); int length = (int)s->get_inputSize(0); - int sw_len = (int)s->get_inputSize(1); + MapReduce *w = (MapReduce*)s->get_param(4); + BMPtr bm = (BMPtr)w->global; - unsigned long long *o_data = (unsigned long long*)s->get_output(wbuf,0); + unsigned long long *o_data = (unsigned long long*)s->get_output(0); - o_data[0] = 0; - o_data[0] = BM_method(i_data,length,search_word,sw_len,skip_table); + o_data[0] = BM_method(i_data,length,bm->search_word,bm->search_word_len,bm->skip_table); //printf("match count : %llu\n",o_data[0]); //long task_count = (long)s->get_param(0); //何番目のtaskか //s->printf("[start exec No: %lld]\n",task_count+1); - return 0; } diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/ppe/Exec.h --- a/example/bm_search/ppe/Exec.h Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/ppe/Exec.h Wed Feb 17 17:37:06 2016 +0900 @@ -1,9 +1,7 @@ #ifndef INCLUDED_TASK_HELLO #define INCLUDED_TASK_HELLO -#ifndef INCLUDED_SCHED_TASK # include "SchedTask.h" -#endif #endif diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/ppe/Print.cc --- a/example/bm_search/ppe/Print.cc Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/ppe/Print.cc Wed Feb 17 17:37:06 2016 +0900 @@ -2,25 +2,23 @@ #include #include "Print.h" #include "Func.h" -#include "WordCount.h" +#include "FileMapReduce.h" /* これは必須 */ -SchedDefineTask1(PRINT_TASK,print_task); +SchedDefineTask1(Print,print_task); static int print_task(SchedTask *s, void *rbuf, void *wbuf) { - WordCount *w = *(WordCount**)rbuf; + MapReduce *w = (MapReduce*)s->get_input(0); unsigned long long *idata = w->o_data; - unsigned int idata_task_num = w->out_size * w->out_task_num; + unsigned int idata_task_num = w->task_num; int match_counter = 0; for (int i = 0;i < idata_task_num;i++) { - match_counter += idata[i]; //s->printf("idata[%d]=:%d\n",i,idata[i]); } - s->printf("HIT:%d\n",match_counter); return 0; } diff -r 58a32f966763 -r f786ffb2a198 example/bm_search/task_init.cc --- a/example/bm_search/task_init.cc Sat Feb 06 20:05:05 2016 +0900 +++ b/example/bm_search/task_init.cc Wed Feb 17 17:37:06 2016 +0900 @@ -3,30 +3,33 @@ #ifdef __CERIUM_GPU__ #include "GpuScheduler.h" #endif +#ifdef __CERIUM_CUDA__ +#include "CudaScheduler.h" +#endif /* 必ずこの位置に書いて */ -SchedExternTask(TASK_EXEC); -SchedExternTask(READ_TASK); -SchedExternTask(BREAD_RUN_TASK_BLOCKS); -SchedExternTask(MMAP_RUN_TASK_BLOCKS); -SchedExternTask(PRINT_TASK); +SchedExternTask(Exec); +SchedExternTask(Exec_Data_Parallel); +SchedExternTask(Print); /** * この関数は ../spe/spe-main と違って * 自分で呼び出せばいい関数なので * 好きな関数名でおk (SchedRegisterTask は必須) */ - void task_init(void) { #ifdef __CERIUM_GPU__ - GpuSchedRegister(TASK_EXEC, "gpu/Exec.cl", "run"); + GpuSchedRegister(TASK_EXEC, "gpu/Exec.cl", "bm_search"); + GpuSchedRegister(TASK_EXEC_DATA_PARALLEL, "gpu/Exec_Data_Parallel.cl","bm_search_parallel"); +#endif +#ifdef __CERIUM_CUDA__ + CudaSchedRegister(TASK_EXEC, "cuda/Exec.ptx", "bm_search"); + CudaSchedRegister(TASK_EXEC_DATA_PARALLEL, "cuda/Exec_Data_Parallel.ptx","bm_search_parallel"); #endif - SchedRegister(TASK_EXEC); - SchedRegister(READ_TASK); - SchedRegister(MMAP_RUN_TASK_BLOCKS); - SchedRegister(BREAD_RUN_TASK_BLOCKS); - SchedRegister(PRINT_TASK); + SchedRegisterTask(TASK_EXEC, Exec); + SchedRegisterTask(TASK_EXEC_DATA_PARALLEL, Exec); + SchedRegisterTask(TASK_PRINT, Print); }