github.com/msinev/pmc@v0.0.0-20220512222710-eb40bf080315/README.md (about) 1 # pmc 2 Poor man's cluster 3 4 There are BIG clusters for the BIG data. Ceph, Mesos, Spark, Hadoop, Flink.. you name it. Complex tools and advanced setup, high salary and great resposibility, advanced perfomance and unbeatable performance. Blah... Blah... Blah... 5 6 But what if you are a simple man with a big file collection... and you have several computers... 7 8 # Case one - very stupid 9 10 Assume that you have a simple shared folder with your favorite movies and you need to recode all movies to new h.2xx latest and greatest slowest ever codec. 11 You have few coputers at home... or just near by. You want to run codec on all of them to accelerate process, but how to split files... 12 Here is an idea. 13 14 Run a file processing script on each machine you have mounted your file storage... 15 ``` 16 #!/bin/bash 17 until [[ $fileprocess == "." ]] 18 do 19 date 20 if [[ $fileprocess ]];then 21 echo "Processing $fileprocess" 22 ff-decoder-recoder /path/to where/files/mounted/$fileprocess 23 24 else 25 echo "Wait to start" 26 sleep 1 27 fi 28 fileprocess=$(curl -s "http://192.168.1.199:3000/get") 29 done 30 date 31 echo "Complete!" 32 ``` 33 34 With simple server below you should get files from URL (aka REST service endpoint) one by one 35 36 ``` 37 // It runs in background pushing file list into channel 38 func pollFiles(dirname string, WorkSource chan<- string) { 39 defer close(WorkSource) 40 41 files, err := ioutil.ReadDir(dirname) 42 if err != nil { 43 log.Fatal(err) 44 } 45 46 for _, file := range files { 47 if !file.IsDir() { 48 WorkSource<-file.Name() 49 log.Println("Queueing... "+file.Name()) 50 } 51 } 52 log.Println("Complete list of "+dirname+"... ") 53 } 54 55 // It serves files from channel one by one on request 56 func getHandler(w http.ResponseWriter, r *http.Request) { 57 select { 58 case inData,ok:=<-WorkSource: 59 if(ok) { 60 w.Write([]byte(inData)) 61 }else { 62 w.Write([]byte(".")) 63 } 64 case <-time.After(1 * time.Second): 65 w.Write([]byte("")) 66 } 67 } 68 69 70 ``` 71 Here it is pkg/servepath Take it and use wisely 72 73 # Case two - better than nothing 74 75 With simple case one all works fine while everything works fine. Crashes or outages not handled. Shared folder also required. 76 Let's try to fix it. 77 Extra features would be required to mention if data processing task being started finished aborted or stalled. 78 Ability to show where task is stalled. 79 Extra bonus - ability to pull and push data over the http.