github.com/msinev/pmc@v0.0.0-20220512222710-eb40bf080315/README.md (about)

     1  # pmc
     2  Poor man's cluster
     3  
     4  There are BIG clusters for the BIG data. Ceph, Mesos, Spark, Hadoop, Flink.. you name it. Complex tools and advanced setup, high salary and great resposibility, advanced perfomance and unbeatable performance. Blah... Blah... Blah... 
     5  
     6  But what if you are a simple man with a big file collection... and you have several computers... 
     7  
     8  # Case one - very stupid
     9  
    10  Assume that you have a simple shared folder with your favorite movies and you need to recode all movies to new h.2xx latest and greatest slowest ever codec.
    11  You have few coputers at home... or just near by. You want to run codec on all of them to accelerate process, but how to split files...  
    12  Here is an idea.  
    13  
    14  Run a file processing script on each machine you have mounted your file storage... 
    15  ```
    16  #!/bin/bash
    17  until [[ $fileprocess == "." ]]
    18    do
    19      date
    20      if [[ $fileprocess ]];then
    21        echo "Processing $fileprocess"
    22        ff-decoder-recoder /path/to where/files/mounted/$fileprocess
    23        
    24      else
    25        echo "Wait to start"
    26        sleep 1
    27      fi
    28    fileprocess=$(curl -s "http://192.168.1.199:3000/get")
    29    done
    30  date
    31  echo "Complete!"
    32  ```
    33  
    34  With simple server below you should get files from  URL (aka REST service endpoint) one by one
    35  
    36  ```
    37  // It runs in background pushing file list into channel
    38  func pollFiles(dirname string, WorkSource chan<- string) {
    39  	defer close(WorkSource)
    40  
    41  	files, err := ioutil.ReadDir(dirname)
    42  	if err != nil {
    43  		log.Fatal(err)
    44  	}
    45  
    46  	for _, file := range files {
    47  		if !file.IsDir() {
    48  			WorkSource<-file.Name()
    49  			log.Println("Queueing... "+file.Name())
    50  		}
    51  	}
    52  	log.Println("Complete list of "+dirname+"... ")
    53  }
    54  
    55  // It serves files from channel one by one on request
    56  func getHandler(w http.ResponseWriter, r *http.Request) {
    57  	select {
    58  	case inData,ok:=<-WorkSource:
    59  		if(ok) {
    60  			w.Write([]byte(inData))
    61  		}else {
    62  			w.Write([]byte("."))
    63  		}
    64  	case <-time.After(1 * time.Second):
    65  		w.Write([]byte(""))
    66  	}
    67  }
    68  
    69  
    70  ```
    71  Here it is pkg/servepath Take it and use wisely
    72  
    73  # Case two - better than nothing
    74  
    75  With simple case one all works fine while everything works fine. Crashes or outages not handled. Shared folder also required. 
    76  Let's try to fix it. 
    77  Extra features would be required to mention if data processing task being started finished aborted or stalled. 
    78  Ability to show where task is stalled. 
    79  Extra bonus - ability to pull and push data over the http.