github.com/voedger/voedger@v0.0.0-20240520144910-273e84102129/design/orchestration/google/README.md (about)

     1  ## Abstract
     2  
     3  This paper overviews the following documents:
     4  
     5  - [Borg, Omega, and Kubernetes][BOK], Brendan Burns Brian Grant David Oppenheimer Eric Brewer John Wilkes, ACM Queue, vol. 14 (2016), pp. 70-93
     6  - [Omega: flexible, scalable schedulers for large compute clusters][Omega], Malte Schwarzkopf Andy Konwinski Michael Abd-El-Malek John Wilkes, SIGOPS European Conference on Computer Systems (EuroSys), ACM, Prague, Czech Republic (2013), pp. 351-364
     7  - [Large-scale cluster management at Google with Borg][Borg], Abhishek Verma Luis Pedrosa Madhukar R. Korupolu David Oppenheimer Eric Tune John Wilkes
     8  Proceedings of the European Conference on Computer Systems (EuroSys), ACM, Bordeaux, France (2015)
     9  
    10  Takeaways:
    11   
    12  1. The scheduler must use a *shared state* model with *optimistic* concurrency control without blocking resources while scheduling jobs.
    13  2. The scheduler must support different scheduling algorithms for *different types of jobs* (service and batch), each of which fulfills a minimum set of common requirements (for example, has a common scale of priority of jobs or common metrics of maximum resources).
    14  3. The default scheduler should schedule jobs *incrementally*, but for individual jobs, allow all-or-nothing scheduling.
    15  4. Сoncurrency conflicts resolution should not use "outdated scheduling shapshot" as a reason, analyse results instead (reason: decrease conflicts frequence)
    16  5. The *scaling* of the scheduler under the increasing load on the job queue should occur, if possible, automatically or by simply changing the configuration of the scheduler, without rewriting its source codes.
    17  
    18  
    19  ## The History of Kubernetes on a Timeline
    20  
    21  https://blog.risingstack.com/the-history-of-kubernetes
    22  - 2003-2004: Birth of the Borg System
    23  - 2013: From Borg to Omega
    24  - 2014: Google Introduces Kubernetes
    25  - 2016: The Year Kubernetes Goes Mainstream!
    26  - 2017: The Year of Enterprise Adoption & Support
    27  
    28  
    29  ## Borg, Omega, and Kubernetes
    30  
    31  [Borg, Omega, and Kubernetes][BOK]
    32  
    33  - Though widespread interest in software containers is a relatively recent phenomenon, at Google we have been managing Linux containers at scale for more than ten years and built three different containermanagement systems in that time.
    34  - Each system was heavily influenced by its predecessors, even though they were developed for different reasons
    35  
    36  ### Borg
    37  
    38  - The first unified container-management system developed at Google was the system we internally call Borg
    39  - Borg remains the primary container-management system within Google because of its scale, breadth of features, and extreme robustness
    40  
    41  ### Omega
    42  
    43  - Omega an offspring of Borg, was driven by a desire to improve the software engineering of the Borg ecosystem.
    44  - It applied many of the patterns that had proved successful in Borg, but was built from the ground up to have a more consistent, principled architecture
    45  - Omega stored the state of the cluster in a centralized Paxos-based transaction oriented store that was accessed by the different parts of the cluster control plane (such as schedulers), using optimistic concurrency control to handle the occasional conflicts.
    46  - Many of Omega’s innovations (including multiple schedulers) have since been folded into Borg
    47  
    48  ### Kubernetes
    49  
    50  - The third container-management system developed at Google was Kubernetes
    51  -  It was conceived of and developed in a world where external developers were becoming
    52  interested in Linux containers, and Google had developed a growing business selling public-cloud infrastructure.
    53  - Kubernetes is open source—a contrast to Borg and Omega, which were developed as purely Google-internal systems.
    54  - Like Omega, Kubernetes has at its core a shared persistent store, with components watching for changes to relevant objects
    55  - In contrast to Omega, which exposes the store directly to trusted control-plane components, state in Kubernetes is accessed exclusively through a domains pecific REST API that applies higher-level versioning, validation, semantics, and policy, in support of a more diverse array of clients
    56  
    57  
    58  ## Omega: flexible, scalable schedulers for large compute clusters
    59  
    60  - [Omega: flexible, scalable schedulers for large compute clusters][Omega]
    61  - Russian translation: [Omega-ru](Omega-ru.md)
    62  
    63  
    64  
    65  ## Large-scale cluster management at Google with Borg
    66  
    67  - [Large-scale cluster management at Google with Borg][Borg]
    68  - To be read
    69  
    70  
    71  [Omega]: https://research.google/pubs/pub41684/
    72  [Borg]: https://research.google/pubs/pub43438/
    73  [BOK]: https://research.google/pubs/pub44843/
    74  
    75