github.com/demonoid81/containerd@v1.3.4/reports/2017-03-17.md (about) 1 # Development Report for Mar 17, 2017 2 3 ## Testing Plan 4 5 Thanks to @gianarb for starting the discussion around a test and CI plan for containerd. We want to make sure that users of containerd can feel secure depending on containerd; having a solid test plan is a must. 6 7 Testing a project like containerd is always a challenge because of the systems that it needs to support. ARM, Windows, Linux, and Power (as well as many more variations) are all examples of platforms that we support today and will need a CI. 8 9 You can view the issue and contribute to the testing plan [here](https://github.com/containerd/containerd/issues/634). 10 11 ## Windows Runtime 12 13 Work has started on porting over the Windows execution code. There is still a lot of testing to do after the port but a PR should be coming soon. 14 15 ## Metrics 16 17 We started the work to get container level metrics exported over prometheus. You can see the initial output here: 18 19 ``` 20 containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Async"} 958464 21 containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Read"} 958464 22 containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Sync"} 0 23 containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Total"} 958464 24 containerd_container_blkio_io_service_bytes_recursive_bytes{id="test",major="8",minor="0",op="Write"} 0 25 containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Async"} 17 26 containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Read"} 17 27 containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Sync"} 0 28 containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Total"} 17 29 containerd_container_blkio_io_serviced_recursive_total{id="test",major="8",minor="0",op="Write"} 0 30 containerd_container_cpu_kernel_nanoseconds{id="test"} 1e+07 31 containerd_container_cpu_throttle_periods_total{id="test"} 0 32 containerd_container_cpu_throttled_periods_total{id="test"} 0 33 containerd_container_cpu_throttled_time_nanoseconds{id="test"} 0 34 containerd_container_cpu_total_nanoseconds{id="test"} 2.1428791e+07 35 containerd_container_cpu_user_nanoseconds{id="test"} 0 36 containerd_container_hugetlb_failcnt_total{id="test",page="1GB"} 0 37 containerd_container_hugetlb_failcnt_total{id="test",page="2MB"} 0 38 containerd_container_hugetlb_max_bytes{id="test",page="1GB"} 0 39 containerd_container_hugetlb_max_bytes{id="test",page="2MB"} 0 40 containerd_container_hugetlb_usage_bytes{id="test",page="1GB"} 0 41 containerd_container_hugetlb_usage_bytes{id="test",page="2MB"} 0 42 containerd_container_memory_active_anon_bytes{id="test"} 0 43 containerd_container_memory_active_file_bytes{id="test"} 659456 44 containerd_container_memory_cache_bytes{id="test"} 925696 45 containerd_container_memory_dirty_bytes{id="test"} 0 46 containerd_container_memory_hierarchical_memory_limit_bytes{id="test"} 9.223372036854772e+18 47 containerd_container_memory_hierarchical_memsw_limit_bytes{id="test"} 9.223372036854772e+18 48 containerd_container_memory_inactive_anon_bytes{id="test"} 73728 49 containerd_container_memory_inactive_file_bytes{id="test"} 266240 50 containerd_container_memory_kernel_failcnt_total{id="test"} 0 51 containerd_container_memory_kernel_limit_bytes{id="test"} 9.223372036854772e+18 52 containerd_container_memory_kernel_max_bytes{id="test"} 0 53 containerd_container_memory_kernel_usage_bytes{id="test"} 0 54 containerd_container_memory_kerneltcp_failcnt_total{id="test"} 0 55 containerd_container_memory_kerneltcp_limit_bytes{id="test"} 9.223372036854772e+18 56 containerd_container_memory_kerneltcp_max_bytes{id="test"} 0 57 containerd_container_memory_kerneltcp_usage_bytes{id="test"} 0 58 containerd_container_memory_mapped_file_bytes{id="test"} 577536 59 containerd_container_memory_oom_total{id="test"} 0I 60 containerd_container_memory_pgfault_bytes{id="test"} 770 61 containerd_container_memory_pgmajfault_bytes{id="test"} 6 62 containerd_container_memory_pgpgin_bytes{id="test"} 651 63 containerd_container_memory_pgpgout_bytes{id="test"} 407 64 containerd_container_memory_rss_bytes{id="test"} 73728 65 containerd_container_memory_rss_huge_bytes{id="test"} 0 66 containerd_container_memory_swap_failcnt_total{id="test"} 0 67 containerd_container_memory_swap_limit_bytes{id="test"} 9.223372036854772e+18 68 containerd_container_memory_swap_max_bytes{id="test"} 1.527808e+06 69 containerd_container_memory_swap_usage_bytes{id="test"} 999424 70 containerd_container_memory_total_active_anon_bytes{id="test"} 0 71 containerd_container_memory_total_active_file_bytes{id="test"} 659456 72 containerd_container_memory_total_cache_bytes{id="test"} 925696 73 containerd_container_memory_total_dirty_bytes{id="test"} 0 74 containerd_container_memory_total_inactive_anon_bytes{id="test"} 73728 75 containerd_container_memory_total_inactive_file_bytes{id="test"} 266240 76 containerd_container_memory_total_mapped_file_bytes{id="test"} 577536 77 containerd_container_memory_total_pgfault_bytes{id="test"} 770 78 containerd_container_memory_total_pgmajfault_bytes{id="test"} 6 79 containerd_container_memory_total_pgpgin_bytes{id="test"} 651 80 containerd_container_memory_total_pgpgout_bytes{id="test"} 407 81 containerd_container_memory_total_rss_bytes{id="test"} 73728 82 containerd_container_memory_total_rss_huge_bytes{id="test"} 0 83 containerd_container_memory_total_unevictable_bytes{id="test"} 0 84 containerd_container_memory_total_writeback_bytes{id="test"} 0 85 containerd_container_memory_unevictable_bytes{id="test"} 0 86 containerd_container_memory_usage_failcnt_total{id="test"} 0 87 containerd_container_memory_usage_limit_bytes{id="test"} 9.223372036854772e+18 88 containerd_container_memory_usage_max_bytes{id="test"} 1.527808e+06 89 containerd_container_memory_usage_usage_bytes{id="test"} 999424 90 containerd_container_memory_writeback_bytes{id="test"} 0 91 containerd_container_per_cpu_nanoseconds{cpu="0",id="test"} 7.530139e+06 92 containerd_container_per_cpu_nanoseconds{cpu="1",id="test"} 4.586408e+06 93 containerd_container_per_cpu_nanoseconds{cpu="2",id="test"} 5.076059e+06 94 containerd_container_per_cpu_nanoseconds{cpu="3",id="test"} 4.236185e+06 95 containerd_container_pids_current{id="test"} 1 96 containerd_container_pids_limit{id="test"} 0 97 ``` 98 99 The `id` label will be the id of the container so users can filter on the metrics for only the containers that they care about. 100 101 The frequency of metric collection is configurable via the prometheus scrape time. Every time the `/metrics` API is hit, that is when container metrics are collected. There is no internal timer in containerd, you only pay the cost of collecting metrics when you are asking for them. If you never ask for metrics the collection never happens. 102 103 There should be a PR up soon so that we can discuss the metrics and label names. 104 105 ## Image Pull 106 107 * https://github.com/containerd/containerd/pull/640 108 109 We now have a proof of concept of end to end pull. Up to this point, the 110 relationship between subsystems has been somewhat theoretical. We now leverage 111 fetching, the snapshot drivers, the rootfs service, image metadata and the 112 execution service, validating the proposed model for containerd. There are a 113 few caveats, including the need to move some of the access into GRPC services, 114 but the basic components are there. 115 116 The first command we will cover here is `dist pull`. This is the analog of 117 `docker pull` and `git pull`. It performs a full resource fetch for an image 118 and unpacks the root filesystem into the snapshot drivers. An example follows: 119 120 ``` console 121 $ sudo ./bin/dist pull docker.io/library/redis:latest 122 docker.io/library/redis:latest: resolved |++++++++++++++++++++++++++++++++++++++| 123 manifest-sha256:4c8fb09e8d634ab823b1c125e64f0e1ceaf216025aa38283ea1b42997f1e8059: done |++++++++++++++++++++++++++++++++++++++| 124 layer-sha256:3b281f2bcae3b25c701d53a219924fffe79bdb74385340b73a539ed4020999c4: done |++++++++++++++++++++++++++++++++++++++| 125 config-sha256:e4a35914679d05d25e2fccfd310fde1aa59ffbbf1b0b9d36f7b03db5ca0311b0: done |++++++++++++++++++++++++++++++++++++++| 126 layer-sha256:4b7726832aec75f0a742266c7190c4d2217492722dfd603406208eaa902648d8: done |++++++++++++++++++++++++++++++++++++++| 127 layer-sha256:338a7133395941c85087522582af182d2f6477dbf54ba769cb24ec4fd91d728f: done |++++++++++++++++++++++++++++++++++++++| 128 layer-sha256:83f12ff60ff1132d1e59845e26c41968406b4176c1a85a50506c954696b21570: done |++++++++++++++++++++++++++++++++++++++| 129 layer-sha256:693502eb7dfbc6b94964ae66ebc72d3e32facd981c72995b09794f1e87bac184: done |++++++++++++++++++++++++++++++++++++++| 130 layer-sha256:622732cddc347afc9360b4b04b46c6f758191a1dc73d007f95548658847ee67e: done |++++++++++++++++++++++++++++++++++++++| 131 layer-sha256:19a7e34366a6f558336c364693df538c38307484b729a36fede76432789f084f: done |++++++++++++++++++++++++++++++++++++++| 132 elapsed: 1.6 s total: 0.0 B (0.0 B/s) 133 INFO[0001] unpacking rootfs 134 ``` 135 136 Note that we haven't integrated rootfs unpacking into the status output, but we 137 pretty much have what is in docker today (:P). We can see the result of our 138 pull with the following: 139 140 ```console 141 $ sudo ./bin/dist images 142 REF TYPE DIGEST SIZE 143 docker.io/library/redis:latest application/vnd.docker.distribution.manifest.v2+json sha256:4c8fb09e8d634ab823b1c125e64f0e1ceaf216025aa38283ea1b42997f1e8059 1.8 kB 144 ``` 145 146 The above shows that we have an image called "docker.io/library/redis:latest" 147 mapped to the given digest marked with a specific format. We get the size of 148 the manifest right now, not the full image, but we can add more as we need it. 149 For the most part, this is all that is needed, but a few tweaks to the model 150 for naming may need to be added. Specifically, we may want to index under a few 151 different names, including those qualified by hash or matched by tag versions. 152 We can do more work in this area as we develop the metadata store. 153 154 The name shown above can then be used to run the actual container image. We can 155 do this with the following command: 156 157 ```console 158 $ sudo ./bin/ctr run --id foo docker.io/library/redis:latest /usr/local/bin/redis-server 159 1:C 17 Mar 17:20:25.316 # Warning: no config file specified, using the default config. In order to specify a config file use /usr/local/bin/redis-server /path/to/redis.conf 160 1:M 17 Mar 17:20:25.317 * Increased maximum number of open files to 10032 (it was originally set to 1024). 161 _._ 162 _.-``__ ''-._ 163 _.-`` `. `_. ''-._ Redis 3.2.8 (00000000/0) 64 bit 164 .-`` .-```. ```\/ _.,_ ''-._ 165 ( ' , .-` | `, ) Running in standalone mode 166 |`-._`-...-` __...-.``-._|'` _.-'| Port: 6379 167 | `-._ `._ / _.-' | PID: 1 168 `-._ `-._ `-./ _.-' _.-' 169 |`-._`-._ `-.__.-' _.-'_.-'| 170 | `-._`-._ _.-'_.-' | http://redis.io 171 `-._ `-._`-.__.-'_.-' _.-' 172 |`-._`-._ `-.__.-' _.-'_.-'| 173 | `-._`-._ _.-'_.-' | 174 `-._ `-._`-.__.-'_.-' _.-' 175 `-._ `-.__.-' _.-' 176 `-._ _.-' 177 `-.__.-' 178 179 1:M 17 Mar 17:20:25.326 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128. 180 1:M 17 Mar 17:20:25.326 # Server started, Redis version 3.2.8 181 1:M 17 Mar 17:20:25.326 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 182 1:M 17 Mar 17:20:25.326 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 183 1:M 17 Mar 17:20:25.326 * The server is now ready to accept connections on port 6379 184 ``` 185 186 Wow! So, now we are running `redis`! 187 188 There are still a few things to work out. Notice that we have to specify the 189 command as part of the arguments to `ctr run`. This is because are not yet 190 reading the image config and converting it to an OCI runtime config. With the 191 base laid in this PR, adding such functionality should be straightforward. 192 193 While this is a _little_ messy, this is great progress. It should be easy 194 sailing from here. 195 196 197