launchpad.net/~rogpeppe/juju-core/500-errgo-fix@v0.0.0-20140213181702-000000002356/doc/simplestreams-metadata.txt (about) 1 Introduction 2 ============ 3 4 When Juju bootstraps, it needs two critical pieces of information: 5 1. The uuid of the image to use when starting new compute instances 6 2. The URL from which to download the correct version of a tools tarball 7 8 The necessary information is stored in a json metadata format called simplestreams. 9 The simplestreams format is used to describe related items in a structural fashion. 10 See the Launchpad project lp:simplestreams for more details. 11 12 For supported public clouds like Amazon, HP Cloud etc, no action is required by the 13 end user so the following information is more for those interested in what happens 14 under the covers. Those setting up a private cloud, or who want to change how things 15 work (eg use a different Ubuntu image), need to pay closer attention. 16 17 Basic Workflow 18 ============== 19 20 Whether images or tools, Juju uses a search path to try and find suitable metadata. 21 The path components (in order of lookup) are: 22 23 1. User supplied location (specified by tools-metadata-url or image-metadata-url config settings) 24 2. The environment's cloud storage 25 3. Provider specific locations (eg keystone endpoint if on Openstack) 26 4. A web location with metadata for supported public clouds (https://streams.canonical.com) 27 28 Metadata may be inline signed, or unsigned. We indicate a metadata file is signed by using 29 a '.sjson' extension. Each location in the path is first searched for signed metadata, and 30 if none is found, unsigned metadata is attempted before moving onto the next path location. 31 32 Juju ships with public keys used to validate the integrity of image and tools metadata obtained 33 from https://streams.canonical.com. So out of the box, Juju will "Just Work" with any supported 34 public cloud, using signed metadata. Setting up metadata for a private (eg Openstack) cloud requires 35 metadata to be generated using tools which ship with Juju (more below). 36 37 Image Metadata Contents 38 ======================= 39 40 Image metadata uses a simplestreams content type of "image-ids". 41 The product id is formed as follows: 42 "com.ubuntu.cloud:server:<series_version>:<arch>" 43 eg 44 "com.ubuntu.cloud:server:13.10:amd64" 45 46 Non-released images (eg beta, daily etc) have product ids like: 47 "com.ubuntu.cloud.daily:server:13.10:amd64" 48 49 The metadata index and product files are required to be in the following directory tree 50 (relative to the URL associated with each path component): 51 52 <path_url> 53 |-streams 54 |-v1 55 |-index.(s)json 56 |-product-foo.(s)json 57 |-product-bar.(s)json 58 59 The index file must be called "index.(s)json" (sjson for signed). The various product files are 60 named according to the Path values contained in the index file. 61 62 Tools Metadata Contents 63 ======================= 64 65 Tools metadata uses a simplestreams content type of "content-download". 66 The product id is formed as follows: 67 "com.ubuntu.juju:<series_version>:<arch>" 68 eg 69 "com.ubuntu.juju:12.04:amd64" 70 71 The metadata index and product files are required to be in the following directory tree 72 (relative to the URL associated with each path component). In addition, tools tarballs 73 which Juju needs to download are also expected. 74 75 <path_url> 76 |-streams 77 | |-v1 78 | |-index.(s)json 79 | |-product-foo.(s)json 80 | |-product-bar.(s)json 81 | 82 |-releases 83 |-tools-abc.tar.gz 84 |-tools-def.tar.gz 85 |-tools-xyz.tar.gz 86 87 The index file must be called "index.(s)json" (sjson for signed). The product file and 88 tools tarball name(s) match whatever is in the index/product files. 89 90 Configuration 91 ============= 92 93 For supported public clouds, no extra configuration is required; things work out-of-the-box. 94 However, for testing purposes, or for non-supported cloud deployments, Juju needs to know 95 where to find the tools and which image to run. Even for supported public clouds where all 96 required metadata is available, the user can put their own metadata in the search path to 97 override what is provided by the cloud. 98 99 1. User specified URLs 100 101 These are initially specified in the environments.yaml file (and then subsequently copied to the 102 jenv file when the environment is bootstrapped). For images, use "image-metadata-url"; for tools, 103 use "tools-metadata-url". The URLs can point to a world readable container/bucket in the cloud, 104 an address served by a http server, or even a shared directory accessible by all node instances 105 running in the cloud. 106 107 eg assume an Apache http server with base URL https://juju-metadata, providing access to 108 information at <base>/images and <base>/tools. The Juju environment yaml file could have 109 the following entries (one or both): 110 111 tools-metadata-url: https://juju-metadata/tools 112 image-metadata-url: https://juju-metadata/images 113 114 The required files in each location is as per the directory layout described earlier. 115 For a shared directory, use a URL of the form "file:///sharedpath". 116 117 2. Cloud storage 118 119 If no matching metadata is found in the user specified URL, environment's cloud storage is searched. 120 No user configuration is required here - all Juju environments are set up with cloud storage which is 121 used to store state information, charms etc. Cloud storage setup is provider dependent; for Amazon 122 and Openstack clouds, the storage is defined by the "control-bucket" value, for Azure, the 123 "storage-account-name" value is relevant. 124 125 The (optional) directory structure inside the cloud storage is as follows: 126 127 <cloud storage top level> 128 |-tools 129 | |-streams 130 | |-v1 131 | |-releases 132 | 133 |-images 134 |-streams 135 |-v1 136 137 Of course, if only custom image metadata is required, the tools directory will not be required, 138 and vice versa. 139 140 As a sidebar, note that if juju bootstrap is run with the --upload-tools option, the tools and metadata 141 are placed according to the above structure. That's why the tools are then available for Juju to use. 142 143 3. Provider specific storage 144 145 Providers may allow additional locations to search for metadata and tools. For Openstack, keystone 146 endpoints may be created by the cloud administrator. These are defined as follows: 147 148 juju-tools : the <path_url> value as described above in Tools Metadata Contents 149 product-streams : the <path_url> value as described above in Image Metadata Contents 150 151 4. Central web location (https://streams.canonical.com) 152 153 This is the default location used to search for image and tools metadata and is used if no matches 154 are found earlier in any of the above locations. No user configuration is required. 155 156 Deploying Private Clouds 157 ======================== 158 159 There are two main issues when deploying a private cloud: 160 1. Images ids will be specific to the cloud 161 2. Often, outside internet access is blocked 162 163 Issue 1 means that image id metadata needs to be generated and made available. 164 Issue 2 means that tools need to be mirrored locally to make them accessible. 165 166 Juju tools exist to help with generating and validating image and tools metadata. 167 For tools, it is often easiest to just mirror https://streams.canonical.com/tools. 168 However image metadata cannot be simply mirrored because the image ids are taken 169 from the cloud storage provider, so this need to be generated and validated using 170 the commands described below. 171 172 The available Juju metadata tools can be seen by using the help command: 173 juju help metadata 174 175 The overall workflow is: 176 - generate image metadata 177 - copy image metadata to somewhere in the metadata search path 178 - optionally, mirror tools to somewhere in the metadata search path 179 - optionally, configure tools-metadata-url and/or image-metadata-url 180 181 1. Image metadata 182 183 Generate image metadata using 184 juju metadata generate-image -d <metadata_dir> 185 186 As a minimum, the above command needs to know the image id to use and a directory in which to write the files. 187 Other required parameters like region, series, architecture etc are taken from the current Juju environment (or 188 an environment specified with the -e option). These parameters can also be overridden on the command line. 189 190 The image metadata command can be run multiple times with different regions, series, architecture, and 191 it will keep adding to the metadata files. Once all required image ids have been added, the index and product 192 json files can be uploaded to a location in the Juju metadata search path. As per the Configuration section, 193 this may be somewhere specified by the image-metadata-url setting or the cloud's storage etc. 194 195 Examples: 196 1. image-metadata-url 197 - upload contents of <metadata_dir> to http://somelocation 198 - set image-metadata-url to http://somelocation/images 199 2. Cloud storage 200 - upload contents of <metadata_dir> directly to environment's cloud storage 201 202 To ensure that the image metadata has been generated and uploaded correctly, use the validation command to 203 ensure an image id can be discovered for a given scenario (region series, arch): 204 juju metadata validate-images 205 206 If run without parameters, the validation command will take all required details from the current Juju 207 environment (or as specified by -e) and print the image id it would use to spin up an instance. 208 Alternatively, series, region, architecture etc can be specified on the command line to override values 209 in the environment config. 210 211 2. Tools metadata 212 213 Generally, tools and related metadata is mirrored from https://streams.canonical.com/tools. However, 214 it is possible to manually generate metadata for a custom built tools tarball using: 215 juju generate-tools -d <tools_dir> 216 217 where the required tools tarballs are first placed in a directory <tools_dir>/tools/releases. 218 Then, the contents of <tools_dir> can be uploaded to a location in the Juju metadata search path. 219 As per the Configuration section, this may be somewhere specified by the tools-metadata-url setting or 220 the cloud's storage etc. 221 222 Examples: 223 1. tools-metadata-url 224 - upload contents of <tools_dir> to http://somelocation 225 - set tools-metadata-url to http://somelocation/tools 226 2. Cloud storage 227 - upload contents of <tools_dir> directly to environment's cloud storage 228 229 As with image metadata, the validation command is used to ensure tools are available for Juju to use: 230 juju metadata validate-tools 231 232 The same comments apply. Run the validation tool without parameters to use details from the Juju 233 environment, or override values as required on the command line. See juju help metadata validate-tools 234 for more details.