Branches

b502c67 Roll recipe dependencies (trivial). by recipe-roller · 9 hours ago main
86a910b Roll recipe dependencies (trivial). by recipe-roller · 13 hours ago
1f97956 Roll recipe dependencies (trivial). by recipe-roller · 14 hours ago
9d0ec81 Roll recipe dependencies (trivial). by recipe-roller · 15 hours ago
fe27a6d check_chrome_preuprev: request less fields by Qijiang Fan · 15 hours ago

ChromeOS Recipes

This is a quick-start guide for ChromeOS Recipes. For more in-depth information about the Recipes framework, see the Recipes User Guide in the Chromium repo. This document only covers the ChromeOS recipes built on top of the Chrome-provided framework.

The recipes and recipe modules in this repo have documentation auto-generated by the ‘test train’ process: README.recipes.md

Getting Started

The recipes.py script in this repo will bootstrap itself. You can verify that it is working by running tests:

./recipes.py test run

After making a change, you will usually need to “re-train” expected output in order for tests to pass:

./recipes.py test train

This will also regenerate the README.recipes.md documentation. That file and the expectation data in *.expected directories should be checked in with your changes.

Navigating the code

The top-level code for the main set of builders can be found in the infra/recipes/recipes directory, in the build_*.py files. For example, the top-level code for the postsubmit builders is in the RunSteps function in build_postsubmit.py. The tests for each builder are in the same file, in the GenTests function.

The builders in infra/recipes/recipes will generally call into support functions in infra/recipes/recipe_modules. In particular, calls to the RPCs defined in chromite/api generally happen from recipe_modules.

Support Tools

This repo contains some non-recipes support tooling in the support/ directory. These tools are delivered to CI hosts via CIPD. This process is automated by the support/deploy_cipd.sh script. See support/README.txt for more details.

Testing Locally

Recipes can be run locally:

./recipes.py run <recipe>

Testing With `led`

Expectation tests only mock the external services a recipe depends on. This means the expectation tests cannot catch bugs caused by calling external services incorrectly. For example, if a recipe made the call

ls --bad_flag

it would not be caught by ./recipes.py test train. A more realistic example is the recipe makes incorrect assumptions about ACLs, paths, responses from the service, etc.

One way to get more confidence the recipe code will work is by running it with the actual builder definitions and swarming environment, using the led command-line tool. This will call the external services with the same user, timeouts, swarming caches, etc. as the submitted recipes.

As stated above, led means the code will actually run commands on swarming. This means the code being tested can impact production resources, e.g. clobber a Google Storage file, break Skylab DUTs, etc. Do NOT blindly test code with led, and use the staging environment where possible, which may limit the blast radius of bugs.

To use led, first call get-build or get-builder to get a job definition. For example, to test a change to the Annealing builder, we could start by running

led get-builder staging:staging-Annealing > builder_def.json

which will produce a JSON object containing properties, swarming information, etc. Sometimes, builders require properties that are not set in the lucicfg definition. For example, build_target requires the gitiles_commit property. In this case, we could get a definition from a previous build. For example, say the last build of staging-amd64-generic-postsubmit has id 12345. To get a job definition with the same properties (including gitiles_commit), run the command

led get-build 12345 > builder_def.json

Next, the led edit-recipe-bundle command patches your local recipes edits into the job definition, and led launch launches the job on swarming. The job definition is piped from each command to the next, so in practice an example command would be

led get-build 12345 | led edit-recipe-bundle | led launch

The output from led launch should give a link to the swarming task, e.g.

[I 2019-05-06 15:31:43] Launched swarming task: https://chrome-swarming.appspot.com/task?id=44a7e0cb7a0d6310
[I 2019-05-06 15:31:43] LUCI UI: https://ci.chromium.org/swarming/task/44a7e0cb7a0d6310?server=chrome-swarming.appspot.com

Test Recipes Presubmit

Recipe changes run through a presubmit job that verifies them on actual ChromeOS CI builders (essentially automating the led testing process).

The builders to be run are controlled with the builders property on the test_recipes recipe. Builders will not be run if the recipe change does not affect them (as determined by ./recipes.py analyze, which checks if the recipe or any of its dependencies has been modified).

If you are confident that your change does not need testing, you can include the Test-Recipes-Skip-Builder footer in your CL. That this should be done sparingly. If the presubmit is flaky, the root cause should be fixed. Please put a justification in the CL description for why the builder is skipped. The footer syntax is a bit picky; an example commit message would look like

A test commit

- Note: Skipping testers because this change must be submitted now.

Test-Recipes-Skip-Builder: Builder1
Test-Recipes-Skip-Builder: Builder2
Change-Id: 1234

Use depot_tools/git-footers to verify the footer syntax. For example, the above commit returns

Test-Recipes-Skip-Builder: Builder2
Test-Recipes-Skip-Builder: Builder1
Change-Id: 1234

Note that adding a space between footers is not allowed. For example, the commit message

A test commit

- Note: Skipping testers because this change must be submitted now.

Test-Recipes-Skip-Builder: Builder1
Test-Recipes-Skip-Builder: Builder2

Change-Id: 1234

will return

Change-Id: 1234

and thus the builders will not be skipped. Also note that each builder to skip must have its own line; Test-Recipes-Skip-Builder: Builder1, Builder2 will not work.