Solution of e2e timed issue
Issue
When running the e2e tests, all e2e tests failed because we can’t get build data from BuildKite.
Reason
- for
free useron BuildKite, BuildKite will save pipeline build data only for 90 days. So after 90 days of the time selected in e2e test, we will not get build data from BuildKite.
Solution
- Modify e2e test regularly
Estimated the time of next modification
- the time we select mainly is from
2024-08-12to2024-09-03in the e2e test, so according to my personal estimate that next modified time should be around2024-11-13. - From
2024-11-13to2024-12-03, the steps are obtained but the result will be changed, because the e2e test can get only partial build data. - After
2024-12-03, the e2e test can’t get any build data from the2024-06-03to2024-06-07time range.
Tips: The estimated time may not be completely accurate
Summary steps
Tips: The following steps are based on our experience and are not required to be followed. If there is any more excellent experience, you can modify the following steps.
| Step | Content | Comments |
|---|---|---|
| Step 1. Configure the latest time period in the local browser and get the report data | 1. Change the time period to a time period that can get BuildKite build data | It’s recommended to change to a time period close to the current date, because this can reduce the number of e2e test modifications. And select multiple time periods, because only selecting one time range can’t go to chart page |
| 2. According to the time period, input all tokens | Select all the metrics and input all tokens. If any metrics are not required in the e2e test, we can modify the configuration file | |
| 3. Input all information that are required, for example, classification, crews and cycle time. Next go to report page | It’s recommended to select all | |
| 4. Export the config file and all the csv file | In the e2e test, we can directly use the config file to e2e test, and the local csv file can also use these csv files | |
| 5. Save the report data to local file, such as images | When you use the config file to e2e test, you will change the report data to assert the report data. And you can directly get these data in the local file without getting the data by executing e2e test multiple times | |
| Step 2. Modify the e2e test | 1. For the e2e test about create a new project, we will modify the time periods and all the information by config file and modify the report result data from the Step 1 | None |
2. For the e2e test about import from config file, change the config file name to the e2e test config file name, and replace origin input-files/*.template.json file | If we directly use the origin file name, we will not need to modify the code. And we only upload input-files/*.template.json file to repo rather than input-files/*.json, so we must modify the *.template.json file | |
3. In the *.template.json file, for data security, we must replace token with placeholder of <E2E_TOKEN_JIRA> , <E2E_TOKEN_BUILD_KITE>，<E2E_TOKEN_GITHUB> like the previous file | We can copy the previous placeholder to use the new placeholder, And when we run pnpm e2e, the generate-config-files.sh file will replace the placeholder with the real token. Tips: there is an e2e test to test the pipeline that is no organization in the unhappy path, and the placeholder is <E2E_TOKEN_PIPELINE_NO_ORG_CONFIG_BUILDKITE>, maybe you need to check if the token is correct | |
4. Modify local token file. We will modify the origin token to the new token in the .env.local file to ensure that the e2e test can be passed locally | In the local, the placeholder from the *.template.json file will be replaced with the token in the .env.local file. And the generate-config-files.sh file will create a new json file with the same name by *.template.json file. | |
| 5. Run the e2e test, and change the report data and local csv files by the Step 1 | If the previous steps are correct, the e2e test will go to the report page as in the browser. So you can input the data by Step 1 | |
| 6. If the download csv files are not equal to local csv files, change the local csv files to download files | When these files are not equal, you can print the download filename, and replace the local csv file with download file saved to temp directory. For board csv file, the current jira cards will be added after the blank line in the file, but at different times, these cards are not same, so when we parse the local csv file and download csv file, we set the number of the lines that need to be parsed, so the value that is usually equal to 1 + the number of non-done cards in the BOARD_CSV_COMPARED_LINES of create-new/report-result.ts file may be modified | |
| 7. Repeat Step 1 and Step 2, until all e2e tests are passed in the local | Tips: For special tests, we need to prepare special time periods. For example, in the unhappy path when import file, we will test the function that board configuration don’t exist in the metrics page, so the board data should not exist in the time periods that we select, and the time periods usually are holiday, such as from 2024-05-01 to 2024-05-05 | |
8. Close the playwright and rerun the cmd of pnpm e2e to re-test the e2e test | The reason is to ensure that all e2e tests are passed in the local env. Becausethe changes of *.template.json will not be hot replacement, however, when we run the pnpm e2e command, the json file will be generated by generate-config-files.sh file | |
| Step 3. Modify the remote env to ensure that all e2e tests are passed in the remote env | 1. Change the Github token to the correct token | Not sure if permissions are needed |
2. Change the AWS token to the correct token | login to AWS by HeartBeatKeyPair.pem in the google drive, and by the cmd of vim /etc/buildkite-agent/hooks/environment to change the env and modify the token to the correct token. Permissions are needed |