Commit Graph

89 Commits

Author SHA1 Message Date
Max Kainov
3247ae81e4 move out ci specific functions to ci.py
#no_merge_commit
2024-01-21 16:36:37 +00:00
Max K
84e5870b71
Reapply "improve CI with digest for docker, build and test jobs" (#57904)
* Revert "Revert "improve CI with digest for docker, build and test jobs""

* fix: docker manifest merge for missing images only
2023-12-18 09:07:22 +01:00
Max K
8c7add0334 Revert "improve CI with digest for docker, build and test jobs (#56317)"
This reverts commit 7844fcc196.
2023-12-15 15:48:01 +01:00
Max K
7844fcc196
improve CI with digest for docker, build and test jobs (#56317)
* CI with digests to tag docker images, reuse builds, reuse tests and others
2023-12-15 15:21:20 +01:00
Mikhail f. Shiryaev
ff58e152d8
Remove os.path from docker related scripts 2023-09-28 01:20:36 +02:00
Mikhail f. Shiryaev
fde8817fb7
Fix PRInfo and docker_images_check typing 2023-09-15 14:54:42 +02:00
János Benjamin Antal
5862c4ec93
Use not the absolute latest docker image tag (#54369)
* Add params for select queries in clickhouse_helper.py

* Find the latest image tags from master

* Get full git history

* Do not checkout submodules

* Automatic style fix

* Fix style errors

* Make logging pretty

* Revert accidentally committed changes

* Eliminate pformat

* Update all workflows

* Remove flag with default value

* Improve exception handling

* Utilizing the loop condition in while loop

* Make the logic robust over temporary errors

* Prettify logs

* Automatic style fix

* Add accidentally removed line

* Revert "Make the logic robust over temporary errors"

This reverts commit 91e6b5d7ba.

* Do not fail in case the tag cannot be queried from ClickHouse

* Support partial finding of tags

---------

Co-authored-by: robot-clickhouse <robot-clickhouse@users.noreply.github.com>
2023-09-08 08:42:56 +02:00
Mikhail f. Shiryaev
cbd81b3758
Partially revert changes from #52395 2023-08-10 22:09:19 +02:00
Azat Khuzhin
0b258dda4e Reproducible builds for Rust
From now on cargo will not download anything from the internet during
builds. This step had been moved for docker image builds (via cargo
vendor).

And now cargo inside docker.io/clickhouse/binary-builder will not use
any crates from the internet, so we don't need to add --offline for
cargo commands in cmake (corrosion_import_crate()).

Also the docker build command had been adjusted to allow following
symlinks inside build context, by using tar, this is required for Rust
packages.

Note, that to make proper Cargo.lock that could be vendored I did the
following:
- per-project locks had been removed (since there is no automatic way to
  sync the workspace Cargo.lock with per-project Cargo.lock, since cargo
  update/generate-lockfile will use only per-project Cargo.toml files
  apparently, -Z minimal-versions does not helps either)
- and to generate Cargo.lock with less changes I've pinned version in
  the Cargo.toml strictly, i.e. not 'foo = "0.1"' but 'foo = "=0.1"'
  then the Cargo.lock for workspace had been generated and afterwards
  I've reverted this part.

Plus I have to update the dependencies afterwards, since otherwise there
are conflicts with dependencies for std library. Non trivial.

Signed-off-by: Azat Khuzhin <a.khuzhin@semrush.com>
2023-07-22 22:46:22 +02:00
Mikhail f. Shiryaev
df33b99907
Rework post_commit_status, add set_status_comment there 2023-04-28 23:52:31 +02:00
Mikhail f. Shiryaev
4f2c336cfc
Update black version, run it for the repository 2023-03-24 11:54:38 +01:00
Mikhail f. Shiryaev
0ab9d6787e
Add format_description for commit status descriptions 2023-02-23 15:21:19 +01:00
Mikhail f. Shiryaev
d0f5b6efb5
Use both release_pr and merged_pr as additional cache 2023-02-09 21:31:12 +01:00
Mikhail f. Shiryaev
7c6bb55eca
Use TeePopen to get logs in the main log 2023-02-09 20:36:39 +01:00
Mikhail f. Shiryaev
db96f9e3db
Rework test_results list/tuple/whatever into class 2023-01-09 12:09:47 +01:00
Alexey Milovidov
851e4ae59f Fix typos 2023-01-08 07:08:20 +01:00
Mikhail f. Shiryaev
6646950f3c
Get rid of unused set-output 2022-12-20 11:40:22 +01:00
Mikhail f. Shiryaev
78e560f78e
Fix docker_image_check and docker_test 2022-11-29 12:51:47 +01:00
Mikhail f. Shiryaev
9584487a1e
Use docker images cache from merged PRs in master and release branches 2022-11-25 14:09:21 +01:00
Mikhail f. Shiryaev
a760c71a0b
Fix the typo preventing building latest images 2022-09-26 12:52:12 +02:00
Mikhail f. Shiryaev
2dff307770
Build latest tags ONLY from master branch 2022-09-20 14:31:04 +02:00
Mikhail f. Shiryaev
372481e770
Rework S3Helper a little bit 2022-08-11 15:01:32 +02:00
Mikhail f. Shiryaev
2fb68078e8
Replace S3 URLs by parameter 2022-08-10 15:22:04 +02:00
Yakov Olkhovskiy
1a52fa183d rename Simple Check to Mergeable Check, refactor processing 2022-08-05 10:39:58 -04:00
Mikhail f. Shiryaev
d42018a2dc
Cleanup the rest of (actions) status names 2022-07-21 13:10:22 +02:00
Mikhail f. Shiryaev
27bd355525
Use cached image from the release branch for backport PRs 2022-07-01 16:49:27 +02:00
Mikhail f. Shiryaev
9ca03d09cc
Add a way to invalidate docker images cache 2022-06-08 14:47:45 +02:00
Mikhail f. Shiryaev
501a8158a5
Don't fail docker images on dispatch or other events 2022-05-24 11:14:14 +02:00
Mikhail f. Shiryaev
44895e5287
Improve build cache for docker images
- Save all layer's cache
- Use cache with fallback to a latest tags
2022-04-07 17:16:43 +02:00
Mikhail f. Shiryaev
00a3734078
Cosmetic improvement 2022-04-01 12:32:24 +02:00
Mikhail f. Shiryaev
5212ff0146
Improve docker_images_check logging 2022-04-01 11:59:48 +02:00
alesapin
cf6898d05b Merge branch 'master' into fail_build_check_in_error_status 2022-03-30 11:08:24 +02:00
Nikolai Kochetov
3849e63ab1 Change database 2022-03-29 19:06:50 +00:00
alesapin
27390799c1 Fix black 2022-03-29 18:23:18 +02:00
alesapin
ebd72e433d Fail checks in case of errors 2022-03-29 14:41:47 +02:00
Mikhail f. Shiryaev
d6fc6b9c44
Add build-url label to built docker images 2022-03-24 16:24:06 +01:00
Mikhail f. Shiryaev
dd180b83a8
Improve --no-* parameters for scripts 2022-02-22 11:40:39 +01:00
Mikhail f. Shiryaev
0f21981694
Add a flag for images architecture 2022-02-11 18:26:45 +01:00
Mikhail f. Shiryaev
dcecbbaf81
Add argument --all for docker imabes builder 2022-02-11 18:26:42 +01:00
Mikhail f. Shiryaev
fb908a31e2
Fix dependencies tree for docker image check 2022-01-25 12:09:10 +01:00
Mikhail f. Shiryaev
70d997cbda
Fix pylint issues 2022-01-24 14:54:37 +01:00
Mikhail f. Shiryaev
18ef029ed3
Fix issue with building from non-existing tags 2022-01-24 14:54:29 +01:00
Mikhail f. Shiryaev
92099302e7
Use stdin for docker login to not expose a password 2022-01-13 16:57:33 +01:00
Mikhail f. Shiryaev
2ae87ad0df
Build docker amd64 and arm64 separately and merge together later 2022-01-12 17:07:55 +01:00
Mikhail f. Shiryaev
7bc89c4f50
Use cache feature of docker buildx 2022-01-12 17:07:53 +01:00
Mikhail f. Shiryaev
2a83a1c05d
Build dependent images from the current version 2022-01-12 17:07:52 +01:00
Mikhail f. Shiryaev
fe130bc547
Use buildx for docker images, improve typing 2022-01-12 17:07:50 +01:00
Mikhail f. Shiryaev
43f39bc051
Add typing to docker_images_check.py and ignore dependencies errors 2022-01-12 17:07:50 +01:00
Mikhail f. Shiryaev
3c6f14ae63
Apply black to docker_images_check.py 2022-01-12 17:07:49 +01:00
Constantine Peresypkin
7c90d4f8e3 refactor CI tests 2021-12-03 12:22:50 +02:00