sot-doc  1.1.0
Documentation entry point for the Stack-Of-Tasks
doc/Tutorial_Release_SOT_Packages.md
1 # Release of a SoT package on ROS {#tutorial_release_sot_packages}
2 
3 \section Tutorial_introduction Introduction
4 
5 This tutorial will help to realease any new SoT package and to maintain already released package on ROS.
6 
7 Before any release you should test the prerelease of the package to see if there is no errors remaining.
8 
9 \section prerelease Prerelease on the local machine to test for issues
10 
11 \subsection Prerequisite
12 
13 Docker need to be installed on the computer (ref.: https://docs.docker.com/engine/install/ubuntu/ ) :
14 
15  sudo apt-get update
16  sudo apt-get install docker-ce docker-ce-cli containerd.io
17 
18 Test if the docker engine is correctly installed:
19 
20  sudo docker run hello-world
21 
22 
23 \subsubsection ROS Set up your PC for a prerelease if ROS is already installed
24 
25 Install python3 ros buildfarm in order to generate the script needed for the prerelease :
26 
27  sudo apt-get install python3-ros-buildfarm
28 
29 \subsubsection not_ROS Set up the PC for a prerelease if ROS is not installed yet
30 
31 Setup of apt-get sources to include packages.ros.org. Run the following commands :
32 
33  sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
34  sudo apt-key adv --keyserver hkp://pool.sks-keyservers.net --recv-key 421C365BD9FF1F717815A3895523BAEEB01FA116
35  sudo apt-get update
36 
37 \subsection prerelease_ros Creating a Prerelease Command
38 
39 Go on this site : http://prerelease.ros.org/
40 
41 Choose your ros distro
42 
43 Then select the package you want to test in the scrolling menu if your package is already on the buildfarm. If not push the red cross and add a custom repository.
44 
45 Push the next button.
46 
47 After a little time the site search for dependencies to test and then you can push the generate command button.
48 
49 Copy/paste the code on the computer terminal.
50 
51 It should create 5 .sh files inside /tmp/prerelease_job/
52 
53 Launch the prerelease with :
54 
55  ./prerelease.sh
56 
57 If everything went well you will arrive until the 11 job where the test should be done. We are not using those catkin test for the SoT packages. You should have the prerelease finishing with :
58 
59  ...
60 
61  Build step 10
62 
63  Build step 11
64  # BEGIN SECTION: Create collated test stats dir
65  # END SECTION
66 
67 
68  Test results of underlay workspace
69 
70  Invoking: catkin_test_results /tmp/prerelease_job/ws/test_results --all
71  Test results directory "/tmp/prerelease_job/ws/test_results" does not exist
72 
73 
74  Skipping empty overlay workspace
75 
76 If not search what is wrong with the package. It could be :
77 
78 - Missing dependencies
79 - Compilation error
80 - Connection error
81 
82 \subsection prerelease_error Possible errors during the prerelease
83 
84 Error :
85 
86  Permission denied to access config.json
87 
88 Solution :
89 
90  sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
91 
92 Error :
93 
94  permission denied for the creation of .ccmake/tmp
95 
96 Solution :
97 
98  sudo chown -R $USER:$USER /home/$USER/.ccache
99 
100 If a new issue is met : https://docs.docker.com/
101 
102 \subsection clean_docker Clean docker
103 
104 After a prerelease there are still generated files left on the computer and it will soon take too much space.
105 
106 Enter :
107 
108  docker system prune -a
109 
110 To remove all the unused images, containers, volume and network of docker.
111 
112 \section first_time_release Release a package for the first time
113 
114 \subsection prerequisite_intro Prerequisite
115 
116 Bloom need to be installed (ref.: http://ros-infrastructure.github.io/bloom/ ) :
117 
118  sudo apt-get install python-bloom
119 
120 To allow bloom to generate automatic pull request make sure to have a file ~/.config/bloom well initiated :
121 
122  {
123  "github_user": "<github-username>",
124  "oauth_token": "<token-you-created-for-bloom>"
125  }
126 
127 Log in to the GitHub account and generate a new token with “public_repo” granted.
128 It is situated inside the GitHub account settings under Developper settings and Personal access token
129 
130 Copy this code inside the bloom file. For example:
131 
132  {
133  "github_user": "Rascof",
134  "oauth_token": "295a7188fbfc36f9esdsdsdqssjd21s2d1"
135  }
136 
137 If you have a two-factor authentication enabled follow this tutorial (Step 2) : http://wiki.ros.org/bloom/Tutorials/GithubManualAuthorization
138 
139 Make sure that the .gitmodules file is clean without unmet path target. Because it can result in errors during the release.
140 
141 \subsection release_repository Create a github repository for the release
142 
143 For the first release, create a github repository with the name : <package-name>-ros-release
144 
145 Check the Create README.md button when creating it so that the first commit is created.
146 
147 Create a tracks.yaml file inside the repository like this (example at : https://github.com/ipab-slmc/eigenpy_catkin-release/blob/master/tracks.yaml) :
148 
149  tracks:
150  <ros_distro>:
151  actions:
152  - bloom-export-upstream :{vcs_local_uri} :{vcs_type} --tag :{release_tag} --display-uri
153  :{vcs_uri} --name :{name} --output-dir :{archive_dir_path}
154  - git-bloom-import-upstream :{archive_path} :{patches} --release-version :{version}
155  --replace
156  - git-bloom-generate -y rosrelease :{ros_distro} --source upstream -i :{release_inc}
157  - git-bloom-generate -y rosdebian --prefix release/:{ros_distro} :{ros_distro}
158  -i :{release_inc} --os-name ubuntu
159  - git-bloom-generate -y rosdebian --prefix release/:{ros_distro} :{ros_distro}
160  -i :{release_inc} --os-name debian --os-not-required
161  - git-bloom-generate -y rosrpm --prefix release/:{ros_distro} :{ros_distro} -i
162  :{release_inc}
163  devel_branch: devel
164  last_release: v<MAJOR.MINOR.PATCH>
165  last_version: <MAJOR.MINOR.PATCH>
166  name: <package_name>
167  patches: null
168  release_inc: '1'
169  release_repo_url: https://github.com/stack-of-tasks/<package-name>-ros-release
170  release_tag: v:{version}
171  ros_distro: <ros_distro>
172  vcs_type: git
173  vcs_uri: https://github.com/stack-of-tasks/https://github.com/stack-of-tasks/<package-name>
174  version: :{auto}
175 
176 Make sure to update:
177 
178 - <ros_distro> to the distro version wanted. To release to several versions just copy past and change the <ros_distro>
179 
180 - <MAJOR.MINOR.PATCH> to the newest version of the released package
181 
182 - <package-name>
183 
184 \subsection release Releasing the package for the first time
185 
186  # Replace <ros_distro> with the ROS distribution, e.g. melodic; noetic
187  # Replace <package-name> with the name used in the ROS distro file
188  bloom-release --rosdistro <ros_distro> --track <ros_distro> <package-name>
189 
190 or
191 
192  bloom-release --rosdistro <ros_distro> --track <ros_distro> <package-name> --edit
193 
194 To edit the tracks.yaml file (ref : https://wiki.ros.org/bloom/Tutorials/FirstTimeRelease#Releasing_Your_Packages )
195 
196 There will be this error showing :
197 
198  Specified repository 'dynamic-graph-release' is not in the distribution file located at 'https://raw.githubusercontent.com/ros/rosdistro/master/melodic/distribution.yaml'
199  Could not determine release repository url for repository 'dynamic-graph-release' of distro 'melodic'
200  You can continue the release process by manually specifying the location of the RELEASE repository.
201  To be clear this is the url of the RELEASE repository not the upstream repository.
202  For release repositories on GitHub, you should provide the `https://` url which should end in `.git`.
203  Here is the url for a typical release repository on GitHub: https://github.com/ros-gbp/rviz-release.git
204  ==> Looking for a release of this repository in a different distribution...
205  No reasonable default release repository url could be determined from previous releases.
206  Release repository url [press enter to abort]:
207 
208 This is ok because the package is not yet on the build farm
209 
210 Enter :
211 
212  https://github.com/<name-of-account>/<package-name>-ros-release.git
213 
214 Don't forget the .git at the end.
215 
216 Enter the login and password for the release repository to push different branches.
217 
218 Then respond y in order to add information:
219 
220 - For documentation on the package:
221 
222  VCS Type must be one of git, svn, hg, or bzr.
223  VCS type: git
224  VCS url: https://github.com/stack-of-tasks/<package-name>.git
225  VCS version must be a branch, tag, or commit, e.g. master or 0.1.0
226  VCS version: devel
227 
228 - For source entry on the package:
229 
230  VCS Type must be one of git, svn, hg, or bzr.
231  VCS type: git
232  VCS url: https://github.com/stack-of-tasks/<package-name>.git
233  VCS version must be a branch, tag, or commit, e.g. master or 0.1.0
234  VCS version: devel
235 
236 - For testing and maintenance statue
237 
238  Would you like to turn on pull request testing? [y/N]? n
239  Would you like to add a maintenance status for this repository? [Y/n]? y
240  Please enter a maintenance status.
241  Valid maintenance statuses:
242  - developed: active development is in progress
243  - maintained: no new development, but bug fixes and pull requests are addressed
244  - unmaintained: looking for new maintainer, bug fixes and pull requests will not be addressed
245  - end-of-life: should not be used, will disappear at some point
246  Status: maintained
247 
248 Then at the end it should ask to generate a pull request to add the package to the buildfarm.
249 
250  Continue [Y/n]? y
251 
252 If there is an error at this stage, make sure to have a ~/.config/bloom well initiated or follow this tutorial if you have two-factor authentication enabled : http://wiki.ros.org/bloom/Tutorials/GithubManualAuthorization
253 
254 or if the error is :
255 
256  ! [remote rejected] bloom-sot-core-0 -> bloom-sot-core-0 (refusing to allow a Personal Access Token to create or update workflow `.github/workflows/build_test.yaml` without `workflow` scope)
257  error: failed to push some refs to 'https://678b1f245c644b91864fd46c9732368c0ab93f0e:x-oauth- basic@github.com/nim65s/rosdistro.git'
258  Failed to open pull request: CalledProcessError - Command 'git push https:// 678b1f245c644b91864fd46c9732368c0ab93f0e:x-oauth-basic@github.com/nim65s/rosdistro.git bloom-sot- core-0' returned non-zero exit status 1
259 
260 Create another token with workflow enabled.
261 
262 
263 \section maintained Maintain packages already on the buildfarm
264 
265 Run :
266 
267  # Replace <ros_distro> with the ROS distribution, e.g. melodic; noetic
268  # Replace <package-name> with the name used in the ROS distro file
269  bloom-release <package-name> --rosdistro <ros_distro>
270 
271 Enter the login and password two times for the release repository to push different branches.
272 
273 At the end it should generate a pull request on GitHub to update the version on the GitHub repository of ROS : https://github.com/ros/rosdistro
274 
275 If not make sure to have a ~/.config/bloom well initiated or follow this tutorial if you have two-factor authentication enabled : http://wiki.ros.org/bloom/Tutorials/GithubManualAuthorization
276 
277 \section patch ROS Patch
278 
279 This section shows how to manage patches. If there is an installation rule specific to ROS you can use patches. Patches are a rule to modify the source code only when installed with ROS.
280 For more insight : http://wiki.ros.org/bloom/Tutorials/ReleaseThirdParty from section 2.4 to 2.6
281 
282 \subsection patch1 Create a patch
283 
284 Clone the ros release repository on the computer
285 
286  git clone https://github.com/<name-of-account>/<package-name>-ros-release
287  cd <package-name>-ros-release
288 
289 Enter the branch of the ros release repository where the source code need to be changed
290 
291  git checkout release/<ros_distro>/<package-name>
292 
293 Make the modifications
294 
295  git add .
296  git commit -m "<Comments>"
297  git-bloom-patch export
298  git push
299  git-bloom-release <ros_distro>
300 
301 Finally, procceed with a realease on ROS.
302 
303 \subsection patch2 Remove patches
304 
305 There is a way to remove all patches (I did not find a way to remove only one patch)
306 
307 Clone the ros release repository on the computer
308 
309  git clone https://github.com/<name-of-account>/<package-name>-ros-release
310  cd <package-name>-ros-release
311 
312 Enter the branch of the ros release repository where the source code need to be changed
313 
314  git checkout release/<ros_distro>/<package-name>
315 
316 Remove the patches
317 
318  git-bloom-patch remove
319  git push --all
320  git-bloom-release <ros_distro>
321 
322 Finally, procceed with a realease on ROS.
323 
324 \subsection patch3 Porting patches between ros version
325 
326 Port patches from one distro to another
327 
328  git checkout patches/release/<new_ros_distro>/<package-name>
329  git ls-tree --name-only -r patches/release/<former_ros_distro>/<package-name> | grep '\.patch' | xargs -I {} sh -c 'git show patches/release/<former_ros_distro>/<package-name>:"$1" > "$1"' -- {}
330  git add .
331  git commit -m "Importing patches from <former_ros_distro> to <new_ros_distro>"
332  git checkout release/<new_ros_distro>/<package-name>
333  git-bloom-patch import
334  git push --all
335  git push --tags
336 
337 Finally, procceed with a realease on ROS
338 
339 \section check Check if the build succeded on the build farm
340 
341 When the release is done and the pull request is accepted by ROS developpers the package will automatically be build on the ROS build farm with Jenkins.
342 
343 The maintainer of the GitHub repository specified inside the package.xml will receive an e-mail with the success or failure of the build.
344 
345 There is a need to check if all jobs succeded on the site where the ROS build farm is held : http://build.ros.org/
346 
347 Search for the name of the package and check every job.
348 
349 The first letter of the jobs is the ROS distro version :
350 
351 - M for melodic
352 - N for noetic
353 
354 \section TestingRepository Testing Repository
355 
356 When the package is finally released it is not yet on the official build farm. It is in a soaking area where you can test your packages before it is released on the public ROS build farm. For more insight : http://wiki.ros.org/action/show/TestingRepository?action=show&redirect=ShadowRepository
357 
358 To change to ros testing :
359 
360 Change ros into ros-testing inside this file /etc/apt/sources.list.d/ros-latest.list
361 
362 Like this :
363 
364  deb http://packages.ros.org/ros-testing/ubuntu bionic main
365 
366 Then :
367 
368  sudo apt-get update
369  sudo apt-get install ros-<ros_distro>-<package-name>
370 
371 To go back to the official ROS build farm change back to ros on /etc/apt/sources.list.d/ros-latest.list
372 
373 Then :
374 
375  sudo apt-get update
376  sudo apt-get dist-upgrade
377