Actions
Idea #16264
closed
Handle R SDK dependencies better
Status:
Resolved
Priority:
Normal
Assigned To:
Ward Vandewege
Category:
SDKs
Target version:
Start date:
Due date:
Story points:
-
Release:
Release relationship:
Auto
Description
Our build was broken for a while because Rcpp (a transitive dependency of a package we depend on) had a new release that is broken on R 3.3.
We were running our tests on Debian 9, which ships with R 3.3.
A fix was committed to upstream Rcpp, so the problem resolved itself, but we should have a more stable strategy for R dependencies.
Recompiling R dependencies every time is also very expensive.
Options¶
Use Debian packages¶
When we did the R SDK originally (about two years ago) most of the packages we needed were not in Debian. Since then, it appears they have been added to Debian 10, and are also available in a Debian 9 backport:
https://packages.debian.org/search?keywords=r-cran-devtools&searchon=names&suite=all§ion=all
Of course this requires targeting Debian for the test builds, but we do that already. To get the maximum benefit it also requires that all the R packages we need are in Debian.
Pinned install¶
R doesn't have anything like a Gemfile.lock or "pip freeze". However someone has created a tool that computes an install plan so each package is installed at exactly each version we want.
This works on any R install. The drawback is that we still end up compiling the whole world every time.
Files
Updated by Peter Amstutz about 6 years ago
- Related to Bug #16267: arvbox build uses arvados-server install added
Updated by Peter Amstutz about 6 years ago
Currently I think the solution is for #16053 to include installing the appropriate R packages from Debian 10.
Updated by Peter Amstutz about 6 years ago
- Related to Idea #16053: [boot] subcommand to install/update dev and runtime dependencies added
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-04-08 Sprint to 2020-04-22
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-04-22 to 2020-05-06 Sprint
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-05-06 Sprint to 2020-05-20 Sprint
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-05-20 Sprint to 2020-06-03 Sprint
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-06-03 Sprint to 2020-06-17 Sprint
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-06-17 Sprint to 2020-07-01 Sprint
Updated by Peter Amstutz almost 6 years ago
- Target version changed from 2020-07-01 Sprint to 2020-07-15
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2020-07-15 to 2020-08-12 Sprint
Updated by Peter Amstutz over 5 years ago
- Related to Bug #16586: ArvadosR: Dependency package(s) 'XML' not available. added
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2020-08-12 Sprint to 2020-08-26 Sprint
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2020-08-26 Sprint to 2020-09-09 Sprint
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2020-09-09 Sprint to 2020-09-23 Sprint
Updated by Peter Amstutz over 5 years ago
- Target version changed from 2020-09-23 Sprint to 2020-10-07 Sprint
Updated by Peter Amstutz over 5 years ago
Now that all the test infrastructure (including arvbox) is on Debian 10, we can install the following packages for development and everything works great:
r-cran-devtools r-cran-knitr r-cran-markdown r-cran-roxygen2 r-cran-xml
Updated by Ward Vandewege over 5 years ago
- Status changed from New to In Progress
Updated by Ward Vandewege over 5 years ago
Peter Amstutz wrote:
Now that all the test infrastructure (including arvbox) is on Debian 10, we can install the following packages for development and everything works great:
r-cran-devtools r-cran-knitr r-cran-markdown r-cran-roxygen2 r-cran-xml
I've added those packages to the dependencies that are installed by `arvados-server install -type test`.
A quick test in arvbox showed a large reduction in the time needed to build the R documentation, I recorded 6.41s for the old way (from flock release to done), and 1:23s with this patch.
Ready for review at 4adf06fe21ff6aa9c08afee2268871c85394b217 on branch 16264-add-r-dependencies.
This is the log for the R doc generation with the patch applied:
2020-09-24_19:36:24.30744 + [[ ! -f /usr/src/arvados/doc/no-sdk ]]
2020-09-24_19:36:24.30746 + cd /usr/src/arvados/sdk/R
2020-09-24_19:36:24.30748 + R --quiet --vanilla --file=install_deps.R
2020-09-24_19:36:24.47131 > # Copyright (C) The Arvados Authors. All rights reserved.
2020-09-24_19:36:24.47134 > #
2020-09-24_19:36:24.47134 > # SPDX-License-Identifier: Apache-2.0
2020-09-24_19:36:24.47134 >
2020-09-24_19:36:24.47134 > options(repos=structure(c(CRAN="http://cran.wustl.edu/")))
2020-09-24_19:36:24.47136 > if (!requireNamespace("devtools")) {
2020-09-24_19:36:24.47138 + install.packages("devtools")
2020-09-24_19:36:24.47139 + }
2020-09-24_19:36:24.47272 Loading required namespace: devtools
2020-09-24_19:36:25.04066 > if (!requireNamespace("roxygen2")) {
2020-09-24_19:36:25.04069 + install.packages("roxygen2")
2020-09-24_19:36:25.04069 + }
2020-09-24_19:36:25.04086 Loading required namespace: roxygen2
2020-09-24_19:36:25.21787 > if (!requireNamespace("knitr")) {
2020-09-24_19:36:25.21789 + install.packages("knitr")
2020-09-24_19:36:25.21790 + }
2020-09-24_19:36:25.21790 > if (!requireNamespace("markdown")) {
2020-09-24_19:36:25.21790 + install.packages("markdown")
2020-09-24_19:36:25.21791 + }
2020-09-24_19:36:25.21806 Loading required namespace: markdown
2020-09-24_19:36:25.22394 > if (!requireNamespace("XML")) {
2020-09-24_19:36:25.22396 + # XML 3.99-0.4 depends on R >= 4.0.0, but we run tests on debian
2020-09-24_19:36:25.22397 + # stable (10) with R 3.5.2 so we install an older version from
2020-09-24_19:36:25.22397 + # source.
2020-09-24_19:36:25.22397 + install.packages("https://cran.r-project.org/src/contrib/Archive/XML/XML_3.99-0.3.tar.gz", repos=NULL, type="source")
2020-09-24_19:36:25.22399 + }
2020-09-24_19:36:25.22412 Loading required namespace: XML
2020-09-24_19:36:25.39980 >
2020-09-24_19:36:25.39982 > devtools::install_dev_deps()
2020-09-24_19:36:27.53800
2020-09-24_19:36:29.26694
2020-09-24_19:36:29.26708 >
2020-09-24_19:36:29.29854 + test '' = --only-deps
2020-09-24_19:36:29.29856 + cd /usr/src/arvados/doc
2020-09-24_19:36:29.29864 + bundle exec rake generate baseurl=http://172.17.0.2:8001 arvados_api_host=172.17.0.2:8000 arvados_workbench_host=http://172.17.0.2
2020-09-24_19:36:29.78374 zenweb-fix-body.rb:4: warning: already initialized constant ZenwebTextile::VERSION
2020-09-24_19:36:29.78379 zenweb-textile.rb:8: warning: previous definition of VERSION was here
2020-09-24_19:36:29.78400 rm -rf sdk/python/arvados
2020-09-24_19:36:29.78410 rm -rf sdk/R
2020-09-24_19:36:29.78947 rm -rf sdk/java-v2/javadoc
2020-09-24_19:36:29.79626 rm -rf
2020-09-24_19:36:29.79634 rm -rf .site
2020-09-24_19:36:29.82145 ^[[0;91;49mWarning: pdoc3 not found, Python documentation will not be generated^[[0m
2020-09-24_19:36:29.83433 cp css/R.css sdk/R/arvados
2020-09-24_19:36:29.83434 tgt
2020-09-24_19:36:29.83434 /usr/src/arvados/doc
2020-09-24_19:36:30.59352 Loading required package: knitr
2020-09-24_19:36:30.59354 Loading required package: markdown
2020-09-24_19:36:30.59354
2020-09-24_19:36:30.59354
2020-09-24_19:36:30.59354 processing file: README.Rmd
2020-09-24_19:36:30.59355 ^M | ^M | | 0%^M | ^M |. | 1%
2020-09-24_19:36:30.59355 ordinary text without R code
2020-09-24_19:36:30.59355
2020-09-24_19:36:30.59356 ^M | ^M |.. | 2%
2020-09-24_19:36:30.59356 label: unnamed-chunk-1 (with options)
2020-09-24_19:36:30.59356 List of 1
2020-09-24_19:36:30.59356 $ include: logi FALSE
2020-09-24_19:36:30.59357
2020-09-24_19:36:30.59357 ^M | ^M |.. | 3%
2020-09-24_19:36:30.59357 ordinary text without R code
2020-09-24_19:36:30.59357
2020-09-24_19:36:30.59358 ^M | ^M |... | 4%
2020-09-24_19:36:30.59358 label: unnamed-chunk-2
2020-09-24_19:36:30.59359 ^M | ^M |.... | 6%
2020-09-24_19:36:30.59359 ordinary text without R code
2020-09-24_19:36:30.59359
2020-09-24_19:36:30.59359 ^M | ^M |..... | 7%
2020-09-24_19:36:30.59360 label: unnamed-chunk-3 (with options)
2020-09-24_19:36:30.59360 List of 1
2020-09-24_19:36:30.59360 $ engine: chr "bash"
2020-09-24_19:36:30.59360
2020-09-24_19:36:30.59361 ^M | ^M |...... | 8%
2020-09-24_19:36:30.59361 ordinary text without R code
2020-09-24_19:36:30.59361
2020-09-24_19:36:30.59362 ^M | ^M |...... | 9%
2020-09-24_19:36:30.59362 label: unnamed-chunk-4 (with options)
2020-09-24_19:36:30.59362 List of 1
2020-09-24_19:36:30.59362 $ engine: chr "bash"
2020-09-24_19:36:30.59363
2020-09-24_19:36:30.59363 ^M | ^M |....... | 10%
2020-09-24_19:36:30.59364 ordinary text without R code
2020-09-24_19:36:30.59364
2020-09-24_19:36:30.59365 ^M | ^M |........ | 11%
2020-09-24_19:36:30.59365 label: unnamed-chunk-5
2020-09-24_19:36:30.59365 ^M | ^M |......... | 12%
2020-09-24_19:36:30.59366 ordinary text without R code
2020-09-24_19:36:30.59366
2020-09-24_19:36:30.59366 ^M | ^M |......... | 13%
2020-09-24_19:36:30.59366 label: unnamed-chunk-6
2020-09-24_19:36:30.59367 ^M | ^M |.......... | 15%
2020-09-24_19:36:30.59367 ordinary text without R code
2020-09-24_19:36:30.59367
2020-09-24_19:36:30.59367 ^M | ^M |........... | 16%
2020-09-24_19:36:30.59368 label: unnamed-chunk-7
2020-09-24_19:36:30.59368 ^M | ^M |............ | 17%
2020-09-24_19:36:30.59369 ordinary text without R code
2020-09-24_19:36:30.59369
2020-09-24_19:36:30.59369 ^M | ^M |............. | 18%
2020-09-24_19:36:30.59369 label: unnamed-chunk-8
2020-09-24_19:36:30.59370 ^M | ^M |............. | 19%
2020-09-24_19:36:30.59370 ordinary text without R code
2020-09-24_19:36:30.59370
2020-09-24_19:36:30.59371 ^M | ^M |.............. | 20%
2020-09-24_19:36:30.59371 label: unnamed-chunk-9
2020-09-24_19:36:30.59371 ^M | ^M |............... | 21%
2020-09-24_19:36:30.59371 ordinary text without R code
2020-09-24_19:36:30.59372
2020-09-24_19:36:30.59372 ^M | ^M |................ | 22%
2020-09-24_19:36:30.59372 label: unnamed-chunk-10
2020-09-24_19:36:30.59373 ^M | ^M |................. | 24%
2020-09-24_19:36:30.59373 ordinary text without R code
2020-09-24_19:36:30.59373
2020-09-24_19:36:30.59374 ^M | ^M |................. | 25%
2020-09-24_19:36:30.59374 label: unnamed-chunk-11
2020-09-24_19:36:30.59374 ^M | ^M |.................. | 26%
2020-09-24_19:36:30.59374 ordinary text without R code
2020-09-24_19:36:30.59375
2020-09-24_19:36:30.59375 ^M | ^M |................... | 27%
2020-09-24_19:36:30.59375 label: unnamed-chunk-12
2020-09-24_19:36:30.59375 ^M | ^M |.................... | 28%
2020-09-24_19:36:30.59376 ordinary text without R code
2020-09-24_19:36:30.59377
2020-09-24_19:36:30.59377 ^M | ^M |.................... | 29%
2020-09-24_19:36:30.59377 label: unnamed-chunk-13
2020-09-24_19:36:30.59378 ^M | ^M |..................... | 30%
2020-09-24_19:36:30.59378 ordinary text without R code
2020-09-24_19:36:30.59378
2020-09-24_19:36:30.59378 ^M | ^M |...................... | 31%
2020-09-24_19:36:30.59379 label: unnamed-chunk-14
2020-09-24_19:36:30.59379 ^M | ^M |....................... | 33%
2020-09-24_19:36:30.59379 ordinary text without R code
2020-09-24_19:36:30.59379
2020-09-24_19:36:30.59380 ^M | ^M |........................ | 34%
2020-09-24_19:36:30.59380 label: unnamed-chunk-15
2020-09-24_19:36:30.59380 ^M | ^M |........................ | 35%
2020-09-24_19:36:30.59381 ordinary text without R code
2020-09-24_19:36:30.59381
2020-09-24_19:36:30.59381 ^M | ^M |......................... | 36%
2020-09-24_19:36:30.59382 label: unnamed-chunk-16
2020-09-24_19:36:30.59382 ^M | ^M |.......................... | 37%
2020-09-24_19:36:30.59382 ordinary text without R code
2020-09-24_19:36:30.59382
2020-09-24_19:36:30.59383 ^M | ^M |........................... | 38%
2020-09-24_19:36:30.59383 label: unnamed-chunk-17
2020-09-24_19:36:30.59383 ^M | ^M |............................ | 39%
2020-09-24_19:36:30.59384 ordinary text without R code
2020-09-24_19:36:30.59384
2020-09-24_19:36:30.59384 ^M | ^M |............................ | 40%
2020-09-24_19:36:30.59385 label: unnamed-chunk-18
2020-09-24_19:36:30.59385 ^M | ^M |............................. | 42%
2020-09-24_19:36:30.59385 ordinary text without R code
2020-09-24_19:36:30.59386
2020-09-24_19:36:30.59386 ^M | ^M |.............................. | 43%
2020-09-24_19:36:30.59386 label: unnamed-chunk-19
2020-09-24_19:36:30.59386 ^M | ^M |............................... | 44%
2020-09-24_19:36:30.59387 ordinary text without R code
2020-09-24_19:36:30.59387
2020-09-24_19:36:30.59387 ^M | ^M |............................... | 45%
2020-09-24_19:36:30.59387 label: unnamed-chunk-20
2020-09-24_19:36:30.59388 ^M | ^M |................................ | 46%
2020-09-24_19:36:30.59388 ordinary text without R code
2020-09-24_19:36:30.59389
2020-09-24_19:36:30.59389 ^M | ^M |................................. | 47%
2020-09-24_19:36:30.59389 label: unnamed-chunk-21
2020-09-24_19:36:30.59390 ^M | ^M |.................................. | 48%
2020-09-24_19:36:30.59390 ordinary text without R code
2020-09-24_19:36:30.59390
2020-09-24_19:36:30.59390 ^M | ^M |................................... | 49%
2020-09-24_19:36:30.59391 label: unnamed-chunk-22
2020-09-24_19:36:30.59391 ^M | ^M |................................... | 51%
2020-09-24_19:36:30.59391 ordinary text without R code
2020-09-24_19:36:30.59391
2020-09-24_19:36:30.59392 ^M | ^M |.................................... | 52%
2020-09-24_19:36:30.59392 label: unnamed-chunk-23
2020-09-24_19:36:30.59392 ^M | ^M |..................................... | 53%
2020-09-24_19:36:30.59393 ordinary text without R code
2020-09-24_19:36:30.59393
2020-09-24_19:36:30.59393 ^M | ^M |...................................... | 54%
2020-09-24_19:36:30.59393 label: unnamed-chunk-24
2020-09-24_19:36:30.59394 ^M | ^M |....................................... | 55%
2020-09-24_19:36:30.59394 ordinary text without R code
2020-09-24_19:36:30.59394
2020-09-24_19:36:30.59394 ^M | ^M |....................................... | 56%
2020-09-24_19:36:30.59395 label: unnamed-chunk-25
2020-09-24_19:36:30.59395 ^M | ^M |........................................ | 57%
2020-09-24_19:36:30.59395 ordinary text without R code
2020-09-24_19:36:30.59395
2020-09-24_19:36:30.59395 ^M | ^M |......................................... | 58%
2020-09-24_19:36:30.59396 label: unnamed-chunk-26
2020-09-24_19:36:30.59396 ^M | ^M |.......................................... | 60%
2020-09-24_19:36:30.59397 ordinary text without R code
2020-09-24_19:36:30.59397
2020-09-24_19:36:30.59397 ^M | ^M |.......................................... | 61%
2020-09-24_19:36:30.59397 label: unnamed-chunk-27
2020-09-24_19:36:30.59398 ^M | ^M |........................................... | 62%
2020-09-24_19:36:30.59398 ordinary text without R code
2020-09-24_19:36:30.59398
2020-09-24_19:36:30.59398 ^M | ^M |............................................ | 63%
2020-09-24_19:36:30.59399 label: unnamed-chunk-28
2020-09-24_19:36:30.59399 ^M | ^M |............................................. | 64%
2020-09-24_19:36:30.59400 ordinary text without R code
2020-09-24_19:36:30.59400
2020-09-24_19:36:30.59400 ^M | ^M |.............................................. | 65%
2020-09-24_19:36:30.59401 label: unnamed-chunk-29
2020-09-24_19:36:30.59401 ^M | ^M |.............................................. | 66%
2020-09-24_19:36:30.59401 ordinary text without R code
2020-09-24_19:36:30.59401
2020-09-24_19:36:30.59402 ^M | ^M |............................................... | 67%
2020-09-24_19:36:30.59402 label: unnamed-chunk-30
2020-09-24_19:36:30.59402 ^M | ^M |................................................ | 69%
2020-09-24_19:36:30.59402 ordinary text without R code
2020-09-24_19:36:30.59403
2020-09-24_19:36:30.59403 ^M | ^M |................................................. | 70%
2020-09-24_19:36:30.59403 label: unnamed-chunk-31
2020-09-24_19:36:30.59403 ^M | ^M |.................................................. | 71%
2020-09-24_19:36:30.59404 ordinary text without R code
2020-09-24_19:36:30.59404
2020-09-24_19:36:30.59405 ^M | ^M |.................................................. | 72%
2020-09-24_19:36:30.59405 label: unnamed-chunk-32
2020-09-24_19:36:30.59405 ^M | ^M |................................................... | 73%
2020-09-24_19:36:30.59405 ordinary text without R code
2020-09-24_19:36:30.59406
2020-09-24_19:36:30.59406 ^M | ^M |.................................................... | 74%
2020-09-24_19:36:30.59406 label: unnamed-chunk-33
2020-09-24_19:36:30.59406 ^M | ^M |..................................................... | 75%
2020-09-24_19:36:30.59407 ordinary text without R code
2020-09-24_19:36:30.59407
2020-09-24_19:36:30.59407 ^M | ^M |..................................................... | 76%
2020-09-24_19:36:30.59407 label: unnamed-chunk-34
2020-09-24_19:36:30.59408 ^M | ^M |...................................................... | 78%
2020-09-24_19:36:30.59408 ordinary text without R code
2020-09-24_19:36:30.59409
2020-09-24_19:36:30.59409 ^M | ^M |....................................................... | 79%
2020-09-24_19:36:30.59409 label: unnamed-chunk-35
2020-09-24_19:36:30.59409 ^M | ^M |........................................................ | 80%
2020-09-24_19:36:30.59410 ordinary text without R code
2020-09-24_19:36:30.59410
2020-09-24_19:36:30.59410 ^M | ^M |......................................................... | 81%
2020-09-24_19:36:30.59410 label: unnamed-chunk-36
2020-09-24_19:36:30.59411 ^M | ^M |......................................................... | 82%
2020-09-24_19:36:30.59411 ordinary text without R code
2020-09-24_19:36:30.59411
2020-09-24_19:36:30.59412 ^M | ^M |.......................................................... | 83%
2020-09-24_19:36:30.59412 label: unnamed-chunk-37
2020-09-24_19:36:30.59413 ^M | ^M |........................................................... | 84%
2020-09-24_19:36:30.59413 ordinary text without R code
2020-09-24_19:36:30.59413
2020-09-24_19:36:30.59413 ^M | ^M |............................................................ | 85%
2020-09-24_19:36:30.59414 label: unnamed-chunk-38
2020-09-24_19:36:30.59414 ^M | ^M |............................................................. | 87%
2020-09-24_19:36:30.59414 ordinary text without R code
2020-09-24_19:36:30.59414
2020-09-24_19:36:30.59414 ^M | ^M |............................................................. | 88%
2020-09-24_19:36:30.59415 label: unnamed-chunk-39
2020-09-24_19:36:30.59415 ^M | ^M |.............................................................. | 89%
2020-09-24_19:36:30.59415 ordinary text without R code
2020-09-24_19:36:30.59415
2020-09-24_19:36:30.59416 ^M | ^M |............................................................... | 90%
2020-09-24_19:36:30.59416 label: unnamed-chunk-40
2020-09-24_19:36:30.59417 ^M | ^M |................................................................ | 91%
2020-09-24_19:36:30.59417 ordinary text without R code
2020-09-24_19:36:30.59417
2020-09-24_19:36:30.59417 ^M | ^M |................................................................ | 92%
2020-09-24_19:36:30.59418 label: unnamed-chunk-41
2020-09-24_19:36:30.59418 ^M | ^M |................................................................. | 93%
2020-09-24_19:36:30.59418 ordinary text without R code
2020-09-24_19:36:30.59418
2020-09-24_19:36:30.59419 ^M | ^M |.................................................................. | 94%
2020-09-24_19:36:30.59419 label: unnamed-chunk-42
2020-09-24_19:36:30.59419 ^M | ^M |................................................................... | 96%
2020-09-24_19:36:30.59420 ordinary text without R code
2020-09-24_19:36:30.59420
2020-09-24_19:36:30.59420 ^M | ^M |.................................................................... | 97%
2020-09-24_19:36:30.59421 label: unnamed-chunk-43
2020-09-24_19:36:30.59421 ^M | ^M |.................................................................... | 98%
2020-09-24_19:36:30.59421 ordinary text without R code
2020-09-24_19:36:30.59421
2020-09-24_19:36:30.59422 ^M | ^M |..................................................................... | 99%
2020-09-24_19:36:30.59422 label: unnamed-chunk-44 (with options)
2020-09-24_19:36:30.59422 List of 1
2020-09-24_19:36:30.59422 $ engine: chr "bash"
2020-09-24_19:36:30.59423
2020-09-24_19:36:30.59423 ^M | ^M |......................................................................| 100%
2020-09-24_19:36:30.59423 ordinary text without R code
2020-09-24_19:36:30.59424
2020-09-24_19:36:30.59424
2020-09-24_19:36:30.59424 output file: /usr/src/arvados/doc/sdk/R/README.md
2020-09-24_19:36:30.59424
2020-09-24_19:36:30.59425 [1] "/usr/src/arvados/doc/sdk/R/README.md"
2020-09-24_19:37:42.36398 Starting a Gradle Daemon, 1 incompatible and 2 stopped Daemons could not be reused, use --status for details
Updated by Peter Amstutz over 5 years ago
Please update the jenkins test image as well
Updated by Ward Vandewege over 5 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Resolved
Applied in changeset arvados|27992e62e71eb2ad4297c4eb3f4e787c4b000b90.
Updated by Ward Vandewege over 5 years ago
After rebuilding the compute image on Jenkins:
Actions