Support #5461: following manual install instructions on laptop - Arvados

Actions

Copy link

Support #5461

closed

following manual install instructions on laptop

Added by Albert Vilella about 11 years ago. Updated about 6 years ago.

Status:

Closed

Priority:

Normal

Assigned To:

Category:

Target version:

Due date:

Story points:

Description

Trying to install arvados locally on an Ubuntu laptop. Details below:

http://doc.arvados.org/install/install-manual-prerequisites.html (click Next bottom right)

sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql sudo wget zlib1g-dev

_{$ cd $HOME # (or wherever you want to install)}$ git clone https://github.com/curoverse/arvados.git

_{$ cd arvados/services/api}/arvados/services/api$ bundle install

My application.yml file looks something like this:

```
development: # Mandatory site secrets. See application.default.yml for more info.
secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
uuid_prefix: foobar
workbench_address: https://localhost:3031

production: # Mandatory site secrets. See application.default.yml for more info.
secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
uuid_prefix: barfoo
workbench_address: https://localhost:9899

workbench_address: https://workbench.foobar.arvadosapi.com
```

I created a keepstore.key file that is 128 characters long.

I followed the instructions here: http://doc.arvados.org/install/install-keepstore.html

Results:

```
[hpenvy15 ~/arvados/services/api] $ keepstore -permission-key-file="/home/avilella/arvados/services/api/keepstore.key"
2015/03/13 11:56:49 Keep started: pid 4460
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.ZeLkn1xGDR/keep
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.XJotjPWD5P/keep
2015/03/13 11:56:49 Error setting up arvados client Missing required environment variable ARVADOS_API_HOST
```

I went to https://console.developers.google.com and created a project and got the following (dummy values shown):

```
Project ID: smthg-standard-12345 Project Number: 123456789012
CLIENT SECRET 1234567890QWERTYadsfghxy
API KEY AIzaSyDCmGeVk5ZXdMbfTLNzOovnzsmSJ84RquE
```

I then continued with the database setup. This is what my database.yml file looks like:

```
#development:

adapter: postgresql
encoding: utf8
database: arvados_development
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost #
#test:
adapter: postgresql
encoding: utf8
database: arvados_test
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost

production:
adapter: postgresql
encoding: utf8
database: arvados_production
username: arvados
password: 12345678900qwertyasdfgxy
host: localhost
```

I then tried to initialize the production database like this:

```
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
psql:/home/avilella/arvados/services/api/db/structure.sql:22: ERROR: must be owner of extension plpgsql
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Uuid not a valid Arvados uuid 'barfoo-tpzed-000000000000000'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:56:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `block in save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:208:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:41:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `block in create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `block in scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/scoping.rb:98:in `with_scope'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:126:in `first_or_create!'
/home/avilella/arvados/services/api/lib/current_api_client.rb:64:in `block in system_user'
/home/avilella/arvados/services/api/lib/current_api_client.rb:222:in `check_cache'
/home/avilella/arvados/services/api/lib/current_api_client.rb:57:in `system_user'
/home/avilella/arvados/services/api/app/models/database_seeds.rb:4:in `install'
/home/avilella/arvados/services/api/db/seeds.rb:5:in `<top (required)>'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `block in load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/railties-3.2.17/lib/rails/engine.rb:525:in `load_seed'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/railties/databases.rake:347:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)
```

Any ideas?

Actions

Copy link

Updated by Albert Vilella about 11 years ago

Same content as above but properly quoted with pre tag:

Trying to install arvados locally on an Ubuntu laptop. Details below:

http://doc.arvados.org/install/install-manual-prerequisites.html (click Next bottom right)

sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql sudo wget zlib1g-dev

_{$ cd $HOME # (or wherever you want to install)}$ git clone https://github.com/curoverse/arvados.git

_{$ cd arvados/services/api}/arvados/services/api$ bundle install

My application.yml file looks something like this:

development:
  # Mandatory site secrets. See application.default.yml for more info.
  secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  uuid_prefix: foobar
  workbench_address: https://localhost:3031

production:
  # Mandatory site secrets. See application.default.yml for more info.
  secret_token: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
  blob_signing_key: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
  uuid_prefix: barfoo
  workbench_address: https://localhost:9899
#  workbench_address: https://workbench.foobar.arvadosapi.com

I created a keepstore.key file that is 128 characters long.

I followed the instructions here: http://doc.arvados.org/install/install-keepstore.html

Results:

[hpenvy15 ~/arvados/services/api] $ keepstore -permission-key-file="/home/avilella/arvados/services/api/keepstore.key" 
2015/03/13 11:56:49 Keep started: pid 4460
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.ZeLkn1xGDR/keep
2015/03/13 11:56:49 adding Keep volume: /tmp/tmp.XJotjPWD5P/keep
2015/03/13 11:56:49 Error setting up arvados client Missing required environment variable ARVADOS_API_HOST

I went to https://console.developers.google.com and created a project and got the following (dummy values shown):

Project ID: smthg-standard-12345 Project Number: 123456789012
CLIENT SECRET    1234567890QWERTYadsfghxy
API KEY    AIzaSyDCmGeVk5ZXdMbfTLNzOovnzsmSJ84RquE

I then continued with the database setup. This is what my database.yml file looks like:

#development:                                                                                                                                                                                                        
#  adapter: postgresql
#  encoding: utf8
#  database: arvados_development
#  username: arvados
#  password: 12345678900qwertyasdfgxy
#  host: localhost
#
#test:
#  adapter: postgresql
#  encoding: utf8
#  database: arvados_test
#  username: arvados
#  password: 12345678900qwertyasdfgxy
#  host: localhost

production:
  adapter: postgresql
  encoding: utf8
  database: arvados_production
  username: arvados
  password: 12345678900qwertyasdfgxy
  host: localhost

I then tried to initialize the production database like this:

[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
[hpenvy15 ~/arvados/services/api] $ RAILS_ENV=production bundle exec rake db:setup
psql:/home/avilella/arvados/services/api/db/structure.sql:22: ERROR:  must be owner of extension plpgsql
rake aborted!
ActiveRecord::RecordInvalid: Validation failed: Uuid not a valid Arvados uuid 'barfoo-tpzed-000000000000000'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:56:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/attribute_methods/dirty.rb:33:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `block in save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:313:in `block in with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:208:in `transaction'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:311:in `with_transaction_returning_status'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/transactions.rb:264:in `save!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/validations.rb:41:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `block in create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `block in scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/scoping.rb:98:in `with_scope'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:241:in `scoping'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:91:in `create!'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/relation.rb:126:in `first_or_create!'
/home/avilella/arvados/services/api/lib/current_api_client.rb:64:in `block in system_user'
/home/avilella/arvados/services/api/lib/current_api_client.rb:222:in `check_cache'
/home/avilella/arvados/services/api/lib/current_api_client.rb:57:in `system_user'
/home/avilella/arvados/services/api/app/models/database_seeds.rb:4:in `install'
/home/avilella/arvados/services/api/db/seeds.rb:5:in `<top (required)>'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `block in load'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:236:in `load_dependency'
/var/lib/gems/2.1.0/gems/activesupport-3.2.17/lib/active_support/dependencies.rb:245:in `load'
/var/lib/gems/2.1.0/gems/railties-3.2.17/lib/rails/engine.rb:525:in `load_seed'
/var/lib/gems/2.1.0/gems/activerecord-3.2.17/lib/active_record/railties/databases.rake:347:in `block (2 levels) in <top (required)>'
Tasks: TOP => db:setup => db:seed
(See full trace by running task with --trace)

Actions

Copy link

Updated by Peter Amstutz about 11 years ago

Hi avilella, your site prefix is too long. It needs to be 5 characters and "barfoo" is 6.

Actions

Copy link

Updated by Albert Vilella about 11 years ago

See below script as it is now

#/bin/bash

Install dependencies
sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
1. 1

Install Ruby and Bundler
pushd $HOME
mkdir -p src
pushd src
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
tar xzf ruby-2.1.5.tar.gz
cd ruby-2.1.5
./configure
make
sudo make install
sudo gem install bundler
popd
rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
1. 2

Install service API
pushd arvados/services/api
bundle install
cp i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
sudo mkdir -p /var/lib/arvados/git
sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
sed -i "s/bogus/gamma/g" config/application.yml
sed -n -l 1 's/$^.*blob_signing_key: $$.*$/\2/p' config/application.yml | uniq > keepstore.key
keepstore --permission-key-file=keepstore.key &
prefix=`arv --format=uuid user current | cut -d -f1`
"Site prefix is '$prefix'"
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk"
}
EOF

3
echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
sudo /usr/bin/apt-get update
sudo /usr/bin/apt-get install keepstore

ruby -e 'puts rand(2**128).to_s(36)'

TODO: hard coded here: d47b6bmmpmm5hceqwsda9at5x
sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
cp -i config/database.yml.sample config/database.yml
sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
RAILS_ENV=production bundle exec rake db:setup
sudo su postgres
createdb arvados_production -E UTF8 -O arvados
createdb arvados_development -E UTF8 -O arvados
exit
RAILS_ENV=production bundle exec rake db:structure:load
RAILS_ENV=production bundle exec rake db:seed
RAILS_ENV=development bundle exec rake db:structure:load
RAILS_ENV=development bundle exec rake db:seed

omniauth
cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
sed -i "s/^$APP_SECRET = $$.*$$/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
bundle exec rails server --port=3030
popd

Install Phusion wheezy
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
sudo apt-get install apt-transport-https ca-certificates
echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
sudo apt-get update
sudo apt-get install nginx-extras passenger

Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
sudo service nginx restart
echo "server {" >> /etc/nginx/sites-available/foo
echo " listen 80;" >> /etc/nginx/sites-available/foo
echo " root `pwd`/public;" >> /etc/nginx/sites-available/foo
echo " index index.html index.htm;" >> /etc/nginx/sites-available/foo
echo "}" >> /etc/nginx/sites-available/foo

ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/
popd

Install workbench
pushd arvados/apps/workbench
bundle install
sudo apt-get install \
bison build-essential gettext libcurl3 libcurl3-gnutls \
libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz

apt-get install libsqlite3-dev
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
pushd ~/arvados/services/api
sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
popd
arvados_login_base: https://prefix_uuid.your.domain/login
arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1
popd

Install SSO
pushd $HOME
git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
pushd sso-devise-omniauth-provider
bundle install
RAILS_ENV=production bundle exec rake db:create
RAILS_ENV=production bundle exec rake db:migrate
cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
sed -i "s/^$.*secret_token = $.*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb

review http://doc.arvados.org/install/install-sso.html
https://console.developers.google.com/project/foo-arvados/apiui/credential#
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/environments/production.rb config/environments/development.rb
openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer
RAILS_ENV=production bundle exec rails console
popd
popd

keep
unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut d -f1` && echo "Site prefix is '$prefix'"
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk"
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" {
"service_host":"workbench.gamma.foo.com",
"service_port":443,
"service_ssl_flag":true,
"service_type":"proxy"
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" {"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
"script_version":"master",
"script":"hash",
"repository":"arvados"}
EOF

Install Perl SDK
sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

Install Python SDK
sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

TODO: extras at bottom of http://doc.arvados.org/install/install-manual-prerequisites.html

Actions

Copy link

Updated by Albert Vilella about 11 years ago

Sorry, again now with pre tags:


#/bin/bash
# Install dependencies
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
### 1

## Install Ruby and Bundler
pushd $HOME
mkdir -p src
pushd src
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
tar xzf ruby-2.1.5.tar.gz
cd ruby-2.1.5
./configure
make
sudo make install
sudo gem install bundler
popd
rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
### 2

# Install service API
pushd arvados/services/api
bundle install
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
sudo mkdir -p /var/lib/arvados/git
sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
sed -i "s/bogus/gamma/g" config/application.yml
sed -n -l 1 's/\(^.*blob_signing_key: \)\(.*\)/\2/p' config/application.yml | uniq > keepstore.key
keepstore --permission-key-file=keepstore.key &
prefix=`arv --format=uuid user current | cut -d- -f1`
"Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
"service_host":"keep0.gamma.foo.com",
"service_port":25107,
"service_ssl_flag":false,
"service_type":"disk" 
}
EOF

### 3
echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
  sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
  sudo /usr/bin/apt-get update
  sudo /usr/bin/apt-get install keepstore

ruby -e 'puts rand(2**128).to_s(36)'
# TODO: hard coded here: d47b6bmmpmm5hceqwsda9at5x
sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
cp -i config/database.yml.sample config/database.yml
sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
RAILS_ENV=production bundle exec rake db:setup
sudo su postgres
createdb arvados_production -E UTF8 -O arvados
createdb arvados_development -E UTF8 -O arvados
exit
RAILS_ENV=production bundle exec rake db:structure:load 
RAILS_ENV=production bundle exec rake db:seed
RAILS_ENV=development bundle exec rake db:structure:load 
RAILS_ENV=development bundle exec rake db:seed

### omniauth
cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
sed -i "s/^\(APP_SECRET = \)\(.*$\)/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
bundle exec rails server --port=3030
popd

## Install Phusion wheezy
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
sudo apt-get install apt-transport-https ca-certificates
echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
sudo apt-get update
sudo apt-get install nginx-extras passenger
# Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
sudo service nginx restart
echo "server {" >> /etc/nginx/sites-available/foo
echo "        listen   80;" >> /etc/nginx/sites-available/foo
echo "        root `pwd`/public;" >> /etc/nginx/sites-available/foo
echo "        index index.html index.htm;" >> /etc/nginx/sites-available/foo
echo "}" >> /etc/nginx/sites-available/foo

ln -s /etc/nginx/sites-available/foo /etc/nginx/sites-enabled/
popd

## Install workbench
pushd arvados/apps/workbench
bundle install
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz

apt-get install libsqlite3-dev
cp -i config/environments/production.rb.example config/environments/production.rb
cp -i config/application.yml.example config/application.yml
pushd ~/arvados/services/api
sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
popd
arvados_login_base: https://prefix_uuid.your.domain/login
arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1
popd

## Install SSO
pushd $HOME
git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
pushd sso-devise-omniauth-provider
bundle install
RAILS_ENV=production bundle exec rake db:create
RAILS_ENV=production bundle exec rake db:migrate
cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
sed -i "s/^\(.*secret_token = \).*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb
# review http://doc.arvados.org/install/install-sso.html
# https://console.developers.google.com/project/foo-arvados/apiui/credential#
cp -i config/environments/production.rb.example config/environments/production.rb
# cp -i config/environments/production.rb config/environments/development.rb
openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer
RAILS_ENV=production bundle exec rails console
popd
popd

### keep
unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut -d- -f1` && echo "Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"keep0.gamma.foo.com",
 "service_port":25107,
 "service_ssl_flag":false,
 "service_type":"disk" 
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"workbench.gamma.foo.com",
 "service_port":443,
 "service_ssl_flag":true,
 "service_type":"proxy" 
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" 
{"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
 "script_version":"master",
 "script":"hash",
 "repository":"arvados"}
EOF

# Install Perl SDK
sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

# Install Python SDK
sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

# TODO: extras at bottom of http://doc.arvados.org/install/install-manual-prerequisites.html

Actions

Copy link

Updated by Davide Fiorentino lo Regio about 11 years ago

#/bin/bash
# Install dependencies
sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 postgresql git wget zlib1g-dev
### 1

## Install Ruby and Bundler
pushd $HOME
    mkdir -p src
    pushd src
        wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.5.tar.gz
        tar xzf ruby-2.1.5.tar.gz
        cd ruby-2.1.5
        ./configure
        make
        sudo make install
        sudo gem install bundler
    popd
    rm -rf src/ruby-2.1.5
popd
git clone https://github.com/curoverse/arvados.git
### 2

# Install service API
pushd arvados/services/api
    bundle install
    cp -i config/environments/production.rb.example config/environments/production.rb
    cp -i config/application.yml.example config/application.yml
    sudo mkdir -p /var/lib/arvados/git
    sudo git clone --bare ../../.git /var/lib/arvados/git/arvados.git
    sed -i "s/secret_token.*$/secret_token: `rake secret`/g" config/application.yml
    sed -i "s/blob_signing_key.*$/blob_signing_key: `rake secret`/g" config/application.yml
    sed -i "s/https:\/\/workbench.*$/https:\/\/workbench.gamma.foo.com/g" config/application.yml
    sed -i "s/bogus/gamma/g" config/application.yml
    sed -n -l 1 's/\(^.*blob_signing_key: \)\(.*\)/\2/p' config/application.yml | uniq > keepstore.key

    echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
     sudo /usr/bin/apt-key adv --keyserver pool.sks-keyservers.net --recv 1078ECD7
     sudo /usr/bin/apt-get update
     sudo /usr/bin/apt-get install keepstore
    keepstore --permission-key-file=keepstore.key &

    prefix=`arv --format=uuid user current | cut -d- -f1`
    "Site prefix is '$prefix'" 
    read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
        {
         "service_host":"keep0.gamma.foo.com",
         "service_port":25107,
         "service_ssl_flag":false,
         "service_type":"disk" 
        }
    EOF

### 3
    ruby -e 'puts rand(2**128).to_s(36)'
    # d47b6bmmpmm5hceqwsda9at5x
    sudo -u postgres createuser --createdb --encrypted --pwprompt -R -S arvados
    cp -i config/database.yml.sample config/database.yml
    sed -i "s/xxxxxxxx/d47b6bmmpmm5hceqwsda9at5x/g" config/database.yml
    RAILS_ENV=production bundle exec rake db:setup
    sudo su postgres
    createdb arvados_production -E UTF8 -O arvados
    createdb arvados_development -E UTF8 -O arvados
    exit
    RAILS_ENV=production bundle exec rake db:structure:load    
    RAILS_ENV=production bundle exec rake db:seed
    RAILS_ENV=development bundle exec rake db:structure:load    
    RAILS_ENV=development bundle exec rake db:seed

    cp -i config/initializers/omniauth.rb.example config/initializers/omniauth.rb
    sed -i "s/^\(APP_SECRET = \)\(.*$\)/\1'`ruby -e 'puts rand(2**512).to_s(36)'`'/g" config/initializers/omniauth.rb
    bundle exec rails server --port=3030
popd

## Install Phusion
    sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
    sudo apt-get install apt-transport-https ca-certificates
    echo "deb https://oss-binaries.phusionpassenger.com/apt/passenger wheezy main" | sudo tee /etc/apt/sources.list.d/passenger.list
    sudo apt-get update
    sudo apt-get install nginx-extras passenger
    # Edit /etc/nginx/nginx.conf and uncomment passenger_root and passenger_ruby.
popd

## Install workbench
pushd arvados/apps/workbench
    bundle install
    sudo apt-get install \
    bison build-essential gettext libcurl3 libcurl3-gnutls \
    libcurl4-openssl-dev libpcre3-dev libpq-dev libreadline-dev \
    libssl-dev libxslt1.1 sudo wget zlib1g-dev graphviz \
    libsqlite3-dev

    cp -i config/environments/production.rb.example config/environments/production.rb
    cp -i config/application.yml.example config/application.yml
    pushd ~/arvados/services/api
    sed -i "s/secret_token.*$/secret_token: `bundle exec rake secret`/g" ~/arvados/apps/workbench/config/application.yml
    popd

    # set these
    arvados_login_base: https://prefix_uuid.your.domain/login
    arvados_v1_base: https://prefix_uuid.your.domain/arvados/v1

popd

## Install SSO
pushd $HOME
    git clone https://github.com/curoverse/sso-devise-omniauth-provider.git
    pushd sso-devise-omniauth-provider
        bundle install
        RAILS_ENV=production bundle exec rake db:create
        RAILS_ENV=production bundle exec rake db:migrate
        RAILS_ENV=development bundle exec rake db:create
        RAILS_ENV=development bundle exec rake db:migrate
        cp -i config/initializers/secret_token.rb.example config/initializers/secret_token.rb
        sed -i "s/^\(.*secret_token = \).*/\1'`bundle exec rake secret`'/g" config/initializers/secret_token.rb
# review http://doc.arvados.org/install/install-sso.html
# https://console.developers.google.com/project/foo-arvados/apiui/credential#
        cp -i config/environments/production.rb.example config/environments/production.rb
        cp -i config/environments/production.rb config/environments/development.rb
        openssl req -nodes -newkey rsa:2048 -keyout foo.key -x509 -days 3650 -out foo.cer

        # RAILS_ENV=production bundle exec rails console
        RAILS_ENV=production bundle exec rails console
    popd
popd

unset ARVADOS_API_HOST_INSECURE
export ARVADOS_API_HOST=localhost:3030
export ARVADOS_API_HOST_INSECURE=1
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
prefix=`arv --format=uuid user current | cut -d- -f1` && echo "Site prefix is '$prefix'" 
read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"keep0.gamma.foo.com",
 "service_port":25107,
 "service_ssl_flag":false,
 "service_type":"disk" 
}
EOF

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' keepservice <<EOF; arv keep_service create --keep-service "$keepservice" 
{
 "service_host":"workbench.gamma.foo.com",
 "service_port":443,
 "service_ssl_flag":true,
 "service_type":"proxy" 
}
EOF

echo "deb http://apt.arvados.org/ wheezy main" | sudo tee /etc/apt/sources.list.d/apt.arvados.org.list
sudo /usr/bin/apt-get install keepproxy

export ARVADOS_API_HOST=workbench.gamma.foo.com
export ARVADOS_API_TOKEN=3baqay8azvan45u7gpduetzd8g1dml8n17sp7znn4ij3dblq39
export ARVADOS_API_HOST=workbench.gamma.foo.com:3030
export ARVADOS_API_HOST_INSECURE=1

read -rd $'\000' newjob <<EOF; arv job create --job "$newjob" 
{"script_parameters":{"input":"a7f932428b30582f54ea26084813eb7b3e8f3251"},
 "script_version":"master",
 "script":"hash",
 "repository":"arvados"}
EOF

sudo apt-get install libjson-perl libio-socket-ssl-perl libwww-perl libipc-system-simple-perl
cd ~/arvados/sdk/perl
perl Makefile.PL
sudo make install
sudo apt-get install libnet-ssleay-perl
sudo apt-get install libcrypt-ssleay-perl
perl -MArvados -e ''

perl <<'EOF'
use Arvados;
my $arv = Arvados->new('apiVersion' => 'v1');
my $me = $arv->{'users'}->{'current'}->execute;
print ("arvados.v1.users.current.full_name = '", $me->{'full_name'}, "'\n");
EOF

sudo apt-get install python-pip python-dev libattr1-dev libfuse-dev pkg-config python-yaml
sudo pip install arvados-python-client
pip install --upgrade pyvcf
adduser crunch

Actions

Copy link

Updated by Peter Amstutz about 6 years ago

Status changed from New to Closed

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Arvados

Custom queries

Support #5461

following manual install instructions on laptop

Updated by Albert Vilella about 11 years ago

Updated by Peter Amstutz about 11 years ago

Updated by Albert Vilella about 11 years ago

Updated by Albert Vilella about 11 years ago

Updated by Davide Fiorentino lo Regio about 11 years ago

Updated by Peter Amstutz about 6 years ago