Cao, Donggang
commit
4b8fc51a57
123
README.md
123
README.md
|
|
@ -2,38 +2,28 @@
|
|||
|
||||
http://docklet.unias.org
|
||||
|
||||
## intro
|
||||
## Intro
|
||||
|
||||
Docklet is an operating system for mini-datacener. Its goal is to help
|
||||
multi-user share cluster resources effectively. Unlike the "application
|
||||
framework oriented" cluster manager such as Mesos and Yarn, Docklet is
|
||||
**user oriented**. In Docklet, every user has their own private
|
||||
**virtual cluster (vcluster)**, which consists of a number of virtual
|
||||
Linux container nodes distributed over the physical cluster. Every
|
||||
vcluster is separated from others and can be operated like a real
|
||||
physical cluster. Therefore, most applications, especially those
|
||||
requiring a cluster environment, can run in vcluster seamlessly.
|
||||
Docklet is a cloud operating system for mini-datacener. Its goal is to
|
||||
help multi-user share cluster resources effectively. In Docklet, every
|
||||
user has their own private **virtual cluster (vcluster)**, which
|
||||
consists of a number of virtual Linux container nodes distributed over
|
||||
the physical cluster. Each vcluster is separated from others and can be
|
||||
operated like a real physical cluster. Therefore, most applications,
|
||||
especially those requiring a cluster environment, can run in vcluster
|
||||
seamlessly.
|
||||
|
||||
Docklet provides a base image for creating virtual nodes. This image has
|
||||
pre-installed a lot of mainstream development tools and frameworks,
|
||||
including gcc/g++, openjdk, python3, R, MPI, scala, ruby, php, node.js,
|
||||
texlive, mpich2, spark,
|
||||
scipy/numpy/matplotlib/pandas/sympy/scikit-learn, jupyter notebook, etc.
|
||||
Users can get a ready vcluster with just one click within 1 second.
|
||||
Users manage and use their vcluster all through web. The only client
|
||||
tool needed is a modern web browser supporting HTML5, like Safari,
|
||||
Firefox, or Chrome. The integrated *jupyter notebook* provides a web
|
||||
**Workspace**. In the Workspace, users can code, debug, test,
|
||||
and runn their programs, even visualize the outputs online.
|
||||
Therefore, it is ideal for data analysis and processing.
|
||||
|
||||
The users are free to install their specific software in their vcluster.
|
||||
Docklet supports operating through **web terminal**. Users can do their
|
||||
work as an administrator working on a console. The base image system is
|
||||
ubuntu. The recommended way of installing new software is by
|
||||
**apt-get**.
|
||||
|
||||
The users manage and use their vcluster all through web. The only client
|
||||
tool needed is a modern web browser, like safari, firefox, chrome. The
|
||||
integrated *jupyter notebook* provides a web workspace. By visiting the
|
||||
workspace, users can do coding, debugging and testing of their programs
|
||||
online. The **python scipy** series of tools can even display graphical
|
||||
pictures in the browser. Therefore, it is ideal for data analysis and
|
||||
processing.
|
||||
Docklet creates virtual nodes from a base image. Admins can
|
||||
pre-install development tools and frameworks according to their
|
||||
interests. The users are also free to install their specific software
|
||||
in their vcluster.
|
||||
|
||||
Docklet only need **one** public IP address. The vclusters are
|
||||
configured to use private IP address range, e.g., 172.16.0.0/16,
|
||||
|
|
@ -47,54 +37,34 @@ The Docklet system runtime consists of four components:
|
|||
- docklet master
|
||||
- docklet worker
|
||||
|
||||
## install
|
||||
## Install
|
||||
|
||||
Currently the docklet runtime is recommend to run in Unbuntu 15.10+.
|
||||
Currently the Docklet system is recommend to run in Unbuntu 15.10+.
|
||||
|
||||
Ensure that python3.5 is the default python3 version.
|
||||
|
||||
Unpack the docklet tarball to a directory ( /root/docklet as an
|
||||
example), will get
|
||||
Clone Docklet from github
|
||||
|
||||
```
|
||||
readme.md
|
||||
prepare.sh
|
||||
conf/
|
||||
container.conf
|
||||
docklet.conf.template
|
||||
lxc-script/
|
||||
bin/
|
||||
docklet-master
|
||||
docklet-worker
|
||||
src/
|
||||
httprest.py
|
||||
worker.py
|
||||
...
|
||||
web/
|
||||
web.py
|
||||
doc/
|
||||
tools/
|
||||
update-basefs.sh
|
||||
start_jupyter.sh
|
||||
git clone https://github.com/unias/docklet.git
|
||||
```
|
||||
|
||||
If it is the first time install, users should run **prepare.sh** to
|
||||
install necessary packages automatically. Note it may need to run this
|
||||
script several times to successfully install all the needed packages.
|
||||
Run **prepare.sh** from console to install depended packages and
|
||||
generate necessary configurations.
|
||||
|
||||
A *root* users will be created for managing the system. The password is
|
||||
recorded in `FS_PREFIX/local/generated_password.txt` .
|
||||
A *root* users will be created for managing the Docklet system. The
|
||||
password is recorded in `FS_PREFIX/local/generated_password.txt` .
|
||||
|
||||
## config ##
|
||||
## Config ##
|
||||
|
||||
The main configuration file of docklet is conf/docklet.conf. Most
|
||||
default setting works for a single host environment.
|
||||
|
||||
First copy docklet.conf.template to get docklet.conf.
|
||||
|
||||
The following settings should be taken care of:
|
||||
Pay attention to the following settings:
|
||||
|
||||
- NETWORK_DEVICE : the network device to use.
|
||||
- NETWORK_DEVICE : the network interface to use.
|
||||
- ETCD : the etcd server address. For distributed muli hosts
|
||||
environment, it should be one of the ETCD public server address.
|
||||
For single host environment, the default value should be OK.
|
||||
|
|
@ -111,7 +81,7 @@ The following settings should be taken care of:
|
|||
by visiting this address. If the system is behind a firewall, then
|
||||
a reverse proxy should be setup.
|
||||
|
||||
## start ##
|
||||
## Start ##
|
||||
|
||||
### distributed file system ###
|
||||
|
||||
|
|
@ -123,8 +93,7 @@ Lets presume the file system server export filesystem as nfs
|
|||
In each physical host to run docklet, mount **fileserver:/pub** to
|
||||
**FS_PEFIX/global** .
|
||||
|
||||
For single host environment, it need not to configure distributed
|
||||
file system.
|
||||
For single host environment, nothing to do.
|
||||
|
||||
### etcd ###
|
||||
|
||||
|
|
@ -133,7 +102,7 @@ Ubuntu releases have included **etcd** in the repository, just `apt-get
|
|||
install etcd`, and it need not to start etcd manually. For others, you
|
||||
should install etcd manually.
|
||||
|
||||
For multi hosts distributed environment, start
|
||||
For multi hosts distributed environment, **must** start
|
||||
**dep/etcd-multi-nodes.sh** in each etcd server hosts. This scripts
|
||||
requires users providing the etcd server address as parameters.
|
||||
|
||||
|
|
@ -146,27 +115,19 @@ address, e.g., 172.16.0.1. This server will be the master.
|
|||
If it is the first time you start docklet, run `bin/docklet-master init`
|
||||
to init and start docklet master. Otherwise, run `bin/docklet-master start`,
|
||||
which will start master in recovery mode in background using
|
||||
conf/docklet.conf. It means docklet will recover workspaces existed.
|
||||
|
||||
This script in fact will start three daemons: the docklet master of
|
||||
httprest.py, the configurable-http-proxy and the docklet web of web.py.
|
||||
conf/docklet.conf.
|
||||
|
||||
You can check the daemon status by running `bin/docklet-master status`
|
||||
|
||||
If the master failed to start, you could try `bin/docklet-master init`
|
||||
to initialize the whole system.
|
||||
|
||||
More usages can be found by typing `bin/docklet-master`
|
||||
|
||||
The master logs are in **FS_PREFIX/local/log/docklet-master.log** and
|
||||
**docklet-web.log**.
|
||||
|
||||
### worker ###
|
||||
|
||||
Worker needs a basefs image to boot container.
|
||||
Worker needs a basefs image to create containers.
|
||||
|
||||
You can create such an image with `lxc-create -n test -t download`,
|
||||
and then copy the rootfs to **FS_PREFIX/local**, and renamed `rootfs`
|
||||
then copy the rootfs to **FS_PREFIX/local**, and renamed `rootfs`
|
||||
to `basefs`.
|
||||
|
||||
Note the `jupyerhub` package must be installed for this image. And the
|
||||
|
|
@ -179,21 +140,17 @@ Run `bin/docklet-worker start`, will start worker in background.
|
|||
|
||||
You can check the daemon status by running `bin/docklet-worker status`
|
||||
|
||||
More usages can be found by typing `bin/docklet-worker`
|
||||
|
||||
The log is in **FS_PREFIX/local/log/docklet-worker.log**
|
||||
|
||||
Currently, the worker must be run after the master has been started.
|
||||
|
||||
## usage ##
|
||||
## Usage ##
|
||||
|
||||
Open a browser, visiting the address specified by PORTAL_URL ,
|
||||
e.g., ` http://docklet.info/ `
|
||||
|
||||
If the system is just deployed in single host for testing purpose,
|
||||
then the PORTAL_URL defaults to `http://MASTER_IP:PROXY_PORT`,
|
||||
e.g., `http://localhost:8000`.
|
||||
|
||||
That is it.
|
||||
|
||||
## system admin ##
|
||||
# Contribute #
|
||||
|
||||
Contributions are welcome. Please check [devguide](doc/devguide/devguide.md)
|
||||
|
|
|
|||
Loading…
Reference in New Issue