Merge remote-tracking branch 'upstream/experiment'

This commit is contained in:
tangshuangpku@hotmail.com 2016-06-08 12:20:04 +08:00
commit 7c7411eff7
24 changed files with 592 additions and 13 deletions

View File

@ -0,0 +1,173 @@
<!DOCTYPE html>
<html>
<head>
<title>Title</title>
<meta charset="utf-8">
<style>
@import url(https://fonts.googleapis.com/css?family=Yanone+Kaffeesatz);
@import url(https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic);
@import url(https://fonts.googleapis.com/css?family=Ubuntu+Mono:400,700,400italic);
body { font-family: 'Droid Serif'; }
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: normal;
}
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
a { text-decoration: none; }
div#my_container a {
text-decoration: none;
}
p.my_class a {
text-decoration: none;
}
ul.my_list a {
text-decoration: none;
}
</style>
</head>
<body>
<textarea id="source">
class: center, middle
# Big File Upload for Jupyter Notebook
Author: Wentao Zhang
Source Code: https://github.com/nsknojj/notebook/tree/big-upload
---
## What is Jupyter Notebook
[Jupyter](http://jupyter.org/) Notebook is a web-based notebook environment for interactive computing.
It satisfies the demands of every user for processing data.
```python
import numpy, matplotlib, scipy, sklearn, ……
```
## Why need big file upload
While big data are accessible for a lot of users nowadays, Notebook doesn't support big file upload with web interface. It only allows uploading files up to 25MB.
Big Upload module is designed to solve this problem and keep this upload function compatible with Notebook.
---
## Usage - Setup Big Upload Module
1. Make sure you have installed Jupyter.
If not, you can find the installation documentation for the
[Jupyter platform, on ReadTheDocs](http://jupyter.readthedocs.org/en/latest/install.html),
or you can install it with pip3:
```python
$ sudo pip3 install jupyter
```
If you want to install it with python2, use "pip" instead of "pip3". This way is also effective in the following steps, e.g. replacing "python" with "python3".
2. Download the Big Upload Module from its [Github Link](https://github.com/nsknojj/notebook/tree/big-upload) and put it in a position you like, then change to this directory in your cmd.
3. Run setup command:
```python
$ sudo python3 setup.py install
```
4. Run Notebook:
```python
$ jupyter notebook
```
---
## Usage - Setup Big Upload Module
It will open your default browser.
Now you can try to upload a big file in your notebook.
<img src="images/updemo.png" alt="Drawing" style="width: 800px; height: 290px"/>
---
## Design - How Big File Upload Works
### Prototype: [jQuery-File-Upload](https://github.com/blueimp/jQuery-File-Upload)
It has provided a jquery front end in which **chunked file upload** has been implemented, and a php back end to handle chunked file.
It provides a series of api, such as **stop** and **abort**. You can define your own callback functions for **add**, **progress**, etc.
Usage:
```javascript
$("big_upload").fileupload({ // big_upload is a <form> in webpage
add: function(e, data) {
// do something
},
done: function (e, data) {
notebook_list.session_list.load_sessions();
}
});
```
All I need to do is transplanting the front end, and building a back end handler with tornado framework based on the given php server.
---
## Design - How Big File Upload Works
### The Front End & Back End of Jupyter Notebook
* Notebook's back end is built with tornado framework.
You can see all the url handlers in *notebook/services*. Handlers in *notebook/services/contents* are related to uploading, downloading, saving and other functions we don't concern.
* What I call front end are those webpages, scripts and style sheets.
Webpage templates are in *notebook/templates*. We only concern "tree.html", which shows the tree structure of the opened file system.
Scripts relevant to my work are in *notebook/statics/tree/js*. They manage the file list and setup the elements in the tree webpage.
---
### Old Upload Process
```c
notebook
|----statics/tree/js/main.js # Setup upload button, upload it to api url
|----services/contents/handlers.py
| # Handle requests from url: api/contents/($file_path)
|----templates/tree.html
`----tree/handlers.py # Handle main url, render the tree template.
```
### Big Upload Process
```c
notebook
|----statics
| |----tree/js/main.js*
| | # Setup big-upload button with scripts in [bigupload/js]
| `----[bigupload/js] # Now upload the chunked file to a new url
|
|----services
| |----[bigupload/handlers.py] # Handle requests from a new url
|
`----templates/tree.html* # Add BigUpload button
```
The suffix star in the file name means this file has been modified. The square brackets means this file is new.
---
## Design - How it Works
### Divide Files to Chunks
* Front end uses [Blob API](https://developer.mozilla.org/en-US/docs/Web/API/Blob) to divide the file to several chunks, then upload it in a form using POST method.
This form contains the content range of this chunk, e.g. 0-99999 or 100000-199999, and the content size.
* Back end just handles the files according to file path, content range, content size and the payload.
It will find the correct file to append the payload to. The file's name should be same, and its current size should be equal to the start of content range.
---
## Experiment
### Bitrate
About 10MB/s ~ 20MB/s, on local host.
### Multifile Upload
Test passed, and the uploaded files had no difference.
### 5GB Big File Upload
Test passed, and the uploaded files had no difference.
### Cancel the Upload
Test passed, and the partially uploaded files were removed.
</textarea>
<script src="https://gnab.github.io/remark/downloads/remark-latest.min.js">
</script>
<script>
var slideshow = remark.create();
</script>
</body>
</html>

View File

@ -0,0 +1,74 @@
# Big File Upload for Jupyter Notebook
Author: [Wentao Zhang](zwt@pku.edu.cn)
Source Code: https://github.com/nsknojj/notebook/tree/big-upload
[Jupyter](http://jupyter.org/) Notebook is a web-based notebook environment for interactive computing. It satisfies the demands of every user for processing data. However, while big data are accessible for a lot of users nowadays, Notebook doesn't support big file upload with web interface. It only allows uploading files up to 25MB. Big Upload Module is designed to solve this problem and keep this upload function compatible with Notebook.
## Usage - Setup Big Upload Module
Make sure you have installed Jupyter.
If not, you can find the installation documentation for the
[Jupyter platform, on ReadTheDocs](http://jupyter.readthedocs.org/en/latest/install.html),
or you can install it with pip3:
```(python)
$ sudo pip3 install jupyter
```
If you want to install it with python2, use "pip" instead of "pip3". This way is also effective in the following steps, e.g. replacing "python" with "python3".
If you have not installed pip3, you can run *apt-get install pip3* or install it by [official instruction](https://pip.readthedocs.org/en/stable/installing/).
Download the Big Upload Module from its [Github Link](https://github.com/nsknojj/notebook/tree/big-upload) and put it in a position you like, then change to this directory in your cmd.
Run setup command:
```(python)
$ sudo python3 setup.py install
```
Run Notebook:
```(python)
$ jupyter notebook
```
It will open your default browser. Now you can try to upload a big file in your notebook.
![image](images/updemo.png)
## Design - How Big File Upload Works
### Prototype: [jQuery-File-Upload](https://github.com/blueimp/jQuery-File-Upload)
It has provided a jquery front end in which **chunked file upload** has been implemented, and a php back end to handle chunked file. It provides a series of api, such as **stop** and **abort**. You can define your own callback functions for **add**, **progress**, etc. All I need to do is transplanting the front end, and building a back end handler with tornado framework based on the given php server.
### The Front End & Back End of Jupyter Notebook
* Notebook's back end is built with tornado framework. You can see all the url handlers in *notebook/services*. Handlers in *notebook/services/contents* are related to uploading, downloading, saving and other functions we don't concern.
* What I call front end are those webpages, scripts and style sheets.
Webpage templates are in *notebook/templates*. We only concern "tree.html", which shows the tree structure of the opened file system.
Scripts relevant to my work are in *notebook/statics/tree/js*. They manage the file list and setup the elements in the tree webpage.
### Old Upload Process
```(python)
notebook
|----statics/tree/js/main.js # Setup upload button, upload it to api url
|----services/contents/handlers.py # Handle requests from url: api/contents/($file_path)
|----templates/tree.html
`----tree/handlers.py # Handle main url, render the tree template.
```
### Big Upload Process
```(python)
notebook
|----statics
| |----tree/js/main.js* # Setup big-upload button with scripts in [bigupload/js]
| `----[bigupload/js] # Now upload the chunked file to a new url: api/upload_handlers/($(file_path))
|
|----services
| |----[bigupload/handlers.py] # Handle requests from a new url
|
`----templates/tree.html* # Add BigUpload button
```
The suffix star in the file name means this file has been modified. The square brackets means this file is new.
### Divide Files to Chunks
* Front end uses [Blob API](https://developer.mozilla.org/en-US/docs/Web/API/Blob) to divide the file to several chunks, then upload it in a form using POST method. This form contains the content range of this chunk, e.g. 0-99999 or 100000-199999, and the content size.
* Back end just handles the files according to file path, content range, content size and the payload. It will find the correct file to append the payload to. The file's name should be same, and its current size should be equal to the start of content range.
## Experiment
### Bitrate
About 10MB/s ~ 20MB/s, on local host.
### Multifile Upload
Test passed, and the uploaded files had no difference.
### 5GB Big File Upload
Test passed, and the uploaded files had no difference.
### Cancel the Upload
Test passed, and the partially uploaded files were removed.

Binary file not shown.

After

Width:  |  Height:  |  Size: 71 KiB

View File

@ -0,0 +1,97 @@
class: center, middle
# Admin Interface
Author: [Fan Shixiong](mailto:807976844@qq.com)
Source Code: https://github.com/fanshibear/docklet
---
# Goals
1. To provide an interface for administrators to browse system settings that are recorded in docklet.conf and container.conf.
2. To enable administrators to modify parameters that will take effect immediately.
---
# Design
1. Add a table to the original Admin page to display system settings.
2. Provide a button for every parameter to view the detailed comments.
3. Provide a button for every aprameter that can be modified to modify it.
4. Display default value and historical values for every parameter.
5. Set these default and historical values clickable to reuse them.
6. For container.conf, add a text field to show its complete content.
---
# Design
![图1](./图1.png)
---
# Design
![图2](./图2.png)
---
# Design
![图3](./图3.png)
---
# Design
![图4](./图4.png)
---
# Design
![图5](./图5.png)
---
# Design
![图6](./图6.png)
---
# Design
![图7](./图7.png)
---
# Design
![图8](./图8.png)
---
# Design
![图9](./图9.png)
---
# Back-end Design
1. Parse the configuration files by regular expression.
2. Store default value and historical values in configuration files with special formats.
---
# Experiments
1. Browse the table to check if all parameters are displayed in the right way. Succeeeded.
2. Click every button to check if it responses in the right way. Succeeded.
3. Modify a parameter to check if it works. Succeeded.
4. Browse the table again to check if there is a change at the historical-value position of the modified parameter. Succeeded.
---
# Gains
1. Learned about many useful tools and frameworks like Git, Flask and Bootstrap.
2. Became more familiar with programming languages like Python, HTML and Shell script when reading and writing codes.
3. Got much knowledge of Linux System especially on its network and file system.
4. Understood what Container is and how it works.

View File

@ -0,0 +1,32 @@
<!DOCTYPE html>
<html>
<head>
<title>Pull Request Documentation</title>
<meta charset="utf-8">
<style>
body { font-family: 'Droid Serif'; }
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: normal;
}
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
</style>
</head>
<body>
<!--
<textarea id="source">
</textarea>
-->
<script src="remark.min.js"> </script>
<script>
var slideshow = remark.create({
sourceUrl: '2016-adminInterface.md'
});
</script>
<!--
<script type="text/javascript" src="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.js"></script>
<link rel="stylesheet" href="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.css">
<script>mermaid.initialize({startOnLoad:true});</script>
-->
</body>
</html>

File diff suppressed because one or more lines are too long

Binary file not shown.

After

Width:  |  Height:  |  Size: 30 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 28 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 24 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 21 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 90 KiB

View File

@ -0,0 +1,32 @@
<!DOCTYPE html>
<html>
<head>
<title>Pull Request Documentation</title>
<meta charset="utf-8">
<style>
body { font-family: 'Droid Serif'; }
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: normal;
}
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
</style>
</head>
<body>
<!--
<textarea id="source">
</textarea>
-->
<script src="../js/remark.min.js"> </script>
<script>
var slideshow = remark.create({
sourceUrl: 'source.md'
});
</script>
<!--
<script type="text/javascript" src="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.js"></script>
<link rel="stylesheet" href="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.css">
<script>mermaid.initialize({startOnLoad:true});</script>
-->
</body>
</html>

View File

@ -0,0 +1,50 @@
class: center, middle
# Monitor功能的优化
信科13级 朱域坚
---
# 主要完成的工作
1. 容器和物理机磁盘使用信息的监控
2. 性能优化以rpc机制代替原来的etcd传递信息
3. 容器真实运行时间的统计
---
# 磁盘使用信息的监控
- 利用python模块psutil的`disk_usage`函数收集信息。
- 该函数可以收集所有挂载的磁盘使用信息,由于容器使用时会分一个单独的磁盘挂载到它的根目录,因此可以收集到。
---
# 用rpc机制代替原来的etcd
## etcd存在的问题
- 读写效率太低etcd每秒大约只能进行千次的读写若每个容器每秒需要写一次那最多只能支持千个容器同时存在这是远远不够的。
- 消耗资源过多etcd的读写是通过http协议进行的效率比较低经测试单个worker每秒写个位数的容器信息也要占用1%以上的CPU
---
# 原来的设计
![](images/monitor.jpg)
---
# rpc机制的设计
![](images/newmonitor.jpg)
- 现有的worker负载cpu使用率降到0.5以下
---
# 真实运行时间的统计
- 根据容器的pid, 利用ps命令收集容器运行的时间只是进程运行时间容器重启后进程会变
- 因此真实时间还要加上lasttime初始时为0
- 若收集过程中pid与之前不同说明容器重启了则把之前的时间放到lasttime里

31
doc/pullreqexp/README.md Normal file
View File

@ -0,0 +1,31 @@
Pull Requests of Experiments
============================
Some contributors may experiment new features and send pull requests to Docklet official repository. Sometimes the requests will be refused for reasons like functional incompatibility, deviation of development plan, lacking of fully testing, etc. However, some experiments may be very interesting and helpful. Therefore we recommend contributors write down the goal, design, and evaluation of their experiments in markdown format, and send pull requests here about the markdown file.
## Guide
1. Experiment on the forked unias/docklet repository and make sure it work as expected
2. Switch to **experiment** branch, write down a markdown doc named to document the experiment
3. Send a pull request to unias/docklet **experiment** branch
## Doc template
```
# Title of the Experiments
Author: [Author name](Author email)
Source Code: https://github.com/yourname/docklet/
## Goal
## Design
## Experiments
## Summary
```
It is recommended to write markdown that could be interepreted by
remark, please refer the **demo** dir.

View File

@ -0,0 +1,32 @@
<!DOCTYPE html>
<html>
<head>
<title>Pull Request Documentation</title>
<meta charset="utf-8">
<style>
body { font-family: 'Droid Serif'; }
h1, h2, h3 {
font-family: 'Yanone Kaffeesatz';
font-weight: normal;
}
.remark-code, .remark-inline-code { font-family: 'Ubuntu Mono'; }
</style>
</head>
<body>
<!--
<textarea id="source">
</textarea>
-->
<script src="../js/remark.min.js"> </script>
<script>
var slideshow = remark.create({
sourceUrl: 'source.md'
});
</script>
<!--
<script type="text/javascript" src="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.js"></script>
<link rel="stylesheet" href="http://cdn.bootcss.com/mermaid/0.5.8/mermaid.min.css">
<script>mermaid.initialize({startOnLoad:true});</script>
-->
</body>
</html>

View File

@ -0,0 +1,26 @@
class: center, middle
# Demo feature name
Author: [Tony](mailto:tony@github.com)
Source Code: https://github.com/tony/docklet/tree/demofeature
---
# Goal
1. goal 1
2. goal 2
---
# Design
architecture
---
# Experiment

16
doc/pullreqexp/js/remark.min.js vendored Normal file

File diff suppressed because one or more lines are too long

View File

@ -1,4 +1,4 @@
from flask import session
from flask import session, redirect
from webViews.view import normalView
from webViews.dockletrequest import dockletRequest
from webViews.dashboard import *
@ -34,7 +34,7 @@ class createClusterView(normalView):
}
result = dockletRequest.post("/cluster/create/", data)
if(result.get('success', None) == "true"):
return dashboardView.as_view()
return redirect("/dashboard/")
#return self.render(self.template_path, user = session['username'])
else:
return self.render(self.error_path, message = result.get('message'))
@ -73,7 +73,7 @@ class scaleoutView(normalView):
}
result = dockletRequest.post("/cluster/scaleout/", data)
if(result.get('success', None) == "true"):
return configView.as_view()
return redirect("/config/")
else:
return self.render(self.error_path, message = result.get('message'))
@ -86,7 +86,7 @@ class scaleinView(normalView):
}
result = dockletRequest.post("/cluster/scalein/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()
@ -112,7 +112,7 @@ class startClusterView(normalView):
}
result = dockletRequest.post("/cluster/start/", data)
if(result):
return dashboardView.as_view()
return redirect("/dashboard/")
else:
return self.error()
@ -126,7 +126,7 @@ class stopClusterView(normalView):
}
result = dockletRequest.post("/cluster/stop/", data)
if(result):
return dashboardView.as_view()
return redirect("/dashboard/")
else:
return self.error()
@ -160,7 +160,7 @@ class deleteClusterView(normalView):
}
result = dockletRequest.post("/cluster/delete/", data)
if(result):
return dashboardView.as_view()
return redirect("/dashboard/")
else:
return self.error()
@ -199,7 +199,7 @@ class saveImageView(normalView):
if(result):
if result.get('success') == 'true':
#return self.render(self.success_path, user = session['username'])
return configView.as_view()
return redirect("/config/")
#res = detailClusterView()
#res.clustername = self.clustername
#return res.as_view()
@ -221,7 +221,7 @@ class shareImageView(normalView):
}
result = dockletRequest.post("/image/share/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()
@ -235,7 +235,7 @@ class unshareImageView(normalView):
}
result = dockletRequest.post("/image/unshare/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()
@ -249,7 +249,7 @@ class deleteImageView(normalView):
}
result = dockletRequest.post("/image/delete/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()
@ -264,7 +264,7 @@ class addproxyView(normalView):
}
result = dockletRequest.post("/addproxy/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()
@ -277,7 +277,7 @@ class deleteproxyView(normalView):
}
result = dockletRequest.post("/deleteproxy/", data)
if(result):
return configView.as_view()
return redirect("/config/")
else:
self.error()