以下均为本人猜测,正确理解请查阅官方资料. s表示script, bin表示可执行文件, local表示本机的, usr表示用户的

/usr/local/sbin #用户自己安装的,脚本可执行文件
/usr/local/bin #用户自己安装的,二进制可执行文件
/usr/sbin #发行版默认安装的,脚本可执行文件
/usr/bin #发行版默认安装的,二进制可执行文件
/sbin #linux自带的,脚本可执行文件
/bin #linux自带的,二进制可执行文件

如果自己安装了一个软件,想在shell中当命令使用, 需要做软连接到相应目录. 比如 假设 /home/xxx/softwares/rspec/bin/rspec是用户自己装的软件, 基于ruby写的脚本程序.那么应该:

ln -s /home/xxx/softwares/rspec/bin/rspec /usr/local/sbin

并且保证/usr/local/sbin在$PATH中能找到,就可以当普通命令用了.

参考文献

http://www.cnblogs.com/forfuture1978/archive/2010/11/14/1877086.html map-reduce http://hbtc2012.hadooper.cn/ http://www.ibm.com/developerworks/cn/opensource/os-log-process-hadoop/ http://www.ibm.com/developerworks/cn/linux/l-hadoop-3/ http://blog.csdn.net/zhaoyl03/article/details/8657031 https://192.168.9.247:8443 azkaban azkaban http://hive001.hadoop.bjy.elong.com:81/data/input/iis/website/tj/20121218/

调试 中间结果(mapping 后的结果)

-numReduceTasks 0

常用命令

hadoop fs -ls /data/logs/webserver/website/hotel/20131128
hadoop fs -cat /data/logs/webserver/website/hotel/20131128/log.1385589547988

删除文件和目录

hadoop fs -rmr /data/logs/webserver/website/hotel/20131128
hadoop fs -rm /data/logs/webserver/website/hotel/20131128/log.1385589547988

结果合并后导出

hadoop fs -getmerge  /data/logs/webserver/website/hotel/20131128 /homedir/username/output

按范围查询hdfs

hadoop fs -ls /data/logs/webserver/website/hotel/{20131129..20131205}

按范围输入

input /homedir/username/keyword_convertion/searchengine/{20131202,20131203,20131204,20131205,20131206,20131207,20131208}

首先安装 python以及 pip

然后继续安装 coursera-dl

pip install coursera-dl

使用方法

coursera-dl -u username -p password classname

比如

coursera-dl -u MYUSERNAME -p MYPASSWORD -d MYPATH algo-2012-001 ml-2012-002

https://github.com/dgorissen/coursera-dl

让authorized_keys生效必须保证~/.ssh权限是700

scp复制远端文件带通配符,就是需要把星号转义一下

scp username@hostname:/path/to/dir/filename\* .

删除文件开头BOM标记

awk '{if(NR==1)sub(/^\xef\xbb\xbf/,"");print}' inputfile > outputfile

查看网络使用情况

lsof -i
netstat -lptu
netstat -tulpn

查看当前使用的shell

echo $0

关闭ssh连接后,仍然保持程序后台运行的方法

nohup program & 例如:

nohup ruby something.rb 2>errors &

crond配置说明

* * * * * command

按前后顺序,分别表示:

分钟 0 ~59 小时 0 ~ 23 Day of Month 0 ~ 31 月 0 ~ 12 Day of Week 1 ~ 7

CentOS 163镜像使用方法

http://mirrors.163.com/.help/centos.html http://mirrors.163.com/.help/CentOS6-Base-163.repo

There is always an error when building ruby scripts in Sublime from “Menu -> Tools -> Build”

The reason is Sublime sticks to the standard $PATH defined by Mac system deafult, while shell is using its own $PATH.

To be specific, zsh refers to ~/.zshrc, and sublime refers to /etc/zshenv which refers to /etc/paths

You can build the ruby scripts in sublime and run them in shell to see the difference

puts `which ruby`
puts `which gem`
puts `echo $PATH`

My sublime prints the system default ruby2.0 linked from /usr/bin while my zsh prints the brew installed ruby2.1 linked from /usr/local/bin. And it goes the same for gem.

The solution is quite simple: to edit /etc/paths until the paths and order are what you need.