2017/09/17
Installing Hadoopy
git clone https://github.com/bwhite/hadoopy.git cd hadoopy sudo python setup.py install
hduser@master:~/hadoopy$ sudo yum install python-devel 没有已启用的仓库。
hduser@master:~/hadoopy$ sudo yum install python-devel
没有已启用的仓库。
执行 "yum repolist all" 查看您拥有的仓库。
您可以用 yum-config-manager --enable <仓库名> 来启用仓库
hduser@master:~/hadoopy$ yum-config-manager --enable
程式 'yum-config-manager' 目前尚未安裝。 您可以由輸入以下內容安裝：
sudo apt-get install yum-utils
hduser@master:~/hadoopy$ sudo apt-get install yum-utils
正在讀取套件清單... 完成
正在重建相依關係
正在讀取狀態資料... 完成
下列的額外套件將被安裝：
python-iniparse
下列【新】套件將會被安裝：
python-iniparse yum-utils
升級 0 個，新安裝 2 個，移除 0 個，有 318 個未被升級。
需要下載 108 kB 的套件檔。
此操作完成之後，會多佔用 516 kB 的磁碟空間。
Do you want to continue? [Y/n] y
下載:1 http://ubuntu.cs.nctu.edu.tw/ubuntu/ trusty/universe python-iniparse all 0.4-2.1build1 [21.5 kB]
下載:2 http://ubuntu.cs.nctu.edu.tw/ubuntu/ trusty/universe yum-utils all 1.1.31-2 [86.6 kB]
取得 108 kB 用了 0秒 (206 kB/s)
選取了原先未選的套件 python-iniparse。
（讀取資料庫 ... 目前共安裝了 178600 個檔案和目錄。）
準備解開 .../python-iniparse_0.4-2.1build1_all.deb ...
解開 python-iniparse (0.4-2.1build1) 中...
選取了原先未選的套件 yum-utils。
準備解開 .../yum-utils_1.1.31-2_all.deb ...
解開 yum-utils (1.1.31-2) 中...
Processing triggers for doc-base (0.10.5) ...
正在處理 1 added doc-base file...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
設定 python-iniparse (0.4-2.1build1) ...
設定 yum-utils (1.1.31-2) ...
2017/09/15
TypeError: not enough arguments for format string
原本
import csv
import MySQLdb
print "DB connecting start"
mydb = MySQLdb.connect(user='root',passwd='root',host='localhost', db='ECDatabase' )
print "DB connecting end"
cursor = mydb.cursor()
csv_data = csv.reader(file('data/productClick.csv'))
for row in csv_data:
cursor.execute("INSERT INTO ProductClick ( userId, productId, clickCount ) VALUES ( '%s', '%s', '%s' );", row)
#close the connection to the database.
mydb.commit()
cursor.close()
print "Done"
出現錯誤：TypeError: not enough arguments for format string
原因是csv的內容為tab分隔
修改
import csv
import MySQLdb
print "DB connecting start"
mydb = MySQLdb.connect(user='root',passwd='root',host='localhost', db='ECDatabase' )
print "DB connecting end"
cursor = mydb.cursor()
csv_data = csv.reader(file('data/productClick.csv'))
for row in csv_data:
data=row[0].split('\t')
# query = """insert into newsletter_subscriber (id, name, email) values ('%s', '%s', '%s')""" %(data[0], data[1], data[2])
cursor.execute("INSERT INTO ProductClick ( userId, productId, clickCount ) VALUES ( '%s', '%s', '%s' );" %(data[0], data[1], data[2]))
#close the connection to the database.
mydb.commit()
cursor.close()
print "Done"
python EnvironmentError: mysql_config not found
sh: 1: mysql_config: not found
Traceback (most recent call last):
File "setup.py", line 18, in
metadata, options = get_config()
File "/home/hduser/MySQL-python-1.2.4b4/setup_posix.py", line 44, in get_config
libs = mysql_config("libs_r")
File "/home/hduser/MySQL-python-1.2.4b4/setup_posix.py", line 25, in mysql_config
raise EnvironmentError("%s not found" % (mysql_config.path,))
EnvironmentError: mysql_config not found
hduser@master:~/MySQL-python-1.2.4b4$ sudo apt-get install libmysqld-dev
找到mysql_config目錄的所在位置
find / -name mysql_config
修改
hduser@master:~/MySQL-python-1.2.4b4$ sudo gedit setup_posix.py
內容，如下
mysql_config.path = "/usr/bin/mysql_config"hduser@master:~/MySQL-python-1.2.4b4$ python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.linux-x86_64-2.7/MySQLdb
running build_ext
hduser@master:~/MySQL-python-1.2.4b4$
hduser@master:~/MySQL-python-1.2.4b4$ python setup.py build
running build
running build_py
copying MySQLdb/release.py -> build/lib.linux-x86_64-2.7/MySQLdb
running build_ext
hduser@master:~/MySQL-python-1.2.4b4$ python setup.py install
running install
running bdist_egg
running egg_info
writing MySQL_python.egg-info/PKG-INFO
writing top-level names to MySQL_python.egg-info/top_level.txt
writing dependency_links to MySQL_python.egg-info/dependency_links.txt
writing MySQL_python.egg-info/PKG-INFO
writing top-level names to MySQL_python.egg-info/top_level.txt
writing dependency_links to MySQL_python.egg-info/dependency_links.txt
reading manifest file 'MySQL_python.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'MySQL_python.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
copying MySQLdb/release.py -> build/lib.linux-x86_64-2.7/MySQLdb
running build_ext
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/_mysql_exceptions.py -> build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/_mysql.so -> build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/MySQLdb
copying build/lib.linux-x86_64-2.7/MySQLdb/cursors.py -> build/bdist.linux-x86_64/egg/MySQLdb
copying build/lib.linux-x86_64-2.7/MySQLdb/connections.py -> build/bdist.linux-x86_64/egg/MySQLdb
copying build/lib.linux-x86_64-2.7/MySQLdb/__init__.py -> build/bdist.linux-x86_64/egg/MySQLdb
creating build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/FIELD_TYPE.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/ER.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/REFRESH.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/CR.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/FLAG.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/CLIENT.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/constants/__init__.py -> build/bdist.linux-x86_64/egg/MySQLdb/constants
copying build/lib.linux-x86_64-2.7/MySQLdb/times.py -> build/bdist.linux-x86_64/egg/MySQLdb
copying build/lib.linux-x86_64-2.7/MySQLdb/release.py -> build/bdist.linux-x86_64/egg/MySQLdb
copying build/lib.linux-x86_64-2.7/MySQLdb/converters.py -> build/bdist.linux-x86_64/egg/MySQLdb
byte-compiling build/bdist.linux-x86_64/egg/_mysql_exceptions.py to _mysql_exceptions.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/cursors.py to cursors.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/connections.py to connections.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/FIELD_TYPE.py to FIELD_TYPE.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/ER.py to ER.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/REFRESH.py to REFRESH.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/CR.py to CR.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/FLAG.py to FLAG.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/CLIENT.py to CLIENT.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/constants/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/times.py to times.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/release.py to release.pyc
byte-compiling build/bdist.linux-x86_64/egg/MySQLdb/converters.py to converters.pyc
creating stub loader for _mysql.so
byte-compiling build/bdist.linux-x86_64/egg/_mysql.py to _mysql.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying MySQL_python.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MySQL_python.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MySQL_python.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying MySQL_python.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing distribute-0.6.28-py2.7.egg
creating /home/hduser/anaconda2/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg
Extracting distribute-0.6.28-py2.7.egg to /home/hduser/anaconda2/lib/python2.7/site-packages
Adding distribute 0.6.28 to easy-install.pth file
Installing easy_install script to /home/hduser/anaconda2/bin
Installing easy_install-2.7 script to /home/hduser/anaconda2/bin
Installed /home/hduser/anaconda2/lib/python2.7/site-packages/distribute-0.6.28-py2.7.egg
Processing dependencies for distribute==0.6.28
Finished processing dependencies for distribute==0.6.28
Processing MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg
creating /home/hduser/anaconda2/lib/python2.7/site-packages/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg
Extracting MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg to /home/hduser/anaconda2/lib/python2.7/site-packages
Adding MySQL-python 1.2.4b4 to easy-install.pth file
Installed /home/hduser/anaconda2/lib/python2.7/site-packages/MySQL_python-1.2.4b4-py2.7-linux-x86_64.egg
Processing dependencies for MySQL-python==1.2.4b4
Finished processing dependencies for MySQL-python==1.2.4b4
執行Eclipse Window >> Preferences >> Apply
2017/09/14
安裝mysqlbench
$ wget http://dev.mysql.com/get/mysql-apt-config_0.3.5-1ubuntu14.04_all.deb
$ sudo dpkg -i mysql-apt-config_0.3.5-1ubuntu14.04_all.deb
$ sudo apt-get update -y
devops@devopsservice:~
2017/09/04
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. org/bson/conversions/Bson
solution:
缺bson-VERSION.jar檔案
確認mongodb版本
db version v3.4.7
git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1404
distarch: x86_64
target_arch: x86_64
FAILED: ParseException line 1:87 Failed to recognize predicate 'Timestamp'. Failed rule: 'identifier' in column specification
solution:
hive > set hive.support.sql11.reserved.keywords=false;
訂閱：
文章 (Atom)