파이썬 언어로 개발되었으며, 몽고 로그 파일 분석하여 Query 비용 분석과 시각화 도구를 제공한다.

# install

https://github.com/rueckstiess/mtools/releases

pip install mtools --upgrade

mloginfo mongod.log --queries

QUERIES

namespace                    pattern                                        count    min (ms)    max (ms)    mean (ms)    95%-ile (ms)    sum (ms)

serverside.scrum_master      {"datetime_used": {"$ne": 1}}                     20       15753       17083        16434         17039.3      328692

serverside.django_session    {"_id": 1}                                       562         101        1512          317           842.6      178168

serverside.user              {"_types": 1, "emails.email": 1}                 804         101        1262          201          684.85      162311

local.slaves                 {"_id": 1, "host": 1, "ns": 1}                   131         101        1048          310           819.5       40738

serverside.email_alerts      {"_types": 1, "email": 1, "pp_user_id": 1}        13         153       11639         2465          8865.2       32053

# 참고링크

http://blog.mongodb.org/post/85123256973/introducing-mtools 

오랜만에 몽고DB 3.0.0-rc8 버젼업되어서 설치해봅니다^^

/*
    mongodb 3.0.0 rc8 install

*/

# http://www.mongodb.org/dl/linux/x86_64
# 다운로드 
cd ~/
wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-3.0.0-rc8.tgz
tar -xf mongodb-linux-x86_64-3.0.0-rc8.tgz
mkdir -p /var/mongodb_3.0.0
mv mongodb-linux-x86_64-3.0.0-rc8/* /var/mongodb_3.0.0/


# 데이타 및 로그 폴더 설정 
mkdir -p /usr1/mongodb_3.0.0
mkdir -p /usr1/mongodb_3.0.0/logs

# 설정 파일 
vi /etc/mongodb_3.0.0.conf
---------------------------------------------------------------
fork = true
#bind_ip = 0.0.0.0
port = 9999
quiet = true
dbpath = /usr1/mongodb_3.0.0
logpath = /usr1/mongodb_3.0.0/logs/mongod.log
logappend = true
journal = true
#auth = true

# 실행 쉘 스크립트 변경 
---------------------------------------------------------------
wget https://ewan.im/wp-content/uploads/2010/04/mongodb.txt 
cp mongodb.txt /etc/init.d/mongodb3
chmod 755 /etc/init.d/mongodb3

>> 설정 파일 변경
mongod path 변경 및 conf 변경
---------------------------------------------------------------
vi /etc/init.d/mongodb3
    --config /etc/mongodb_3.0.0.conf

>> 서비스 시작 설정    
chkconfig --add mongodb3
chkconfig mongodb3 on   
  
# Starts up MongoDB right now
/etc/init.d/mongodb3 start
   
# 로그 확인 
tail -30 /usr1/mongodb_3.0.0/logs/mongod.log
---------------------------------------------------------------
2015-02-13T08:57:42.067+0900 I JOURNAL  [initandlisten] journal dir=/usr1/mongodb_3.0.0/journal
2015-02-13T08:57:42.067+0900 I JOURNAL  [initandlisten] recover : no journal files present, no recovery needed
2015-02-13T08:57:42.129+0900 I JOURNAL  [durability] Durability thread started
2015-02-13T08:57:42.129+0900 I JOURNAL  [journal writer] Journal writer thread started
2015-02-13T08:57:42.131+0900 I INDEX    [initandlisten] allocating new ns file /usr1/mongodb_3.0.0/local.ns, filling with zeroes...
2015-02-13T08:57:42.286+0900 I STORAGE  [FileAllocator] allocating new datafile /usr1/mongodb_3.0.0/local.0, filling with zeroes...
2015-02-13T08:57:42.286+0900 I STORAGE  [FileAllocator] creating directory /usr1/mongodb_3.0.0/_tmp
2015-02-13T08:57:42.352+0900 I STORAGE  [FileAllocator] done allocating datafile /usr1/mongodb_3.0.0/local.0, size: 64MB,  took 0.057 secs
2015-02-13T08:57:42.369+0900 I NETWORK  [initandlisten] waiting for connections on port 9999

# storage engine 변경해보자~~
vi /etc/init.d/mongodb3
--storageEngine wiredTiger
---------------------------------------------------------------
 # Source function library
 12 . /etc/rc.d/init.d/functions
 13
 14 if [ -f /etc/sysconfig/mongodb ]; then
 15     . /etc/sysconfig/mongodb
 16 fi
 17
 18 prog="mongod"
 19 mongod="/var/mongodb_3.0.0/bin/mongod"
 20 RETVAL=0
 21
 22 start() {
 23     echo -n $"Starting $prog: "
 24     daemon $mongod "--fork --storageEngine wiredTiger --config /etc/mongodb_3.0.0.conf"
 25     RETVAL=$?
 26     echo
 27     [ $RETVAL -eq 0 ] && touch /var/lock/subsys/$prog
 28     return $RETVAL
 29 }
---------------------------------------------------------------

# 서비스 재 시작~~
/etc/init.d/mongodb3 start

# 오류 발생 
tail -30 /usr1/mongodb_3.0.0/logs/mongod.log
Detected data files in /usr1/mongodb_3.0.0 created by storage engine 'mmapv1'. The configured storage engine is 'wiredTiger'., terminating

# 아하~~ 좋아~
# 1. 기존 폴더 삭제 
rm -rf  /usr1/mongodb_3.0.0

# 2. 폴더 재생성 
mkdir -p /usr1/mongodb_3.0.0
mkdir -p /usr1/mongodb_3.0.0/logs

# 서비스 재 시작~~
/etc/init.d/mongodb3 start
--------------

2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] MongoDB starting : pid=6350 port=9999 dbpath=/usr1/mongodb_3.0.0 64-bit host=localhost.localdomain
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten]
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten]
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten]
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten]
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] db version v3.0.0-rc8
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] git version: 9d0714cdcffadb203ff68730acedbc40875403da
2015-02-13T09:05:13.514+0900 I CONTROL  [initandlisten] build info: Linux build4.nj1.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2015-02-13T09:05:13.515+0900 I CONTROL  [initandlisten] allocator: tcmalloc
2015-02-13T09:05:13.515+0900 I CONTROL  [initandlisten] options: { config: "/etc/mongodb_3.0.0.conf", net: { port: 9999 }, processManagement: { fork: true }, storage: { dbPath: "/usr1/mongodb_3.0.0", engine: "wiredTiger", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/usr1/mongodb_3.0.0/logs/mongod.log", quiet: true } }
2015-02-13T09:05:13.536+0900 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1G,session_max=20000,eviction=(threads_max=4),statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2015-02-13T09:05:13.730+0900 I NETWORK  [initandlisten] waiting for connections on port 9999

mongodb 2.8.0-rc4 몽고 성능 테스트 측정을 위해 sysbench-mongo라는 툴을 사용했다. 

https://github.com/tmcallaghan/sysbench-mongodb


1. 테스트 전 필요 설치 목록 

가. java sdk 설치 
    java sdk 1.6 or 1.7+ 이상 설치되어야 한다. 

    # 설치 가능한 목록 확인
    yum list java-*-devel

    # java sdk 설치 
    yum -y install java-1.6.0-openjdk-devel.x86_64

    [root@localhost ~]# java -version
    java version "1.6.0_33"
    OpenJDK Runtime Environment (IcedTea6 1.13.5) (rhel-1.13.5.1.el6_6-x86_64)
    OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)
    
나. mongo java driver 다운로드 설치 

    wget http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.11.4/mongo-java-driver-2.11.4.jar
    wget http://central.maven.org/maven2/org/mongodb/mongo-java-driver/2.13.0-rc1/mongo-java-driver-2.13.0-rc1.jar
    
다.  Sysbench 설치    

    git clone https://github.com/tmcallaghan/sysbench-mongodb.git

    # 2.4 수행시 설정 
    export CLASSPATH=$PWD/mongo-java-driver-2.11.4.jar:$CLASSPATH
    
    # 2.8 수행시 설정 
    export CLASSPATH=$PWD/mongo-java-driver-2.13.0-rc1.jar:$CLASSPATH

    #. 2.4와 2.8 함수명 변경으로 각각 수행시 드라이버 지정 변경해줘야 한다. 
    cp -r sysbench-mongodb  sysbench-mongodb-2.8    
    cd sysbench-mongodb        
    
    # 환경변수 해제 
    # unset CLASSPATH
    
    # java driver의 함수가 변경되었군!!
    https://github.com/mongodb/mongo-java-driver/releases

    Replace any calls to MongoCredential#createMongoCRCredential with calls to MongoCredential#createCredential.
    createMongoCRCredential -> createCredential     (함수명 변경)    
    
라. mongodb 2.8 rc4 가동 

    # 향상된 기능을 사용하기 위해서는  --storageEngine wiredtiger 엔진으로 옵션 지정해야 한다.    
    /var/mongodb_2.8.0/bin/mongod --config /etc/mongodb_2.8.0.conf --storageEngine wiredtiger
    
2. 테스트 진행

    # 환경 정보 설정 변경   (ip, port, collection 갯수 등등.. 설정 변경)
    vi config.bash

    테스트 실행
    ------------------------------------------------------------------------------------------

    # 2.8 버젼 
    ./run.simple.bash   config_28.bash

    # 2.4 버젼 
    ./run.simple.bash   config_24.bash    

    

증상 : 잘 되던 몽고DB의 맵리듀스가 갑자기 실행이 안되는 문제가 발생햇다. mapreduce의 반환 오류는 property를 읽을 수 없다고 하는데 도대체 먼 문제인지 알 수가 없었고, 몽고DB 로그 읽어 확인을 했다.

Fri Dec 20 14:46:05.328 [conn712] JavaScript execution failed: TypeError: Cannot read property 'Computer' of undefined near 'log.PcInfo.Computer;         if (reval.ip'  (line 12)
Fri Dec 20 14:46:05.328 [conn712] warning: ClientCursor::YieldLock not closed properly

Fri Dec 20 14:46:05.328 [conn712] mr failed, removing collection :: caused by :: 16722 JavaScript execution failed: TypeError: Cannot read property 'Computer' of undefined near 'log.PcInfo.Computer;         if (reval.ip'  (line 12)
Fri Dec 20 14:46:05.801 [conn712] JavaScript execution failed: TypeError: Cannot read property 'Computer' of undefined near 'log.PcInfo.Computer;         if (reval.ip'  (line 12)
Fri Dec 20 14:46:05.801 [conn712] warning: ClientCursor::YieldLock not closed properly

해결 : Mapreduce의 value 값이 16M가 넘어서 그렇다고하는데 ㅡ.ㅡ 정확한 것인가? 일단 그래서 map의 value값을 this에서  필요한 값들로 변경해서 해결했다.

https://jira.mongodb.org/browse/SERVER-5452

this error means that one document being created during MR is larger than the max size of 16MB.
You should try to break any large document up in smaller ones with the emit key, or change the process in some way.

https://sites.google.com/site/tadpolefordb/  -- 국내 오픈 소스

http://edgytech.com/umongo/
http://robomongo.org/

http://www.hodgin.ca/downloads/Mongo%20Shell%20Cheat%20Sheet.pdf

Daum's Business Analytics Use-cases based on Bigdata technology

http://www.slideshare.net/Channy/daum-bigdata-analytics-usecases

MongoDB Configuration File Tuning

http://architects.dzone.com/articles/mongodb-configuration-file

http://www.dzone.com/links/r/2012_nosql_in_review.html

A more subtle MongoDB limitation is that when adding or updating a field in a document, the entire document must be re-written. If you pre-allocate space for each document, you can avoid the associated fragmentation, but even with pre-allocation updating your document gets slower as it grows.

http://blog.pythonisito.com/2012/09/mongodb-schema-design-at-scale.html

업데이트 할때 전체 문서를 다시 쓰기 발생~ 헉헉.... in-place update