pika 简单分析

[toc]

why

梳理思路，省着忘了

0. trivial

• 全局PikaServer对象，所有变量都用读写锁保护了。真的有必要吗。
• 模块拆分 （pika主程序，pink网络模块，blackwidow编码模块，slash公共组件（这个公共组件模块很坑爹），glog），底层模块又互相依赖。挺头疼的（需要画个图）

• 利用多线程优势，类似memcache用libevent哪种用法。主事件循环处理io事件，写pipe通知子线程搞定

• redis协议分析我以为得放在pika主程序中，结果没想到在pink里。糊一起了。之前还好奇，翻pika代码没发现redis代码，难道解析redis居然没用到redis源码自己搞的，结果在pink里。

1. 整体架构

一图胜千言 很多redis over rocksdb的实现都是在编码上有各种异同，比如ssdb ardb之类的，pika怎么做的？上图

复杂数据结构 set zset hash 是分成元数据和实体数据来做的。(大家都抄linux)

pika编译踩坑

1. 运行测试，二进制文件运行几次测试用例就挂了，应该是不支持wait直接崩了

下面是log记录。我个人猜测是运行环境的问题，准备从头编译

bash pikatests.sh wait

ERROR:path : ./tests/tmp/redis.conf.29436.2
-----------Pika server 3.0.5 ----------
-----------Pika config list----------
1 thread-num 1
2 sync-thread-num 6
3 sync-buffer-size 10
4 log-path ./log/
5 loglevel info
6 db-path ./db/
7 write-buffer-size 268435456
8 timeout 60
9 requirepass
10 masterauth
11 userpass
12 userblacklist
13 dump-prefix
14 dump-path ./dump/
15 dump-expire 0
16 pidfile ./pika.pid
17 maxclients 20000
18 target-file-size-base 20971520
19 expire-logs-days 7
20 expire-logs-nums 10
21 root-connection-num 2
22 slowlog-write-errorlog no
23 slowlog-log-slower-than 10000
24 slowlog-max-len 128
25 slave-read-only yes
26 db-sync-path ./dbsync/
27 db-sync-speed -1
28 slave-priority 100
29 server-id 1
30 double-master-ip
31 double-master-port
32 double-master-server-id
33 write-binlog yes
34 binlog-file-size 104857600
35 identify-binlog-type new
36 max-cache-statistic-keys 0
37 small-compaction-threshold 5000
38 compression snappy
39 max-background-flushes 1
40 max-background-compactions 2
41 max-cache-files 5000
42 max-bytes-for-level-multiplier 10
-----------Pika config end----------
WARNING: Logging before InitGoogleLogging() is written to STDERR
W1217 17:22:48.644249 29438 pika.cc:167] your 'limit -n ' of 1024 is not enough for Redis to start. pika have successfully reconfig it to 25000
I1217 17:22:48.644497 29438 pika.cc:184] Server at: ./tests/tmp/redis.conf.29436.2
I1217 17:22:48.644691 29438 pika_server.cc:192] Using Networker Interface: eth0
I1217 17:22:48.644783 29438 pika_server.cc:235] host: 192.168.1.104 port: 0
I1217 17:22:48.644820 29438 pika_server.cc:68] Prepare Blackwidow DB...
I1217 17:22:48.776921 29438 pika_server.cc:73] DB Success
I1217 17:22:48.776942 29438 pika_server.cc:89] Worker queue limit is 20100
I1217 17:22:48.777190 29438 pika_binlog.cc:103] Binlog: Manifest file not exist, we create a new one.
I1217 17:22:48.777328 29438 pika_server.cc:109] double recv info: filenum 0 offset 0
F1217 17:22:48.779913 29438 pika_server.cc:284] Start BinlogReceiver Error: 1: bind port 1000 conflict, Listen on this port to handle the data sent by the Binlog Sender
*** Check failure stack trace: ***
@ 0x83c5aa google::LogMessage::Fail()
@ 0x83e2ff google::LogMessage::SendToLog()
@ 0x83c1f8 google::LogMessage::Flush()
@ 0x83ec2e google::LogMessageFatal::~LogMessageFatal()
@ 0x5f5725 PikaServer::Start()
@ 0x423db4 main
@ 0x7fe733418bb5 __libc_start_main
@ 0x58ac71 (unknown)
[1/1 done]: unit/wait (5 seconds)

The End

Execution time of different units:
5 seconds - unit/wait

\2. 源码编译由于我的服务器限制不能访问外网，自己手动git clone然后导回服务器（这里又有一个坑）（win7没有wsl，只能用服务器搞）

然后需要依赖子模块

git submodule init
git submodule update

重新在windows上更新后传回

分别编译，

子模块编译失败，其中有文件格式不对的问题 手动转dos2unix

然后才发现tortoisegit有个自动添加换行符的选项，默认打勾。所以基本上涉及到的shell文件都得手动dos2unix

我的服务器只有4.8.3，所以编译rocksdb需要c++11 需要修改Makefile（其实是detect脚本没加权限的问题）

编译glog失败

./configure make失败 提示没有什么repo

搜到了类似的问题，https://stackoverflow.com/questions/18839857/deps-po-no-such-file-or-directory-compiler-error

按照提示 （需要安装libtool）

autoreconf -if

提示

.ibtoolize: AC_CONFIG_MACRO_DIR([m4]) conflicts with ACLOCAL_AMFLAGS=-I m4 

结果还是gitwindows加换行符导致的。 That darn “libtoolize: AC_CONFIG_MACRO_DIR([m4]) conflicts with ACLOCAL_AMFLAGS=-I m4” error Is caused by using CRLFs in Makefile.am. “m4" != "m4" and thus the libtoolize script will produce an error. If you're using git, I strongly advise adding a .gitattributes file with the following: *.sh -crlf *.ac -crlf *.am -crlf http://pete.akeo.ie/2010/12/that-darn-libtoolize-acconfigmacrodirm4.html

最终编译过了，运行提示找不到glog 还要手动ldconfig一下

echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig

还有一种解决方案，export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

总算能运行测试了。

redis 中runtest脚本不能直接用，但是测试用例可以直接拿过来用，support/server.tcl文件改下start_server函数就行（pika已经改过了，直接用那个文件就好了）

测试，类似下面，改改应该就行了

#!/bin/bash
for f in ${PWD}/tests/unit/*;do
    filename=${f##*/}
    tclsh tests/test_helper.tcl --clients 1 --single unit/${filename%.*}
done

for f in ${PWD}/tests/unit/type/*;do
    filename=${f##*/}
    tclsh tests/test_helper.tcl --clients 5 --single  unit/type/${filename%.*}
    #tclsh tests/test_helper.tcl --clients 1 --single unit/type/${filename%.*}
done

附调用图一张。组件太多，确实找不到在哪。

遇到的问题

• valgrind不能attach到daemon进程上

只能停掉daemon从命令行执行

• vallgrind提示非法指令

vex amd64->IR: unhandled instruction bytes: 0x62 0xF1 0x7D 0x48 0xEF 0xC0 0x48 0x8D
vex amd64->IR:   REX=0 REX.W=0 REX.R=0 REX.X=0 REX.B=0
vex amd64->IR:   VEX=0 VEX.L=0 VEX.nVVVV=0x0 ESC=NONE
vex amd64->IR:   PFX.66=0 PFX.F2=0 PFX.F3=0
==17407== valgrind: Unrecognised instruction at address 0x815dc7.
==17407==    at 0x815DC7: rocksdb::BlockBasedTableFactory::BlockBasedTableFactory(rocksdb::BlockBasedTableOptions const&) (block_based_table_factory.cc:44)
==17407==    by 0x7FC64D: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions() (options.cc:99)
==17407==    by 0x4762EF: __static_initialization_and_destruction_0(int, int) [clone .constprop.1298] (options_helper.h:387)
==17407==    by 0x8E011C: __libc_csu_init (in /home/vdb/pika/output/pika)
==17407==    by 0x65F2B44: (below main) (in /usr/lib64/libc-2.17.so)
==17407== Your program just tried to execute an instruction that Valgrind
==17407== did not recognise.  There are two possible reasons for this.
==17407== 1. Your program has a bug and erroneously jumped to a non-code
==17407==    location.  If you are running Memcheck and you just saw a
==17407==    warning about a bad jump, it's probably your program's fault.
==17407== 2. The instruction is legitimate but Valgrind doesn't handle it,
==17407==    i.e. it's Valgrind's fault.  If you think this is the case or
==17407==    you are not sure, please let us know and we'll try to fix it.
==17407== Either way, Valgrind will now raise a SIGILL signal which will
==17407== probably kill your program.
==17407==
==17407== Process terminating with default action of signal 4 (SIGILL)
==17407==  Illegal opcode at address 0x815DC7
==17407==    at 0x815DC7: rocksdb::BlockBasedTableFactory::BlockBasedTableFactory(rocksdb::BlockBasedTableOptions const&) (block_based_table_factory.cc:44)
==17407==    by 0x7FC64D: rocksdb::ColumnFamilyOptions::ColumnFamilyOptions() (options.cc:99)
==17407==    by 0x4762EF: __static_initialization_and_destruction_0(int, int) [clone .constprop.1298] (options_helper.h:387)
==17407==    by 0x8E011C: __libc_csu_init (in /home/vdb/pika/output/pika)
==17407==    by 0x65F2B44: (below main) (in /usr/lib64/libc-2.17.so)
==17407==
==17407== Events    : Ir
==17407== Collected : 11565385

rocksdb 检测脚本 加上 PORTABLE=1 主要还是-march=native这条导致的

• valgrind执行权限不足？子进程pika 无法修改sockte个数

14:17:17.436130 16158 pika.cc:173] your ‘limit -n ‘ of 1024 is not enough for Redis to start. pika can not reconfig it(Operation not permitted), do it by yourself

ulimit -n 102400

• 无法启动，原因是目录下有个叫log的文件

I0527 14:47:45.087769 18132 pika_binlog.cc:87] Binlog: Manifest file not x
exist, we create a new one.                                              x
Could not create logging file: Not a directory                           x
COULD NOT CREATE A LOGGINGFILE!F0527 14:47:45.087787 18132 pika_binlog.ccx
:92] Binlog: new ./log/log_db0/write2file IO error: ./log/log_db0/write2fx
ile0: Not a directory                                                    x
*** Check failure stack trace: ***                                       x
    @     0x7fc9dd5d1b5d  google::LogMessage::Fail()                     x
    @     0x7fc9dd5d37dd  google::LogMessage::SendToLog()                x
    @     0x7fc9dd5d1773  google::LogMessage::Flush()                    x
    @     0x7fc9dd5d41fe  google::LogMessageFatal::~LogMessageFatal()    x
    @           0x6032e6  Binlog::Binlog()                               x
    @           0x615519  Partition::Partition()                         x
    @           0x5f5f83  Table::Table()                                 x
    @           0x621525  PikaServer::InitTableStruct()                  x
    @           0x625d0c  PikaServer::Start()                            x
    @           0x429a54  main                                           x
    @     0x7fc9dbaa9bb5  __libc_start_main                              x
    @           0x54ea41  (unknown)                                      x
Aborted 

测试select命令路径的脚本 需要安装redis-tools

#!/bin/bash
for i in {0..10000}
do
    echo select $(expr $i % 8 ) |redis-cli -p 9221
done

spop命令不支持count参数，需要拓展

pika跑单测

  1 #!/bin/bash
  2 ./runtest --list-tests |while read line
  3 do
  4 ./runtest --clients 1 --single $line
  5 done

• 无法生成core文件

ulimit -c unlimited

core 目录

cat /proc/sys/kernel/core_pattern

也可以改core格式, 见参考链接

echo "core-%e-%p-%t" > /proc/sys/kernel/core_pattern

这样默认生成在程序所在目录

• 原生redis test适配，tcl脚本修改

  • 需要把pika改成redis-server pika需要指定配置文件 -c，tests/support/server.tcl

    set pid [exec src/redis-server -c $config_file > $stdout 2> $stderr &]
    

  • 配置文件和redis不一样，a : b格式的，redis是a b格式，脚本用dict保存，前面第一个字段和剩下的组成dict，为了生成独一无二的端口，需要重新设置，这可能就丢掉了:，需要手动加上

    • port前加上list

     set config {}
     foreach line $data {
          if {[string length $line] > 0 && [string index $line 0] ne "#"} {
              set elements [split $line " "]
              set directive [lrange $elements 0 0]
              set arguments [lrange $elements 1 end]
              dict set config $directive $arguments
           }
      }
      # use a different directory every time a server is started
      dict set config dir [tmpdir server]
      dict set config db-path [list ":" [tmpdir server]]
        
      # start every server on a different port
      set ::port [find_available_port [expr {$::port+1}]]
      dict set config port [list ":" $::port]
      #    dict set config port $::port
    

  • 文件目录，pika是db-path, redis是dir，需要加上个db-path定义，同上，需要加上:

  • 大量测试不支持，需要注释掉，tcl注释见参考链接

• python driver修改
  • conftest redis_version 改成pika_version #添加redis_version
  • conftest 注意db数，改成7 0~7 原生是0~9 #添加数据库

• redigo 测试修改 redisx/db_test.go : DialTest函数 不然会报 out of range

  _, err = c.Do("SELECT", "7")
  //_, err = c.Do("SELECT", "9")
  

• c driver 测试

  • 大量assert失败，简单hook

    /* The assert() calls below have side effects, so we need assert()
     * even if we are compiling without asserts (-DNDEBUG). */
    #ifdef NDEBUG
    #undef assert
    #define assert(e) (void)(e)
    #else
    #define assert(expr)     \
        if(expr) \
          do{ \
              fprintf(stderr, "Assertion failed:%s, file %s, line %d\n", #expr, __FILE__,__LINE__); \
             }while(0)
    #endif
    

  • 注意有redis_version 获取，要改成pika_version

  • unix sock，(/tmp/redis.sock）文件没有处理，相关测试屏蔽掉了，基本和上面没差别

redis release note

只列举功能相关部分，增强优化点不列出

兼容性对比

• bit操作部分支持主要是精度问题

• stream 不支持，未实现，5.0之后的命令都没有实现

• lua脚本相关命令都没有实现 这个依赖lua虚拟机。改动较大

• module相关命令都没有实现 这个涉及api移植，改动较大

• bit操作部分支持主要是精度问题

• 集群相关命令都没有实现

• 阻塞命令都没有实现

• rename没有实现 这个涉及到pika的元数据。由于跨db，改名也稍稍麻烦

• select数据库，pika只支持0-7 原生支持0-9 已经解决

• client命令只支持list和kill

• config命令 原生配置字段和pika配置字段不一致

• info命令显式的字段和redis原生不一致

• pexpire命令精度有限

• spop不支持count指令

• zadd不支持xxnxincr ch指令

• ping不支持参数

• 不支持quit

• 不支持flushdb和randomkey

• zset range 还有zincrby命令，double判断inf有问题，回复也不标准

• lpushx rpushx不支持多参数

• setrange 回复不标准

• ping在pubsub中的回复不标准

• expire ttl精度有问题，改时间戳解决

• zadd 加入元素最后的会更新，pika没有这个逻辑，主要是实现insert_or_assign这种东西

• smove行为也不太一样

• incr incrbyfloat行为也不一致

• zpopmax zpopmin命令没实现

ref

• shell for循环的几种写法https://www.cnblogs.com/EasonJim/p/8315939.html

• 命令行执行 redis命令，需要redis-tools redis-clihttps://stackoverflow.com/questions/31287826/how-to-execute-redis-command-in-shell

• gcc march选项 https://stackoverflow.com/questions/16273561/gcc-generic-march-for-all-the-x64-platforms

• valgrind不能attach https://stackoverflow.com/questions/26778959/attach-valgrind-with-daemon-and-collect-logs-for-each-daemon-call

• core文件控制https://blog.csdn.net/u011417820/article/details/71435031

• assert hook https://blog.csdn.net/ustcjackylau/article/details/40308321

• tcl 注释 http://noyesno.net/page/tcltk/comment.html

• 一个源码分析，已经有人做了。https://scottzzq.gitbooks.io/pika/content/
• 项目wiki https://github.com/Qihoo360/pika/wiki