Chapter 4, 5, 6

Encoding formats

xml, json, msgpack are text based encoding format, they can’t carry binary bytes (useless you encode them in base64, size grows 33%). And they cary schema definition with data, wast a lot of space.

thrift, protobuf are binary format, can take binary bytes, only carry data, the schema is defined with IDL(interface definition language). They have code generation tool to generate code to encode and decode data, along with check. Every field of data is binded with a tag(mapped to a field in IDL file). If a field is defined is required, it can’t by removed or change tag value, otherwise old code will not be able to decode it.

avro (used in hadoop), have a write schema and a read schema, when store a large file in avro format(contain many records with same schema), the avro write schama file is appended to the data. If use avro in RPC, the avro schema is exchanged during connection setup. When decoding avro, the lib will look both write schema and read schema, and translate write schema into read schema. Forward compatibility means that you can have a new version of the schema as writer and an old version of the schema as reader, backward compatibility means that you can have a new version of the schema as reader and an old version as writer.

Notes when reading chapter 2 “Data models and query languages”, chapter 3 “Storage and retrieval”

月初的时候搬了家, 之后的周末一直在忙些琐琐碎碎的事情，嘛，仔细一想，除了去宜家搬了个电视柜回来都不记得干了啥…

这周末心血来潮去听了两个讲座，一个人文的，一个技术的，还都碰到了点有意思的事情。

Create virtual resource on AWS is very convenient, but how to manage them will be a problem when your size grow.

You will come to:

• How to explain the detail online settings for your colleagues (like: how our prod vpc is setup?what’s the DHCP option set?), navigate around AWS console is okay, but not convenient.
• Who did what to which resource at when? AWS have a service called Config, can be used to track this change, but if you want to make things as clear as viewing git log, still a lot of works to do.

Ideally, we should manage AWS resources like code, all changes kept in VCS, so called Infrastructure as Code.

I’ve tried three ways to do it:

• ansible
• CloudFormation
• terraform

In this article, I’ll compare them, however, the conclusion is to use terraform 🙂

Ansible

Provision tools, like ansible/chef/puppet, all can be used to create aws resources, but they have some common problems:

• Hard to track changes after bootstrap.
• No confident what it will do to existing resources.

For example, I define a security group in ansibble:

ec2_group:
  name: "web"
  description: "security group in web"
  vpc_id: "vpc-xxx"
  region: "us-east-1"
  rules:
    - proto: tcp
      from_port: 80
      to_port: 80
      cidr_ip: 0.0.0.0/0

It will create a security group named “web” in vpc-xxx. At first glance, it’s convenient and straightforward.

A few notes taken when reading

Overview

MySQL has buildin partition table support, which can help split data accross multi tables,

and provide a unified query interface as normal tables.

Benefit:

• Easy data management: If we need to archive old data, and our table is partitioned by datetime, we can drop old partition directly.
• Speed up query based on partition key(partitoin pruning)

Limit:

• For partition table, every unique key must use every column in table’s partition expression(include primary key)
• For innodb engine, paritioned table can’t have foreign key,and can’t have columns referenced by foreign keys.
• For MyISAM engine, mysql version <= 5.6.5, DML operation will lock all partition as a whole.

In this article, let’s talk about ElasticSearch’s cluster mode, which means multi nodes ElasticSearch.

Basic concepts

cluster: A collection of server nodes with same cluster.name settings in elasticsearch.yaml

primary shards: Divide a index into multi parts(by default 5), shards of an index can be distributed over multi nodes. It enables scale index horizontally and make access to index parallelly(accross multi nodes).

replicas: backup for shards, also replicas can handle search requests, which means you can scale your search capacity horizontally via replicas.

心血来潮, 又看了遍黑客帝国三部曲, 当年的沃卓斯基兄弟都变成沃卓斯基姐妹了, 唏嘘啊…

第一次看的时候, 好像是初中吧, 记得看第三部还是姑父的盗版碟上看的, 那天还拉了个同学和我一起看,然后请他吃了泡面+冰淇淋,结果他回家就拉肚了,抱怨了我好久,所以印象特别深刻,哈哈.

杂乱笔记，辅助读paper.

看了下很久前 google 的 GFS 论文， 做点笔记。