Tweet

AntPickax

AntPickax includes open source software products which are necessary for Internet services in Yahoo! JAPAN. It includes basic libraries, components and systems.

AntPickax is a series of a challenging product that made it easy to solve complicated problems. We hope AntPickax products are widely used and create next innovations!

We will keep challenging to publish new open source software as a AntPickax product(like an Ant working with a pickax).

Background

Though we use and contribute a lot of open source software in Yahoo! JAPAN, we have started producing AntPickax with the following background.

• Basic functions that are necessary internally are not sufficient.
• Adopting a new architecture to drastically reduce operating costs.
• Performance (mainly speed and scalability) is insufficient.
• License restrictions, it can not be introduced inside the company.

Among the created software, we have released software as an AntPickax product that has performance comparable to that of existing OSS and has useful functions not found in existing OSS.

AntPickax Product List

AntPickax publishes the following products as open source.

K2HDKC DBaaS

K2HDKC DBaaS (Database as a Service for K2HDKC) is a Database as a Service that uses K2HR3 and works with OpenStack and kubernetes to build a K2HDKC(K2Hash based Distributed Kvs Cluster) Cluster for distributed KVS.

K2HR3

K2HR3 (K2Hdkc based Resource and Roles and policy Rules) is an RBAC(Role Based Access Control) system that cooperates with IaaS(Infrastructure as a Service).

K2HDKC

K2HDKC (K2Hash based Distributed Kvs Cluster) is a high performance and horizontal scalable distributed KVS cluster system based on K2HASH and CHMPX.

K2HFTFUSE

K2HFTFUSE (K2Hash File Transaction by FUSE based file system) is a component based on FUSE library which can transfer files, texts and logs at high speed and relay them and aggregate them.

CHMPX

CHMPX (Consistent Hashing Mq inProcess data eXchange) is a very fast network communication middleware to construct a cluster.

K2HASH

K2HASH is a vey fast Key Value Store(KVS) library featuring very large file support and many useful functions.

FULLOCK

FULLOCK (Fast User Level LOCK library) is a very fast and powerful lock library for multithread and multiprocess used in K2HASH, CHMPX.

AntPickax Products

K2HDKC DBaaS

K2HDKC DBaaS (Database as a Service for K2HDKC) is a Database as a Service that uses K2HR3 and works with OpenStack to build a K2HDKC(K2Hash based Distributed Kvs Cluster) Cluster for distributed KVS.

K2HDKC DBaaS (Database as a Service for K2HDKC) provides four ways.

• K2HDKC DBaaS with Trove
  This is DBaaS (Database as a Service) using Trove(Trove is Database as a Service for OpenStack), which is a product of OpenStack.
  This type embeds K2HDKC as one database(distributed KVS) of Trove(Trove is Database as a Service for OpenStack) and realizes DBaaS.

• K2HDKC DBaaS CLI
  Realize DBaaS (Database as a Service) by using Command Line Interface (CLI) for OpenStack.
  Delivered as a K2HDKC DBaaS CLI (Command Line Interface), it automates the construction and scaling of K2HDKC clusters, data merging, etc. as Database as a Service.

• K2HDKC DBaaS on Kubernetes CLI
  Realize DBaaS (Database as a Service) by using Command Line Interface (CLI) for kubernetes.
  Delivered as K2HDKC DBaaS on kubernetes CLI (Command Line Interface), it automates the construction and scaling of K2HDKC clusters as Database as a Service.

• K2HDKC Helm Chart
  Implement DBaaS (Database as a Service) using Helm(The package manager for Kubernetes) for kubernetes.
  Delivered as K2HDKC Helm Chart, it automates the construction and scaling of K2HDKC clusters as Database as a Service.
  In addition, K2HDKC Helm Chart can be registered as a repository in RANCHER as RANCHER Helm Chart and used.

K2HDKC DBaaS is built using Trove and K2HDKC, K2HR3, CHMPX, etc. published by Yahoo! JAPAN.

Please refer to the following for details.

• Information aggregation of K2HDKC DBaaS - Codes on github and Documents

• K2HDKC DBaaS with Trove - Codes on github and Usage

• K2HDKC DBaaS CLI(Command Line Interface) - Codes on github and Usage

• K2HDKC DBaaS on kubernetes CLI(Command Line Interface) - Codes on github and Usage

• K2HDKC Helm Chart - Codes on github and Usage

• Utility - K2HDKC DBaaS Override Configuration - Codes on github

K2HR3

K2HR3 (K2Hdkc based Resource and Roles and policy Rules) is one of RBAC (Role Based Access Control) systems provided by Yahoo! JAPAN.

K2HR3 works as RBAC in cooperation with OpenStack which is one of IaaS (Infrastructure as a Service), and also provides useful functions for using RBAC.

K2HR3 gathers common management information for the cloud, and is built K2HASH, K2HTP_DTOR, CHMPX and K2HDKC components.

K2HR3 consists of the following components and tools.

• Information aggregation of K2HR3 - Codes on github and Documents

• Web Application - Codes on github and Demonstration

• Command Line Interface(CLI) - Codes on github and Usage

• REST API - Codes on github and Demonstration

• Helm Chart - Codes on github and Documents

• OpenStack Notification Listener - Codes on github

• Utilities - Codes on github

• Container Registration Sidecar - Codes on github

• K2HR3 Get Resource - Codes on github

K2HDKC

K2HDKC (k2hash based Distributed Kvs Cluster) is a high performance and horizontal scalable distributed KVS cluster system based on K2HASH, CHMPX Distributed Key Value Store(KVS).

The K2HDKC has unique features shown below.

• Consistency (Automatic Data Synchronization between Server Nodes)
  Provides automatic merging function of data due to failure/recovery of server nodes in the cluster.
• Automatic Scaling
  Server nodes can be added/deleted to the cluster, and the data automatic merging function at this time is also provided.
• Nested key structure
  Provides the association function of key and subkey which is the feature of K2HASH as distributed KVS.
• Queue(FIFO/LIFO)
  Provides the queuing function which is the feature of K2HASH as distributed KVS.
• Transaction Plugins
  Provides a function that can perform arbitrary processing using data update processing which is the feature of K2HASH as a trigger of transaction.
• Encryption
  Provides encryption function as distributed KVS for the key’s data held as the feature of K2HASH.
• Expiration
  Provides the key expiration function which is the feature of K2HASH as distributed KVS.

For details, please refer to Codes on github, Documents.

K2HDKC nodejs addon

K2HDKC nodejs addon（K2HDKC nodejs addon - K2Hash based Distributed Kvs Cluster) is a Node.js addon library of K2HDKC.
Using this library, K2HDKC’s slave side client application can be implemented with Node.js based JavaScript.
K2HDKC nodejs addon provides all functions and features necessary for the slave side client of distributed KVS of K2HDKC.

For details, please refer to Codes on github, Documents.

K2HDKC phpext

K2HDKC phpext（K2HDKC phpext - Consistent Hashing Mq inProcess data eXchange) is a php extension library of K2HDKC.
Using this library, K2HDKC’s slave side client application can be implemented with this PHP Extention library.
K2HDKC phpext provides all functions and features of K2HDKC.

For details, please refer to Codes on github, Documents.

K2HDKC java driver

Java Driver for K2HDKC（Java Driver for K2HDKC) is an official Java client library for K2HDKC.
This library provies all functions and features necessary for the slave side client of distributed KVS of K2HDKC.
That means eveyone is able to implement K2HDKC’s slave side client application by using this library.

For details, see the Code on GitHub, Documents.

K2HDKC golang

K2HDKC golang (K2HDKC golang - K2Hash based Distributed Kvs Cluster) is K2HDKC’s Go language client library.
You can use this library to implement K2HDKC slave-side client applications in Go language.
K2HDKC golang provides all the features and features required for the slave side client of the distributed KVS of K2HDKC.

For details, see the Code on GitHub, Documents.

K2HDKC python

K2HDKC python (K2HDKC python - K2Hash based Distributed Kvs Cluster) is K2HDKC’s Python language client library.
You can use this library to implement K2HDKC slave-side client applications in Python language.
K2HDKC python provides all the features and features required for the slave side client of the distributed KVS of K2HDKC.

For details, see the Code on GitHub, Documents.

K2HFTFUSE

K2HFTFUSE(k2hash File Transaction by FUSE based file system) is FUSE (Filesystem in Userspace) is a file/message transfer system using the user space mounting function.

K2HFTFUSE is a system developed to realize reliable and fast file/message transfer at low cost.
K2HFTFUSE provides a virtual file system, you can use it by writing the file to the mounted directory.
Just by mounting the directory of the output file of the existing program with K2HFTFUSE, you can transfer files/messages without changing the existing program.

• Zero cost integration with applications
  Additional API dependency for client applications is zero!
• Very fast and reliable file transfer
  Very fast and reliable file transfer based on K2HASH and CHMPX.
• Various data format
  Supported data formats are a single data message, a text file and a binary file.
• Filter
  Processing data for your purpose very easily!
• Trigger
  Invoking your own function against particular data.

For details, please refer to Codes on github, Documents.

CHMPX

CHMPX (Consistent Hashing Mq inProcess data eXchange) is a communication middleware for sending and receiving binary data between processes that cross network.

• Basic functions
  CHMPX is responsible for communication between the server program and the client program, and hides the network communication connection from each program.
• Cluster/Multiplexing/Auto scaling
  CHMPX is a communication middleware that can create a cluster configuration, has high fault tolerance and multiplexing, and can be autoscaled.
• Communication data
  The format of the data to communicate is free, and it can communicate binary data and large size data.
• Communication encryption
  Supports communication with SSL.
• Communication multiplexing/Parallel processing
  Can multiplex and parallel communication between CHMPX server and slave.
• Queuing communication data
  Data to be transmitted/received is queued and data will not be lost even with small delay due to high load etc.
• Multi-threading/Multi-processing
  The client program can use this component in multi process, multithreading.

For details, please refer to Codes on github, Documents.

CHMPX nodejs addon

CHMPX nodejs addon（CHMPX nodejs addon - Consistent Hashing Mq inProcess data eXchange) is a Node.js addon library of CHMPX.
Using this library, both server side and slave side of communication program using CHMPX can be implemented with Node.js based JavaScript program.
CHMPX nodejs addon provides all functions and features of CHMPX.

For details, please refer to Codes on github, Documents.

CHMPX phpext

CHMPX phpext（CHMPX phpext - Consistent Hashing Mq inProcess data eXchange) is a php extension library of CHMPX.
Using this library, both server side and slave side of communication program using CHMPX can be implemented with your PHP application program.
CHMPX phpext provides all functions and features of CHMPX.

For details, please refer to Codes on github, Documents.

K2HASH

K2HASH is a NoSQL Key Value Store(KVS) library. K2HASH has basic KVS functions and unique features shown below.

• High speed
  Access to data(read and write) is very fast.
• Multi-threadeding/multi-processing available
  The client program can use this library in multi process, multithreading.
• Binary data
  Can use binary(any types of value) and variable length array(VLA) for Key and Value.
• Automatic expansion of data area/low fragment
  Dynamically extends the data area(for data and hash table).
  Use paging to minimize fragments.
• Data storage/mapping type
  Can store data on memory(volatile), persistent file(including temporary file).
  In the case of persistent files, it provides a mapping method for the whole file(data maintained at high speed and persisted) and index only(large capacity).
• Nested key structure
  In addition to KVS’s ability to store values for keys, you can associate other keys as keys to subkeys.
• Transaction Plugins
  Can implement your own processing by triggering data update processing as a transaction trigger.
• Archiving
  Data update processing can be output as an archive file as embedded transaction processing.
• Queue(FIFO/LIFO)
  Queue(FIFO/LIFO) can be configured as data of K2HASH and you can push/pop from it.
• Attributes
  Attributes(embedded or customized) can be set for the key.
• Encryption
  Can encrypt the key value and save it.
• History
  Can keep the update history of the key and this function can be used as a versioning for data.
• Expiration
  A function that can set the expiration date of the key as an attribute.

For details, please refer to Codes on github, Documents.

K2HASH nodejs addon

K2HASH nodejs addon（K2HASH nodejs addon - NoSQL Key Value Store(KVS) nodejs library) is a Node.js addon library of K2HASH.
Using this library, you can operate K2HASH’s data from your Node.js based JavaScript program.
K2HASH nodejs addon provides all functions and features of K2HASH.

For details, please refer to Codes on github, Documents.

K2HASH java driver

Java Driver for K2HASH（Java Driver for K2HASH) is an official Java client library for K2HASH.
This library provies all functions and features necessary for the slave side client of distributed KVS of K2HASH.
That means eveyone is able to implement K2HASH’s slave side client application by using this library.

For details, see the Code on GitHub, Documents.

K2HASH golang

K2HASH golang (K2HASH golang - NoSQL Key Value Store(KVS) Go library) is K2HASH’s Go language client library.
You can use this library to implement K2HASH slave-side client applications in Go language.
K2HASH golang provides all the functions and features of K2HASH.

For details, see the Code on GitHub, Documents.

K2HASH phpext

K2HASH phpext（K2HASH phpext - NoSQL Key Value Store(KVS) php extension library) is a php extension library of K2HASH.
Using this library, you can operate K2HASH’s data from your PHP application.
K2HASH phpext provides all functions and features of K2HASH.

For details, please refer to Codes on github, Documents.

K2HASH python

K2HASH python（K2HASH python - NoSQL Key Value Store(KVS) python library) is a python library of K2HASH.
Using this library, you can operate K2HASH’s data from your Python application.
K2HASH python provides all functions and features of K2HASH.

For details, please refer to Codes on github, Documents.

K2HTP_DTOR

K2HTP_DTOR(k2hash Transaction Plugin Distributed Transaction Of R epeater) easily duplicates the K2HASH data by transferring the transaction data of K2HASH to another host using CHMPX.

This library is a standard transaction plugin library compatible with the K2HASH library provided by Yahoo! JAPAN.
This provides a general tool for users to process their own transactions as transaction triggers.

For details, please refer to Codes on github, Documents.

K2HTP_MDTOR

K2HTP_MDTOR(k2hash Transaction Plugin Multiple Distributed Transaction Of R epeater) easily duplicates the K2HASH data by transferring the transaction data of K2HASH to another host using CHMPX.

The main purpose of K2HTP_MDTOR, which loads multiple K2HASH transaction plugins, is to call the arbitrary function defined in them. When calling the each function, K2HTP_MDTOR passes K2HASH transaction logs to it as a function’s parameter. A common K2HASH transaction plugin loads no other one. We recommend you to use K2HTP_MDTOR if you want to use multiple K2HASH transaction plugins.

For details, please refer to Codes on github, Documents.

FULLOCK

FULLOCK(Fast User Level LOCK library) is a low-level lock library that provides a safe and fast locking function for multiprocess, multithreaded programs.
This library is also used by other AntPickax products such as K2HASH, CHMPX.

For details, please refer to Codes on github, Documents.