From fd119840a36bd5e204f49f354824d12b98357dbf Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Mon, 17 Sep 2018 16:53:46 +1000 Subject: [PATCH 01/11] Update README with overview --- ssz/README.md | 241 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 233 insertions(+), 8 deletions(-) diff --git a/ssz/README.md b/ssz/README.md index ea31acfa8..451d90e70 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -1,8 +1,11 @@ -# simpleserialize (ssz) +# simpleserialize (ssz) [WIP] -This is a **work-in-progress** crate designed to perform the "simpleserialize" -serialization described by Vitalik Buterin. The method is tentatively intended -for use in the Ethereum Beacon Chain. +This is currently a ***Work In Progress*** crate. + + +SimpleSerialize is a serialization protocol described by Vitalik Buterin. The +method is tentatively intended for use in the Ethereum Beacon Chain as +described in the [Ethereum 2.1 Spec](https://notes.ethereum.org/s/Syj3QZSxm). There are two primary sources for this spec, and they are presently conflicting: @@ -14,8 +17,230 @@ conflicting: This implementation is presently a placeholder until the final spec is decided. Do not rely upon it for reference. -## TODO +### TODO + + * [ ] Wait for spec to finalize. + * [ ] Implement encoding for all useful types. + * [ ] Implement decoding. + +## SimpleSerialize Overview + +The ``simpleserialize`` method for serialization follows simple byte conversion, +making it effective and efficient for encoding and decoding. + +The decoding requires knowledge of the data **type** and the order of the +serialization. + +Syntax: + +| Shorthand | Meaning | +|:-------------|:---------------------------------------------------| +| `big` | ``big endian`` | +| `little` | ``little endian`` | +| `to_bytes` | convert to bytes params ``(size, byte order)`` | +| `from_bytes` | convert from bytes params ``(bytes, byte order)`` | +| `value` | the value to serialize | +| `len(value)` | get the length of the value. (number of bytes etc) | + +### Serialize/Encode + +#### int or uint: 8/16/32/64/256 + +Convert directly to bytes the size of the int. (e.g. ``int16 = 2 bytes``) + +| Check to perform | Code | +|:-----------------------|:------------------------| +| Int size is not 0 | ``int_size > 0`` | +| Size is a byte integer | ``int_size % 8 == 0`` | +| Value is less than max | ``2**int_size > value`` | + +``` +buffer_size = int_size / 8 +return value.to_bytes(buffer_size, 'big') +``` + +#### Address + +The address should already come as a hash/byte format. Ensure that length is +**20**. + +``` +assert( len(value) == 20 ) +return value +``` + +#### Hash32 + +The hash32 should already be a 32 byte length serialized data format. The safety +check ensures the 32 byte length is satisfied. + +``` +assert( len(value) == 32 ) +return value +``` + +#### Bytes + +For general `byte` type: +1. Get the length/number of bytes; Encode into a 4byte integer. +2. Append the value to the length and return: ``[ length_bytes ] + [ + value_bytes ]`` + +``` +byte_length = (len(value)).to_bytes(4, 'big') +return byte_length + value +``` + +#### List + +For lists of values, get the length of the list and then serialize the value +of each item in the list: +1. For each item in list: + 1. serialize. + 2. append to string. +2. Get size of serialized string. Encode into a 4 byte integer. + +``` +serialized_list_string = '' + +for item in value: + serialized_list_string += serialize(item) + +serialized_len = len(serialized_list_string) + +return serialized_len + serialized_list_string +``` + +### Deserialize/Decode + +The decoding requires knowledge of the type of the item to be decoded. When +performing decoding on an entire serialized string, it also requires knowledge +of what order the objects have been serialized in. + +Note: Each return will provide ``deserialized_object, new_index`` keeping track +of the new index. + +At each step, the following checks should be made: + +| Check Type | Check | +|:-------------------------|:----------------------------------------------------------| +| Ensure sufficient length | ``length(rawbytes) > current_index + deserialize_length`` | + +#### Int or Uint: 8/16/32/64/256 + +Convert directly from bytes into integer utilising the number of bytes the same +size as the integer length. (e.g. ``int16 == 2 bytes``) + +``` +byte_length = int_size / 8 +new_index = current_index + int_size +return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), new_index +``` + +#### Address + +Return the 20 bytes. + +``` +new_index = current_index + 20 +return rawbytes[current_index:current_index+20], new_index +``` + +#### Hash32 + +Return the 32 bytes. + +``` +new_index = current_index + 32 +return rawbytes[current_index:current_index+32], new_index +``` + +#### Bytes + +Get the length of the bytes, return the bytes. + +``` +bytes_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') +new_index = current_index + 4 + bytes_lenth +return rawbytes[current_index+4:current_index+4+bytes_length], new_index +``` + +#### List + +1. Get the length of the serialized list bytes. +2. Loop through the bytes; + 1. Deserialize the object with that length. + 2. Keep track of current position + +Note Before: there are a number of checks to be performed, ensuring there is +enough room left. + +| Check type | code | +|:------------------------------------|:--------------------------------------| +| rawbytes has enough left for length | ``len(rawbytes) > current_index + 4`` | + +``` +total_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') +new_index = current_index + 4 + total_length +item_index = current_index + 4 +deserialized_list = [] + +while item_index < new_index: + object, item_index = deserialize(rawbytes, item_index, item_type) + deserialized_list.append(object) + +return deserialized_list, new_index +``` + +## Technical Overview + +The SimpleSerialize is a simple method for serializing objects for use in the +Ethereum beacon chain proposed by Vitalik Buterin. There are currently two +implementations denoting the functionality, the [Reference +Implementation](https://github.com/ethereum/beacon_chain/blob/master/beacon_chain/utils/simpleserialize.py) +and the [Module](https://github.com/ethereum/research/tree/master/py_ssz) in +Ethereum research. It is being developed as a crate for the [**Rust programming +language**](https://www.rust-lang.org). + +The crate will provide the functionality to serialize several types in +accordance with the spec and provide a serialized stream of bytes. + +## Building + +ssz currently builds on **rust v1.27.1** + +### Installing Rust + +The [**Rustup**](https://rustup.rs/) tool provides functionality to easily +manage rust on your local instance. It is a recommended method for installing +rust. + +Installing on Linux or OSX: + +``` +curl https://sh.rustup.rs -sSf | sh +``` + +Installing on Windows: + +* 32 Bit: [ https://win.rustup.rs/i686 ](https://win.rustup.rs/i686) +* 64 Bit: [ https://win.rustup.rs/x86_64 ](https://win.rustup.rs/x86_64) + +## Dependencies + +All dependencies are listed in the ``Cargo.toml`` file. + +To build and install all related dependencies: + +``` +cargo build +``` + +### Running Tests + +Tests are included at the bottom of each of the implementation files. + +``` +cargo test +``` - - Wait for spec to finalize. - - Implement encoding for all useful types. - - Implement decoding. From 79ef5200b21b17c2daf43c8f09242977a28d3663 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Tue, 18 Sep 2018 11:59:08 +1000 Subject: [PATCH 02/11] Add ToC; Add Interface; Add Usage --- ssz/README.md | 159 ++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 153 insertions(+), 6 deletions(-) diff --git a/ssz/README.md b/ssz/README.md index 451d90e70..f76f88370 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -17,12 +17,46 @@ conflicting: This implementation is presently a placeholder until the final spec is decided. Do not rely upon it for reference. + +## Table of Contents + +* [SimpleSerialize Overview](#simpleserialize-overview) + + [Serialize/Encode](#serializeencode) + - [int or uint: 8/16/24/32/64/256](#int-or-uint-816243264256) + - [Address](#address) + - [Hash32](#hash32) + - [Bytes](#bytes) + - [List](#list) + + [Deserialize/Decode](#deserializedecode) + - [Int or Uint: 8/16/24/32/64/256](#int-or-uint-816243264256) + - [Address](#address-1) + - [Hash32](#hash32-1) + - [Bytes](#bytes-1) + - [List](#list-1) +* [Technical Overview](#technical-overview) +* [Building](#building) + + [Installing Rust](#installing-rust) +* [Dependencies](#dependencies) +* [Interface](#interface) + + [Encodable](#encodable) + + [SszStream](#sszstream) + - [new()](#new) + - [append(&mut self, value: &E) -> &mut Self](#appendmut-self-value-e---mut-self) + - [append_encoded_val(&mut self, vec: &Vec)](#append_encoded_valmut-self-vec-vec) + - [append_vec(&mut self, vec: &Vec)](#append_vecmut-self-vec-vec) + - [drain(self) -> Vec](#drainself---vec) +* [Usage](#usage) + + [Serializing/Encoding](#serializingencoding) + - [Rust](#rust) + ### TODO * [ ] Wait for spec to finalize. * [ ] Implement encoding for all useful types. * [ ] Implement decoding. +--- + ## SimpleSerialize Overview The ``simpleserialize`` method for serialization follows simple byte conversion, @@ -44,7 +78,7 @@ Syntax: ### Serialize/Encode -#### int or uint: 8/16/32/64/256 +#### int or uint: 8/16/24/32/64/256 Convert directly to bytes the size of the int. (e.g. ``int16 = 2 bytes``) @@ -126,7 +160,7 @@ At each step, the following checks should be made: |:-------------------------|:----------------------------------------------------------| | Ensure sufficient length | ``length(rawbytes) > current_index + deserialize_length`` | -#### Int or Uint: 8/16/32/64/256 +#### Int or Uint: 8/16/24/32/64/256 Convert directly from bytes into integer utilising the number of bytes the same size as the integer length. (e.g. ``int16 == 2 bytes``) @@ -236,11 +270,124 @@ To build and install all related dependencies: cargo build ``` -### Running Tests +--- -Tests are included at the bottom of each of the implementation files. +## Interface -``` -cargo test +### Encodable + +A type is **Encodable** if it has a valid ``ssz_append`` function. This is +used to ensure that the object/type can be serialized. + +```rust +pub trait Encodable { + fn ssz_append(&self, s: &mut SszStream); +} ``` + +### SszStream + +The main implementation is the `SszStream` struct. The struct contains a +buffer of bytes, a Vector of `uint8`. + +#### new() + +Create a new, empty instance of the SszStream. + +```rust +let mut ssz = SszStream::new() +``` + +#### append(&mut self, value: &E) -> &mut Self + +Appends a value that can be encoded into the stream. + +| Parameter | Description | +|:---------:|:-----------------------------------------| +| ``value`` | Encodable value to append to the stream. | + +```rust +ssz.append(&x) +``` + +#### append_encoded_val(&mut self, vec: &Vec) + +Appends some ssz encoded bytes to the stream. + +| Parameter | Description | +|:---------:|:----------------------------------| +| ``vec`` | A vector of serialized ssz bytes. | + +```rust +let mut a = [0, 1]; +ssz.append_encoded_val(&a.to_vec()); +``` + +#### append_vec(&mut self, vec: &Vec) + +Appends some vector (list) of encodable values to the stream. + +| Parameter | Description | +|:---------:|:----------------------------------------------| +| ``vec`` | Vector of Encodable objects to be serialized. | + +```rust +ssz.append_vec(attestations); +``` + +#### drain(self) -> Vec + +Consumes the ssz stream and returns the buffer of bytes. + +```rust +ssz.drain() +``` + + + +--- + +## Usage + +### Serializing/Encoding + +#### Rust + +Create the `simpleserialize` stream that will produce the serialized objects. + +```rust +let mut ssz = SszStream::new(); +``` + +Encode the values that you need by using the ``append(..)`` method on the `SszStream`. + +The **append** function is how the value gets serialized. + +```rust +let x: u64 = 1 << 32; +ssz.append(&x); +``` + +To get the serialized byte vector use ``drain()`` on the `SszStream`. + +```rust +ssz.drain() +``` + +**Example** + +```rust +// 1 << 32 = 4294967296; +// As bytes it should equal: [0,0,0,1,0,0,0] +let x: u64 = 1 << 32; + +// Create the new ssz stream +let mut ssz = SszStream::new(); + +// Serialize x +ssz.append(&x); + +// Check that it is correct. +assert_eq!(ssz.drain(), vec![0,0,0,1,0,0,0]); +``` From c024e7fd7dbf56ed84663330db2a05c574d8ea55 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Tue, 18 Sep 2018 12:12:35 +1000 Subject: [PATCH 03/11] Add detailed dependencies --- ssz/README.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/ssz/README.md b/ssz/README.md index f76f88370..c6643a260 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -270,6 +270,21 @@ To build and install all related dependencies: cargo build ``` +### bytes v0.4.9 + +The `bytes` crate provides effective Byte Buffer implementations and +interfaces. + +Documentation: [ https://docs.rs/bytes/0.4.9/bytes/ ](https://docs.rs/bytes/0.4.9/bytes/) + +### ethereum-types + +The `ethereum-types` provide primitives for types that are commonly used in the +ethereum protocol. This crate is provided by [Parity](https://www.parity.io/). + +Github: [ https://github.com/paritytech/primitives ](https://github.com/paritytech/primitives) + + --- ## Interface From 15ec33e790ccb76c06cc90f601ebaec4c107bd0c Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Tue, 18 Sep 2018 13:11:22 +1000 Subject: [PATCH 04/11] Update ToC with dependencies (Forgot in last commit) --- ssz/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/ssz/README.md b/ssz/README.md index c6643a260..53ff32abb 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -37,6 +37,8 @@ Do not rely upon it for reference. * [Building](#building) + [Installing Rust](#installing-rust) * [Dependencies](#dependencies) + + [bytes v0.4.9](#bytes-v049) + + [ethereum-types](#ethereum-types) * [Interface](#interface) + [Encodable](#encodable) + [SszStream](#sszstream) From 04b02854bccbc3aa928d89042980708462de0e5b Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Tue, 18 Sep 2018 17:32:51 +1000 Subject: [PATCH 05/11] Update readme syntax; Add explicit table for checks to be performed --- ssz/README.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/ssz/README.md b/ssz/README.md index 53ff32abb..338deeb1b 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -69,14 +69,14 @@ serialization. Syntax: -| Shorthand | Meaning | -|:-------------|:---------------------------------------------------| -| `big` | ``big endian`` | -| `little` | ``little endian`` | -| `to_bytes` | convert to bytes params ``(size, byte order)`` | -| `from_bytes` | convert from bytes params ``(bytes, byte order)`` | -| `value` | the value to serialize | -| `len(value)` | get the length of the value. (number of bytes etc) | +| Shorthand | Meaning | +|:-------------|:----------------------------------------------------| +| `big` | ``big endian`` | +| `to_bytes` | convert to bytes. Params: ``(size, byte order)`` | +| `from_bytes` | convert from bytes. Params: ``(bytes, byte order)`` | +| `value` | the value to serialize | +| `rawbytes` | raw encoded/serialized bytes | +| `len(value)` | get the length of the value. (number of bytes etc) | ### Serialize/Encode @@ -100,6 +100,10 @@ return value.to_bytes(buffer_size, 'big') The address should already come as a hash/byte format. Ensure that length is **20**. +| Check to perform | Code | +|:-----------------------|:---------------------| +| Length is correct (20) | ``len(value) == 20`` | + ``` assert( len(value) == 20 ) return value @@ -110,6 +114,10 @@ return value The hash32 should already be a 32 byte length serialized data format. The safety check ensures the 32 byte length is satisfied. +| Check to perform | Code | +|:-----------------------|:---------------------| +| Length is correct (32) | ``len(value) == 32`` | + ``` assert( len(value) == 32 ) return value From c5062bc4c41def1ec7e02a66d41a582e7ab52ed5 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Tue, 18 Sep 2018 11:20:43 -0700 Subject: [PATCH 06/11] Fixed SSZ link --- ssz/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ssz/README.md b/ssz/README.md index 338deeb1b..e7deec4fb 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -10,7 +10,7 @@ described in the [Ethereum 2.1 Spec](https://notes.ethereum.org/s/Syj3QZSxm). There are two primary sources for this spec, and they are presently conflicting: - - The ethereum/beacon_chain reference implementation [simpleserialize.py](https://github.com/ethereum/beacon_chain/blob/master/beacon_chain/utils/simpleserialize.py) file. + - The ethereum/beacon_chain reference implementation [simpleserialize.py](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py) file. - The [py_ssz module](https://github.com/ethereum/research/tree/master/py_ssz) in ethereum/research. From 7b4f72439e90d5c3ed964edf6a7e1f33f6283378 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Thu, 20 Sep 2018 00:22:29 +1000 Subject: [PATCH 07/11] Update README with deserialize --- ssz/README.md | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+) diff --git a/ssz/README.md b/ssz/README.md index e7deec4fb..89b80d2c0 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -310,6 +310,16 @@ pub trait Encodable { } ``` +### Decodable + +A type is **Decodable** if it has a valid ``ssz_decode`` function. This is +used to ensure the object is deserializable. + +```rust +pub trait Decodable: Sized { + fn ssz_decode(bytes: &[u8], index: usize) -> Result<(Self, usize), DecodeError>; +} +``` ### SszStream @@ -320,6 +330,8 @@ buffer of bytes, a Vector of `uint8`. Create a new, empty instance of the SszStream. +**Example** + ```rust let mut ssz = SszStream::new() ``` @@ -332,6 +344,8 @@ Appends a value that can be encoded into the stream. |:---------:|:-----------------------------------------| | ``value`` | Encodable value to append to the stream. | +**Example** + ```rust ssz.append(&x) ``` @@ -344,6 +358,8 @@ Appends some ssz encoded bytes to the stream. |:---------:|:----------------------------------| | ``vec`` | A vector of serialized ssz bytes. | +**Example** + ```rust let mut a = [0, 1]; ssz.append_encoded_val(&a.to_vec()); @@ -357,6 +373,8 @@ Appends some vector (list) of encodable values to the stream. |:---------:|:----------------------------------------------| | ``vec`` | Vector of Encodable objects to be serialized. | +**Example** + ```rust ssz.append_vec(attestations); ``` @@ -365,11 +383,82 @@ ssz.append_vec(attestations); Consumes the ssz stream and returns the buffer of bytes. +**Example** + ```rust ssz.drain() ``` +### decode_ssz(ssz_bytes: &[u8], index: usize) -> Result<(T, usize), DecodeError> +Decodes a single ssz serialized value of type `T`. Note: `T` must be decodable. + +| Parameter | Description | +|:-------------:|:------------------------------------| +| ``ssz_bytes`` | Serialized list of bytes. | +| ``index`` | Starting index to deserialize from. | + +**Returns** + +| Return Value | Description | +|:-------------------:|:----------------------------------------------| +| ``Tuple(T, usize)`` | Returns the tuple of the type and next index. | +| ``DecodeError`` | Error if the decoding could not be performed. | + +**Example** + +```rust +let res: Result<(u16, usize), DecodeError> = decode_ssz(&encoded_ssz, 0); +``` + +### decode_ssz_list(ssz_bytes: &[u8], index: usize) -> Result<(Vec, usize), DecodeError> + +Decodes a list of serialized values into a vector. + +| Parameter | Description | +|:-------------:|:------------------------------------| +| ``ssz_bytes`` | Serialized list of bytes. | +| ``index`` | Starting index to deserialize from. | + +**Returns** + +| Return Value | Description | +|:------------------------:|:----------------------------------------------| +| ``Tuple(Vec, usize)`` | Returns the tuple of the type and next index. | +| ``DecodeError`` | Error if the decoding could not be performed. | + +**Example** + +```rust +let decoded: Result<(Vec, usize), DecodeError> = decode_ssz_list( &encoded_ssz, 0); +``` + +### decode_length(bytes: &[u8], index: usize, length_bytes: usize) -> Result + +Deserializes the "length" value in the serialized bytes from the index. The +length of bytes is given (usually 4 stated in the reference implementation) and +is often the value appended to the list infront of the actual serialized +object. + +| Parameter | Description | +|:----------------:|:-------------------------------------------| +| ``bytes`` | Serialized list of bytes. | +| ``index`` | Starting index to deserialize from. | +| ``length_bytes`` | Number of bytes to deserialize into usize. | + + +**Returns** + +| Return Value | Description | +|:---------------:|:-----------------------------------------------------------| +| ``usize`` | The length of the serialized object following this length. | +| ``DecodeError`` | Error if the decoding could not be performed. | + +**Example** + +```rust +let length_of_serialized: Result = decode_length(&encoded, 0, 4); +``` --- @@ -416,3 +505,39 @@ ssz.append(&x); // Check that it is correct. assert_eq!(ssz.drain(), vec![0,0,0,1,0,0,0]); ``` + +## Deserializing/Decoding + +#### Rust + +From the `simpleserialize` bytes, we are converting to the object. + +```rust +let ssz = vec![0, 0, 8, 255, 255, 255, 255, 255, 255, 255, 255]; + +// Returns the result and the next index to decode. +let (result, index): (u64, usize) = decode_ssz(&ssz, 3).unwrap(); + +// Check for correctness +// 2**64-1 = 18446744073709551615 +assert_eq!(result, 18446744073709551615); +// Index = 3 (initial index) + 8 (8 byte int) = 11 +assert_eq!(index, 11); +``` + +Decoding a list of items: + +```rust +// Encoded/Serialized list with junk numbers at the front +let serialized_list = vec![ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0, 0, 32, 0, 0, 0, + 0, 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 15, 0, 0, 0, 0, + 0, 0, 0, 15, 0, 0, 0, 0, 0, 0, 0, 15]; + +// Returns the result (Vector of usize) and the index of the next +let decoded: (Vec, usize) = decode_ssz_list(&serialized_list, 10).unwrap(); + +// Check for correctness +assert_eq!(decoded.0, vec![15,15,15,15]); + +assert_eq!(decoded.1, 46); +``` From 8e2c1a9f18404aacbf59d89cae45a98949d810e4 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Thu, 20 Sep 2018 00:24:32 +1000 Subject: [PATCH 08/11] Update ToC on README --- ssz/README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/ssz/README.md b/ssz/README.md index 89b80d2c0..122cea93d 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -41,15 +41,21 @@ Do not rely upon it for reference. + [ethereum-types](#ethereum-types) * [Interface](#interface) + [Encodable](#encodable) + + [Decodable](#decodable) + [SszStream](#sszstream) - [new()](#new) - [append(&mut self, value: &E) -> &mut Self](#appendmut-self-value-e---mut-self) - [append_encoded_val(&mut self, vec: &Vec)](#append_encoded_valmut-self-vec-vec) - [append_vec(&mut self, vec: &Vec)](#append_vecmut-self-vec-vec) - [drain(self) -> Vec](#drainself---vec) + + [decode_ssz(ssz_bytes: &(u8), index: usize) -> Result](#decode_sszssz_bytes-u8-index-usize---resultt-usize-decodeerror) + + [decode_ssz_list(ssz_bytes: &(u8), index: usize) -> Result, usize), DecodeError>](#decode_ssz_listssz_bytes-u8-index-usize---resultvec-usize-decodeerror) + + [decode_length(bytes: &(u8), index: usize, length_bytes: usize) -> Result](#decode_lengthbytes-u8-index-usize-length_bytes-usize---resultusize-decodeerror) * [Usage](#usage) + [Serializing/Encoding](#serializingencoding) - [Rust](#rust) +* [Deserializing/Decoding](#deserializingdecoding) + - [Rust](#rust-1) ### TODO From 7217c6d62fb8f5db0f62f298b02bd4729229c766 Mon Sep 17 00:00:00 2001 From: terence tsao Date: Wed, 19 Sep 2018 10:09:10 -0700 Subject: [PATCH 09/11] fixed one more broken link --- ssz/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ssz/README.md b/ssz/README.md index 122cea93d..f80681c63 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -247,7 +247,7 @@ return deserialized_list, new_index The SimpleSerialize is a simple method for serializing objects for use in the Ethereum beacon chain proposed by Vitalik Buterin. There are currently two implementations denoting the functionality, the [Reference -Implementation](https://github.com/ethereum/beacon_chain/blob/master/beacon_chain/utils/simpleserialize.py) +Implementation](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py) and the [Module](https://github.com/ethereum/research/tree/master/py_ssz) in Ethereum research. It is being developed as a crate for the [**Rust programming language**](https://www.rust-lang.org). From 2add249f8fe3f7f185ad2636c1fe9890310f3623 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Fri, 21 Sep 2018 13:05:26 +1000 Subject: [PATCH 10/11] Address comments from #13 in ssz README --- ssz/README.md | 44 ++++++++++++++++++++------------------------ 1 file changed, 20 insertions(+), 24 deletions(-) diff --git a/ssz/README.md b/ssz/README.md index f80681c63..159e78c3f 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -57,12 +57,6 @@ Do not rely upon it for reference. * [Deserializing/Decoding](#deserializingdecoding) - [Rust](#rust-1) -### TODO - - * [ ] Wait for spec to finalize. - * [ ] Implement encoding for all useful types. - * [ ] Implement decoding. - --- ## SimpleSerialize Overview @@ -90,13 +84,15 @@ Syntax: Convert directly to bytes the size of the int. (e.g. ``int16 = 2 bytes``) +All integers are serialized as **big endian**. + | Check to perform | Code | |:-----------------------|:------------------------| | Int size is not 0 | ``int_size > 0`` | | Size is a byte integer | ``int_size % 8 == 0`` | | Value is less than max | ``2**int_size > value`` | -``` +```python buffer_size = int_size / 8 return value.to_bytes(buffer_size, 'big') ``` @@ -110,7 +106,7 @@ The address should already come as a hash/byte format. Ensure that length is |:-----------------------|:---------------------| | Length is correct (20) | ``len(value) == 20`` | -``` +```python assert( len(value) == 20 ) return value ``` @@ -124,7 +120,7 @@ check ensures the 32 byte length is satisfied. |:-----------------------|:---------------------| | Length is correct (32) | ``len(value) == 32`` | -``` +```python assert( len(value) == 32 ) return value ``` @@ -132,11 +128,11 @@ return value #### Bytes For general `byte` type: -1. Get the length/number of bytes; Encode into a 4byte integer. +1. Get the length/number of bytes; Encode into a 4 byte integer. 2. Append the value to the length and return: ``[ length_bytes ] + [ value_bytes ]`` -``` +```python byte_length = (len(value)).to_bytes(4, 'big') return byte_length + value ``` @@ -150,7 +146,7 @@ of each item in the list: 2. append to string. 2. Get size of serialized string. Encode into a 4 byte integer. -``` +```python serialized_list_string = '' for item in value: @@ -181,7 +177,9 @@ At each step, the following checks should be made: Convert directly from bytes into integer utilising the number of bytes the same size as the integer length. (e.g. ``int16 == 2 bytes``) -``` +All integers are interpreted as **big endian**. + +```python byte_length = int_size / 8 new_index = current_index + int_size return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), new_index @@ -191,7 +189,7 @@ return int.from_bytes(rawbytes[current_index:current_index+int_size], 'big'), ne Return the 20 bytes. -``` +```python new_index = current_index + 20 return rawbytes[current_index:current_index+20], new_index ``` @@ -200,7 +198,7 @@ return rawbytes[current_index:current_index+20], new_index Return the 32 bytes. -``` +```python new_index = current_index + 32 return rawbytes[current_index:current_index+32], new_index ``` @@ -209,7 +207,7 @@ return rawbytes[current_index:current_index+32], new_index Get the length of the bytes, return the bytes. -``` +```python bytes_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') new_index = current_index + 4 + bytes_lenth return rawbytes[current_index+4:current_index+4+bytes_length], new_index @@ -217,13 +215,11 @@ return rawbytes[current_index+4:current_index+4+bytes_length], new_index #### List -1. Get the length of the serialized list bytes. -2. Loop through the bytes; - 1. Deserialize the object with that length. - 2. Keep track of current position +Deserailize each object in the list. +1. Get the length of the serialized list. +2. Loop through deseralizing each item in the list until you reach the +entire length of the list. -Note Before: there are a number of checks to be performed, ensuring there is -enough room left. | Check type | code | |:------------------------------------|:--------------------------------------| @@ -267,7 +263,7 @@ rust. Installing on Linux or OSX: -``` +```bash curl https://sh.rustup.rs -sSf | sh ``` @@ -282,7 +278,7 @@ All dependencies are listed in the ``Cargo.toml`` file. To build and install all related dependencies: -``` +```bash cargo build ``` From b0cce139e08969ea30aa09e773ebe8d7e80a4714 Mon Sep 17 00:00:00 2001 From: NatoliChris Date: Fri, 21 Sep 2018 13:14:34 +1000 Subject: [PATCH 11/11] Rewrite first paragraph, fix remaining example block syntax --- ssz/README.md | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/ssz/README.md b/ssz/README.md index 159e78c3f..7355ca4cc 100644 --- a/ssz/README.md +++ b/ssz/README.md @@ -2,20 +2,18 @@ This is currently a ***Work In Progress*** crate. - SimpleSerialize is a serialization protocol described by Vitalik Buterin. The method is tentatively intended for use in the Ethereum Beacon Chain as described in the [Ethereum 2.1 Spec](https://notes.ethereum.org/s/Syj3QZSxm). +The Beacon Chain specification is the core, canonical specification which we +are following. -There are two primary sources for this spec, and they are presently -conflicting: +The current reference implementation has been described in the [Beacon Chain +Repository](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py). - - The ethereum/beacon_chain reference implementation [simpleserialize.py](https://github.com/ethereum/beacon_chain/blob/master/ssz/ssz.py) file. - - The [py_ssz module](https://github.com/ethereum/research/tree/master/py_ssz) - in ethereum/research. - -This implementation is presently a placeholder until the final spec is decided. -Do not rely upon it for reference. +*Please Note: This implementation is presently a placeholder until the final +spec is decided.*\ +*Do not rely upon it for reference.* ## Table of Contents @@ -225,7 +223,7 @@ entire length of the list. |:------------------------------------|:--------------------------------------| | rawbytes has enough left for length | ``len(rawbytes) > current_index + 4`` | -``` +```python total_length = int.from_bytes(rawbytes[current_index:current_index+4], 'big') new_index = current_index + 4 + total_length item_index = current_index + 4