feat(orm)!: ordered variable length encoding for uint32 and uint64 types (#11090)

## Description

`uint64` values are used in the ORM as auto-incrementing primary keys. Always using 8 bytes for these values is a bit of a waste of space. Unfortunately, varint encoding does not support ordered prefix iteration.

This PR introduces a compact, well-ordered variable length encoding for `uint32` and `uint64` types. `fixed32` and `fixed64` integers are still encoded as 4 and 8 byte fixed-length big-endian arrays. With this, users have a choice of encoding based on what type of data they are storing. An auto-incrementing primary key should prefer the variable length `uint64` whereas a fixed precision decimal might want to use `fixed64`.

See the golden test updates to see how this reduces key lengths.

This encoding works by using the first two bits to encode the buffer length (4 possible lengths). I'm not sure if my choice of 2,4,6 and 9 bytes is the right choice of 4 lenths for `uint64` - there are many alternate choices. I could have also chosen 3 bits and allowed for 8 possible lengths, but way waste an extra bit? Input on the right design parameters would be appreciated.



---

### Author Checklist

*All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.*

I have...

- [ ] included the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] added `!` to the type prefix if API or client breaking change
- [ ] targeted the correct branch (see [PR Targeting](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#pr-targeting))
- [ ] provided a link to the relevant issue or specification
- [ ] followed the guidelines for [building modules](https://github.com/cosmos/cosmos-sdk/blob/master/docs/building-modules)
- [ ] included the necessary unit and integration [tests](https://github.com/cosmos/cosmos-sdk/blob/master/CONTRIBUTING.md#testing)
- [ ] added a changelog entry to `CHANGELOG.md`
- [ ] included comments for [documenting Go code](https://blog.golang.org/godoc)
- [ ] updated the relevant documentation or specification
- [ ] reviewed "Files changed" and left comments if necessary
- [ ] confirmed all CI checks have passed

### Reviewers Checklist

*All items are required. Please add a note if the item is not applicable and please add
your handle next to the items reviewed if you only reviewed selected items.*

I have...

- [ ] confirmed the correct [type prefix](https://github.com/commitizen/conventional-commit-types/blob/v3.0.0/index.json) in the PR title
- [ ] confirmed `!` in the type prefix if API or client breaking change
- [ ] confirmed all author checklist items have been addressed 
- [ ] reviewed state machine logic
- [ ] reviewed API design and naming
- [ ] reviewed documentation is accurate
- [ ] reviewed tests and test coverage
- [ ] manually tested (if applicable)
This commit is contained in:
Aaron Craelius
2022-02-07 17:58:55 +00:00
committed by GitHub
parent a0a1197c27
commit 1944a0883e
7 changed files with 780 additions and 377 deletions
+13 -13
View File
@@ -1,54 +1,54 @@
GET 03000000000000000005
GET 03000005
PK testpb.ExampleAutoIncrementTable 5 -> {"id":5}
GET 03808002
SEQ testpb.ExampleAutoIncrementTable 0
GET 03000000000000000001
GET 03000001
PK testpb.ExampleAutoIncrementTable 1 -> {"id":1}
ORM INSERT testpb.ExampleAutoIncrementTable {"id":1,"x":"foo","y":5}
HAS 0301666f6f
ERR:EOF
SET 03000000000000000001 1203666f6f1805
SET 03000001 1203666f6f1805
PK testpb.ExampleAutoIncrementTable 1 -> {"id":1,"x":"foo","y":5}
SET 03808002 01
SEQ testpb.ExampleAutoIncrementTable 1
SET 0301666f6f 0000000000000001
SET 0301666f6f 0001
UNIQ testpb.ExampleAutoIncrementTable x : foo -> 1
GET 03808002 01
SEQ testpb.ExampleAutoIncrementTable 1
GET 03000000000000000002
GET 03000002
PK testpb.ExampleAutoIncrementTable 2 -> {"id":2}
ORM INSERT testpb.ExampleAutoIncrementTable {"id":2,"x":"bar","y":10}
HAS 0301626172
ERR:EOF
SET 03000000000000000002 1203626172180a
SET 03000002 1203626172180a
PK testpb.ExampleAutoIncrementTable 2 -> {"id":2,"x":"bar","y":10}
SET 03808002 02
SEQ testpb.ExampleAutoIncrementTable 2
SET 0301626172 0000000000000002
SET 0301626172 0002
UNIQ testpb.ExampleAutoIncrementTable x : bar -> 2
GET 03808002 02
SEQ testpb.ExampleAutoIncrementTable 2
ITERATOR 0300 -> 0301
VALID true
KEY 03000000000000000001 1203666f6f1805
KEY 03000001 1203666f6f1805
PK testpb.ExampleAutoIncrementTable 1 -> {"id":1,"x":"foo","y":5}
NEXT
VALID true
KEY 03000000000000000002 1203626172180a
KEY 03000002 1203626172180a
PK testpb.ExampleAutoIncrementTable 2 -> {"id":2,"x":"bar","y":10}
NEXT
VALID false
ITERATOR 0300 -> 0301
VALID true
KEY 03000000000000000001 1203666f6f1805
KEY 03000001 1203666f6f1805
PK testpb.ExampleAutoIncrementTable 1 -> {"id":1,"x":"foo","y":5}
KEY 03000000000000000001 1203666f6f1805
KEY 03000001 1203666f6f1805
PK testpb.ExampleAutoIncrementTable 1 -> {"id":1,"x":"foo","y":5}
NEXT
VALID true
KEY 03000000000000000002 1203626172180a
KEY 03000002 1203626172180a
PK testpb.ExampleAutoIncrementTable 2 -> {"id":2,"x":"bar","y":10}
KEY 03000000000000000002 1203626172180a
KEY 03000002 1203626172180a
PK testpb.ExampleAutoIncrementTable 2 -> {"id":2,"x":"bar","y":10}
NEXT
VALID false
File diff suppressed because it is too large Load Diff